ESP8266 and frequent reconnects fail with latest AP firmware MR 25.9

[mazevedo]

Since the "forced" upgrade to MR 25.9 firmware, my ESP8266 devices fail to reconnect to Meraki if they disconnect and connect frequently. 

These devices can enter deep sleep and connect again every few minutes to update their data. If those periods are smaller than about 10 min, they won't connect to Meraki networks.

Before MR 25.9 was released, I tried the beta firmware and noticed this problem. Believing it to be a bug to be solved, I reverted back to the previous version, and didn't think more about it, as the devices worked well before.

I now notice that if I'm developing with the ESP and I have multiple restarts, this device won't connect again to Meraki until a few minutes later.

On the event log for these devices I only see messages like this:

Access point SSID Client Event type Details

Time (CET)
Feb 20 23:22:48	ap-eglisau	calvin	ESP_21BB73	802.11 disassociation	client has left AP
Feb 20 23:22:39	ap-eglisau	calvin	ESP_21BB73	WPA authentication
Feb 20 23:22:39	ap-eglisau	calvin	ESP_21BB73	802.11 association	channel: 1, rssi: 33
Feb 20 23:22:38	ap-eglisau	calvin	ESP_21BB73	WPA authentication
Feb 20 23:22:38	ap-eglisau	calvin	ESP_21BB73	802.11 association	channel: 1, rssi: 34
Feb 20 23:22:21	ap-eglisau	calvin	ESP_21BB73	WPA authentication
Feb 20 23:22:21	ap-eglisau	calvin	ESP_21BB73	WPA deauthentication	radio: 0, vap: 2, client_mac: 60:01:94:21:BB:73  more »
Feb 20 23:22:21	ap-eglisau	calvin	ESP_21BB73	802.11 association	channel: 1, rssi: 30
Feb 20 23:22:17	ap-eglisau	calvin	ESP_21BB73	802.11 disassociation	client has left AP
Feb 20 23:22:02	ap-eglisau	calvin	ESP_21BB73	WPA authentication

Anybody with this issue too?

[MilesMeraki]

I've not got any ESP8266 devices to test however I'm not having any problems reported within our Wireless environments.

I assume you've engaged support to rule out any bugs? Roll back if it's causing you a pain. I'm sure you can work with support to troubleshoot, they'll be able to pin the firmware version to a single AP to be able to troubleshoot in real-time with you.

[mazevedo]

Thanks, I thought firmware downgrade for this version was no longer permited.

I've rolled back and I'll open a support ticket.

[Puma]

I have probably the same problem with the same symptomps and devices (Meraki AP, software version and ESP8266). With another AP ESP8266 works correctly.

Did you open the case already?

[mazevedo]

Yes, case open.

We have found out so far that it's a DHCP related issue.

With fixed IP address, the ESP works great with both versions.

Waiting on TAC to analyse the logs.

Will keep posted.

[moechine]

I am having the same issue only with iOS devices.  Constant disconnects and the device is reporting no internet connection and when checking the network info it reports a self-assigned IP address.  I changed my edge router as well thinking that was where the problem was but it wasn't.  I have engaged support as well.  

[mazevedo]

TAC says that the ESP rejects the IP address give by the Access Point

 

ESP8266 rejects DHCP offerI've tried with other access points, but still using Meraki's DHCP given by the gateway. It works.

So, I'm sure the problem is in the handling of the DHCP relay by the Access Point.

Lets hope TAC finds what's causing this soon.

[mazevedo · Accepted solution]

Just a quick follow-up on this issue.

I've eventually discovered that having "WPA encryption mode" set to "WPA1 and WPA2" was causing this issue.

Setting to "WPA2 only" the ESP8266 devices connect without problem.

I'm not sure if TAC will follow this up. If they will, I'll continue to update this post.

For me this is now solved.

[kkd702]

Our MR18 has been set to "WPA2 Only" for quite some time.  I tried upgrading the firmware to 25.9 and our TempStick immediately loses connectivity.  Can you list the other settings on your device, such as how frequently it is checking in?

[mazevedo]

My AP is a MR32.

Here are the settings:

• Pre-shared key: WPA2 
• WPA Encryption mode: WPA2 only
• 802.11r: Disabled
• 802.11w: Disabled
• Splash: None
• Client IP Assignment: Bridge Mode
• VLAN tagging: No
• Bonjour forwarding: Disabled
• Dual band: 2.4 and 5 Ghz
• Minimum bitrate: 1Mbps (Full compatibility with legacy devices (pre-1999))

I also noticed that with devices using ESP8266 Core 2.3 that this still doesn't work, but re-flashing with  code compiled with Core 2.4 the problem is solved.

As I'm writing this, I found a device still running with Core 2.3 and it in fact does not connect using the 25.9 firmware after you reset it multiple times.

So it seems the problem is in fact two folded. With WPA2 only we still get this:

1. ESPCore 2.3              
   1. AP firmware 24.11    OK
   2. AP firmware 25.9      Not OK
2. ESPCore 2.4
   1. AP firmware 24.11   OK
   2. AP firmware 25.9     OK

Could you please try with your Arduino code to upgrade to Core 2.4? I've found it's generally safe and includes the WPA CRACK fix. Some libraries might need to be updated.

Very rarely, after a reset, the ESP8266 does not connect, but this already happened before and on other networks, so I'm not sure if this is related. My code stop/restarts connections after a few minutes if not connected, so usually I don't see it.

This could probably need more extensive testing, but it's working with me.

I'll inform Cisco about the Core 2.3 and 25.9

[kkd702]

Thank you for the settings.  

Another temporary workaround for ESPCore 2.3 devices is to run your AP in NAT mode.  I switched mine, upgraded to firmware 25.9 and my wireless device is still connecting.  

[MrHarrisbeach]

I was having this issue with IOS devices as well. Drop/Connect. Standing up a new SSID was the only workaround for me. TAC is also looking at my Monitored Capture Logs and I am waiting to hear back. But in the mean time I made a change on one of my original SSIDs that everyone was using until the issue started. I enabled the 802.11w option in the Encryption Mode section and the individuals who could not connect using that SSID connected immediately!!

Now, I'm going to leave this in place for now and see how it goes.

[exetico]

I have problems with the ESP-chips, too.

The wierd thing is is that it's working just fine the first time the device calling the DHCP-server, and recieve 10.0.0.128 - but after this moment, it's DHCPDECLINE over and over again. However the `deauthentication` is also present in our logs for the ESP-devices.

Our AP's is in bridge-mode, and they have the 2.4 GHz-option (and it's enabled) for the SSID - and all ESP-chips are trying to connect to the 2.4GHz SSID. But it doesn't help. WPA1 and WPA2 vs WPA2-only is, sadly, the same story.

The ESP-chips works just fine on one of our old and dump ASUS AP's - no problems at all...

The devices works fine without static DHCP-mapping in pfSense (dhcpd), so temporary we manage to just use fx. SonOff's with the MQTT-protocol, but it's not an option for some of our other devices, fx. a few ESP-based smart light-bulbs.

[exetico]

Hi @mazevedo

I'm facing the same issue, and have selected WPA2 only. It doesnt' work. I can also see DHCPDECLINE in my DHCP-logs, but this only happens if the Meraki-AP is hosting the devices, so i guess it's a question about the current configuration of the Meraki-AP.

Are you able to share the rest of your settings?

Fx. Minimum bitrate (Mbps), Client IP assignment and fx. support for 802.11r /802.11w.

Thanks! 🙂

[J901]

Still nothing?  ESP8266 devices just do not work with Meraki?

[J901]

I want to add, that this is extremely frustrating, especially as I fill out with ESP8266 devices in nearly everything - from wall sockets, to dimmers, to other devices. Some of them have requirements to communicate (broadcast) on another segment hence why I cannot live with NAT mode. I just switched to NAT mode temporarily and devices -instantly- connected. I have no trouble with other devices. Whats the deal???

Meraki dudes - PLEASE - When will this be fixed? I post about one "frustrated" post on the internet once every 5 years. This is the one.

[exetico]

I had raised two supportticket to them - however they just blame our DHCP service (powered by PFSense).

If i swap the Meraki device with our old ASUS device as AP, the ESP devicds connects without any issues, what so ever.

With that said, i'm able to connect our devices, if i skip the IP mapping, but that's only an option, cause all our ESP devicss now supporting MQTT.

All at Meraki still blame my setup (DHCP), but i'm sure that it's but in the AP's.

EDIT: Damn auto-corrections. Will leave them here, like the bugs in the Meraki AP's 😉

[mattvirus]

what's the ticket number... PM me or post it here.

I'm a engineer at Cisco, and i had this same exact bug impact a big workshop i was planning.  Luckily, i did tests at the last hour and was able to revert to a different wlan solution.  This needs to be fixed, absolutely. 

[exetico]

Hi mattvirus

I've sent you a private message. Please take a look at it - and answer by mail. I'll forward all needed information, but i'll also like to secure that you're actually from the Cisco-team, before i repose all our information to a stranger.

Thanks!

[mattvirus]

i responded via email the day you sent the PM.  I will send again....

[exetico]

Hi mattvirus

I've not heard anything from Meraki since we had a short conversation by mail. They asked a single time about, if it still was a problem - and i kindly told them, that it's still a problem.

Could anyone please take the proper steps to fix this?

I've seen a lot of ESP8266 + Meraki problems, and after Googling a bit, it's petty clear that it's a combination of a mishandling in the ESP-chip, but also maybe a bit to hard handling by the Meraki device. So i would love to have that fixed, so our devices can stay connected.

I've also tried to test a NodeMCU on the network, and that's the same. It won't connect to our network. However, it's joining my private hotspot with no issues what so ever.

So please......

https://github.com/espressif/arduino-esp32/issues/1245

https://help.konnected.io/support/discussions/topics/32000001234/page/2?url_locale=

https://www.letscontrolit.com/forum/viewtopic.php?t=4927

[jameswah]

I had a very similar problem on a Mobility Express network. The issue seems to be that the ESP8266 firmware, on receiving a DHCP offer, will send an ARP request to see if anyone is using the same IP. If it gets a reply, it rejects the IP and asks for another. The problem is, at least with ME, the infrastructure responds from its own ARP cache. Then, if the ESP8266 has been on the network recently, it gets an ARP response with its own MAC address! - and barfs. The workaround on ME is to disable ARP caching with `config flexconnect arp-caching disable`, though it's a band-aid. It looks like there is a relevant bug on the Espressif core: https://github.com/espressif/esp-idf/issues/2182 It also seems like other LWIP apps could have the same behaviour.