Anyone else seeing MR46 AP's dropping / loosing connection from Dashboard when running 28.6 firmware?
We see 20 - 30 devices drop across the country, different networks / geographic locations usually all at the same time.
Looks like a back end dashboard issue, but trying to get past "its in engineering" is getting old.
I'm looking for other Meraki Customers with similar experiences so we can compare AP version / Firmware versions and get some visibility.
The AP's do continue to pass traffic and are online. So not an outage issue, but a PITA is you manage 3,500+ access points.
If you reboot the AP's they come back online in dashboard, of course taking the AP out of service in doing so. Or if you wait for a DHCP 1/2 life checking point (24 hour leases, renew at 12 hours) they will check back into Dashboard and go green.
Just one of many issues, but posting his in the AP group to see what others are experiencing.
If there Meraki Product owners for MR, MS, MX or Dashboard components I'd love to have a conversation.
Just replying back to your thread since you posted helpful info to my question. Our MR46 AP's began dropping as soon as we pushed out 28.6, prior to that they were rock solid and pretty much never went down unless power went down.
Same here, Trying to find out if we should rollback until they get this corrected, but don't want to risk dropping any AP's on a rollback if there is a solution coming soon. So far no guidance either way. Roll back and expose security issues, stay current and drive network team crazy. ☹️
Also use MR55s, as well as MR56, MR84 and MR86 on 28.6. For us the AP drops started after updating our MS390 firmware to 15.14 on 4/29/22.
Can confirm it’s not a cosmetic issue, the APs stop responding to pings and clients cannot connect.
Power cycling the APs switchport restores functionality and I don’t believe I’ve had to restart the same unit twice but still frustrating.
This morning I had 9 APs down:
Also considered upgrading to 28.6.1 but it doesn’t seem to address the issue.
Similar experience. I'm on 28.6.1 now. With no pattern at all, here and there I have APs that loose the connection. Before, they would regain fairly quick... 3-5 minutes. Now, occasionally, they will stay offline unless I power cycle the port in the switch. Some weeks it happens to 1-5 APs, some weeks like this past one, none.
3 APs disconnected earlier today. For some reason they don't come back online automatically. If I cycle the port the reconnect. If I wait long enough, eventually they may come online. There is no pattern to this, which frustrates me. Firmware 28.6.1.
Today it's these 3. Tomorrow most likely none. Next week it will be another 2-4 and so on. Different switches, different APs.
These 3 are on the same switch. But there are other APs on the same switch and they didn't disconnect.
They come back when they hit their DHCP 1/2 life renewal. Why? assuming they check back in with a potential IP address change. So depending on when they drop, and what your renewal window is they could be down for some time. Its defiantly tied to the NeXT Tunnel issues. Got pretty ugly on our side last night, sounds like they were making changes. In our case we have done enough packet captures off of switches to know they are still passing traffic and serving clients. We can ping them and they reply. We can see them talking to the Meraki Cloud servers, just show down in the dashboard.
PITA as we still need to verify every AP.
Waiting on them to put the Dashboard and MR teams into a room and get this figured out. At least let us watch the cage fight!
Updates at bottom of posting.
There is definitely a known issue with some WAPs running 28.X code dropping / losing connection with the Meraki cloud dashboard. A change was made between MR 27.x and 28.x code on how the WAPs communicate to the Cisco Meraki Cloud to support FIPS compliance, and that appears to be the likely root cause. See FIPS links at bottom. If you are seeing this issue and have not opened a case, it's probably time to do so. My case number is 08050752 if you'd like to reference it.
I have a pair of MR56 WAPs, one running 28.6.1 and the other running 27.7.1. The historical device data for connectivity to the cloud (below) speaks for itself. The Support Engineer I am working with says I am seeing it a lot worse than most folks.
Lastly, a similar change also appears to have been made to the MS 15.1+ code and to the MX 16.4+ code as well. I have no idea if these new code levels will have similar problems, but a little extra caution is warranted for now.
UPDATE from Meraki Support: "I have narrowed down your issue to only the MR56s or Wifi 6 APs for now. The switches typically do not show this behavior and it is not expected you will experience this for the switches."
Update with SOLUTION to my issue.
The issue I have been seeing is separate and distinct from the issue of WAPs disconnecting from the cloud occasionally. If you are seeing the connect and disconnect happen every 3-10 minutes over and over and over, read on.
Actual problem found was that our new Internet link (about 3 weeks old) was misconfigured for an MTU of 1496. (Should have been 1500). As soon as the MTU was corrected, connectivity from the MR56s running 28.6.1 to the Meraki cloud became stable.
I hope this saves someone else a few hours of aggravation.
One more thing: An Engineer from another firewall company took packet captures at the firewall upstream from the WAPs and before the Internet link where the issue was. He noticed that we had a lot of TCP packet retransmissions on larger packets with the DF (don't fragment) bit set. That was the hint that led to finding the MTU problem on the provider's link.
Same here. We have 38 MR56 access points currently offline within the dashboard. I can ping the local IP addresses but nothing else. They're coming up the going down intermittently.
Same issue here - my MR46 and MR36 running 28.6 are dropping off the dashboard without any real regularity - will show as offline for 45 minutes to 3+ hours, still passing traffic and getting new connections, but it is frustrating from an alerting standpoint.
I had a very similar experience (Wi-Fi 6 AP running 28.6 shows offline, continues to server clients, didn't respond to ping, I could get to the local status page of the AP from a connected client, cycling the switch port on my MS switch resolved it or it self- resolved after time). I opened a ticket and support asked that I don't cycle the port and let them know next time it happens. It hasn't happened since. I also have some peers who are also seeing this occur.
For all those of us experiencing this, if you have an open case with support, please add the URL of this thread to your case notes. You can do so in case management in help menu in the dashboard.
I got the problem too. Once I went to 28.6 code it all went downhill, tried 28.6.1 as well with no help. My case number: 07967754. It's only my MR56 units. My MR42 and MR34 units are fine.
Back again today. 2 APs earlier and 2 APs now, all MR46. I need to cycle the switch port for the MRs device in order to regain cloud connection.
Same. We have at least 4-5 offline alerts everyday. Typically false positives. Very frustrating that Meraki deems these are "backend issues" without additional context.
I have a rather large system....had 15 go offline this AM across 13 different networks. Toggled PoE on their switch port on all of them to regain connectivity to the cloud. No sooner was I done, 12 more dropped off. This is nuts.
I added a link to this thread in my case (07948308). I have been lucky as this is occurring infrequently for me and only affects a couple APs at a time. However, what is the point of a dashboard if you can trust it and its notifications.
Has anyone noticed if when the MR's stay offline from the dashboard for an extended period that they stop allowing clients to connect ? Had one this morning and had to bounce the port on it
I have not experienced that. When mine go "offline" they still allow connections and pass traffic but the AP and any clients attached to it will appear offline in the dashboard
Hmm strange ours were offline for 8 hours, they also showed config not up date.
different question then do MR's stop accepting clients if they cant contact the dashboard for a set time and be deemed a dead ap or does the last known config apply and the MR carries on serving clients?