Some APS offline after Internet connection restored

[A4WanMan]

Scenario:  All switches (MS390s running beta 15.9) and APs (MR46 & MR56 running 28.6) were online in the dashboard.  Our Internet connection that supports 6 schools was bouncing this morning - enough that all devices were offline.  After the Internet connection became stable, all switches are now connected to the Meraki cloud and showing online.

However, some of the APs across the district are showing offline. 

What is the best way to get the APs back online?  In the past, I have cycled the switchport the AP is connected to.  This is fine for a few, but approximately half of the APs at each school is showing offline.  Shouldn't the AP try to reconnect?  Internet connection has been restored for about 2 hours now.

[DarrenOC]

Hi @A4WanMan , whilst they’re still showing as offline the APs will still be active and passing traffic locally to clients.

Have you tried giving one AP a kick and seeing what happens?

[A4WanMan]

The APs are mounted on the celing grid.  I can't kick that high. 🙂

What do you mean by "giving an AP a kick"?  If I go to the switch port where the AP is connected, cycling the port usually brings it back Online.  I have had to disable/enable the port and POE.

I have about 130 offline across 6 sites.  That's a lot of kicking and clicking.

[Brash]

I'm assuming by giving it a kick, @DarrenOC meant power cycling one AP to test and confirm whether that brings it back online.

The AP's should constantly be trying to reach out to the cloud. Do you have any not connected to MS390's? Given they're fairly flakey and are running beta code I would initially be thinking they might be an issue.

A couple of other things to check, is the AP"s MGMT IP reachable within your network? Can you do packet captures to verify the AP is sending packets to the Meraki cloud? I can't remember if capturing of the AP"s LAN interface will capture management traffic.

[A4WanMan · Accepted solution]

From Meraki:

There is an internal ticket that was opened about MRs on Firmware 28.6 losing cloud connectivity.

All the APs are back online now.  The reconnect process took longer (4 - 5 hours) than I expected. 

[JGill]

We should update the "Solved" for this issue to reference 28.7 firmware.  So far appears to have corrected this issue.

[LilRhody]

I am also suffering from this same issue since moving to 28.6. I see there is a 28.6.1 firmware now available, have you tried that one to see if it fixes the issue?

[JGill]

The release notes don't mention that being fixed.  Just new device support.   I just posted a similar issue.  All MR46 / MR55.  MR42's don't seem to be impacted.   They do pass traffic, but a true PITA if you have a lot of devices to manage.   And its not just comm losses that cause it, but is very much a catalyst if you have a comm issue.    Ticket is in Engineering as a NextTunnel issue.   We see them drop across the country at the same time, so unless Meraki Cloud is having hiccups, not just a local communication outage causing the  issue. 

[LilRhody]

Just popping in to see how it's been going for the rest of you? I have to say I haven't had an access point go down in about 2 weeks, I wonder if they've corrected the issue or maybe I'm just lucky this week

[DarrenOC]

That’s jinxed it 😂🤣

[LilRhody]

100% but at least it's summer and there's nobody here if one goes down 🙂

[LilRhody]

Had 3 go down this weekend so apparently I did jinx myself lol

[JGill]

I had 19 locations (out of 240) down this morning.  18 had 1 AP down,  1 location had 2 down.

Heard there is a firmware update in testing to address the issue.

[JGill]

At least the new dashboard makes if very clear throughout the day.  Of course is so noisy we have missed real events.  Maybe someone on the development team(s) should spend some time on the support side of network operations.  Over 4 months to fix this?  Meraki you can do better than this!

[pjc]

I had a site with 8 x MR44 that had lost it's internet connection for 3 days, once the internet was restored, the AP's would not come online in the dashboard but I could ping them OK- I had to cycle the POE ports on the switch (reboot the AP's) to get them to come online. V28.6.1 firmware.  Clients were unable to connect while they were showing as offline

[LENNIE1]

We are seeing this A LOT at the moment- random loss of single MR APs that are still passing traffic. Also recovered a school this morning and three hours later there are still 6 MRs claiming to be offline

Even worse, the Summary page seems to update infrequently- sending you off down a rabbit hole of panic, when all is actually well...

Running 28.6.1

[LilRhody]

FYI I received an email from Meraki this morning that the issue has supposedly been corrected in firmware 28.7

[JGill]

We have been slowly rolling out 28.7.   So far we have not had one of those sites show on our list of failed AP's.   Still early with < 5% of the AP's converted, but we are hopeful!    Rolling out another 766 AP's over 53 networks tonight so should have a good feel by the end of the week if this fixes the issue.

[JGill]

1,792 of 3,385 Completed.    No Access Points dropped off so far today!    Looking like 28.7 may have this issue fixed!

[A4WanMan]

I may try 1 site to see if it works.  I don't see the problem until connection is lost to the cloud either by power going down or the Internet connection goes down.  When the Internet connection is restored and the Meraki switches connect back to the cloud, the AP make take 4 hours or so before they re-connect.  The APs are functional and wireless clients can connect and data is passed.  They just show being offline in the dashboard until the AP eventually tries to connect to the dashboard.  I quit worrying about it and just have been waiting until they come back online.

[JGill]

What we saw was AP still operated, but showed down in dashboard.  If the DHCP 1/2 life or renewal window hit they would check-in and come back online in the dashboard.  Well over 50% of our AP's updated to 28.7 with no drop offs and no new issues reported. 

[LENNIE1]

Just an update for all of you- everything has improved since the rollout of MR28.7