Access points going offline every 2 hrs

[That1marine]

Just deployed 3 new AP's to my environment. MR36 and 2 CW9162I. These AP were added to a network with a mixture of older AP's (MR-12's, MR-18 and MR-42) running the 27.7.1 code. After upgrading all the AP's in my environment the older model AP's worked fine (running MR 26.8.3)  but the MR-36 and CW9162I (running MR 29.5.1) seem to loosing connectivity to the Meraki cloud platform every 2 hours. During this "offline" phase end users are still able to connect to the AP and access the internet. I am beginning to suspect this is a bug but would like to get the community perspective on this issue. The first 2 images are of the deployed CW9162I and the last image is of the MR-36. The MR-36 has seem to "fix" itself as it is not disconnecting any longer so I will continue to monitor. 

[DOC_Meraki]

Hi @That1marine , anything of note being logged in your firewall upstream?

[That1marine]

Nothing really on the Firewall that stands out.

[DOC_Meraki]

Is that a packet capture before/during the outage?

[That1marine]

This is currently when the AP is "online".

[That1marine]

This is FW logs when the AP is "offline".

[CoreyDavoll1]

Sounds like a failed firmware install.  I had something similar with some Z3's recently.  I'd open a case with tac and have them verify the right code is running since the dashboard will only tell you what code is supposed to be running.

[DOC_Meraki]

On that note, if they’re easily accessible give them a hard reset.  This would initiate the device to re-download firmware from the cloud

[That1marine]

Will they download the firmware code based on the base Network template? The MR36 seem to have stopped going online/offline and has been stable for 10 hrs now. The MR36 is currently running  MR 29.5.1 and CW9162I both appear to be running  MR 29.5. 

[PhilipDAth]

See if you cab check the local status page of an MR while it is reporting it is offline to see what issue it is reporting.

[That1marine]

I was not aware I can check the local status page remotely. Can you please share the steps?

[That1marine]

I was able to download the logs. Just sent it off to TAC for review.

[carver_g]

I had a very similar issue. I have about 20 Meraki AP's across 5 sites. This morning, I added a new AP to one of the sites and within a few minutes I started getting email alerts that APs went down. I logged into the Meraki Dashboard which reported that about 75% of my AP's across all sites were offline, although, like yours, they were still accepting clients and passing traffic. After about 45 minutes, they all came back "online" on their own. They are all running 29.5.1

The really strange part is that the APs that were "down" were seemingly random. For example, I put eight new MR-46's in our airport this year. They were all purchased at the same time, have sequential Mac addresses, are all connected to the same network, and six of the eight reported as off-line. Weird wild stuff.

[DOC_Meraki]

Bit of a running theme recently with sites/Orgs dropping APs!

You spot that @PhilipDAth ?

[PhilipDAth]

Yes I did notice that.  It is not affecting anyone on my side of the planet.

This might be just a Europe or US issue.  Where are you located @That1marine ?

[That1marine]

So after 24 hrs of being stable the site with the MR36 randomly went offline for about 4 hrs this morning then just came back online on its own. While the AP was "offline" user traffic was still being passed over the AP but was not accessible from the Meraki cloud. We never had issue with our AP's until our hardware refresh where we decided to upgrade to MR36 and CW9162I.

[cmr]

We had one AP go offline this morning (about 9 hours ago) but I just cycled the port it was connected to...  That's in Europe on shard 189.

[That1marine]

So we figured out the issue. The WSA was the culprit. According to TAC:

Once an MR is upgraded to firmware version 28.1 or higher, it requires TCP port 443 to be open on any upstream firewalls for Meraki Device-to-Cloud connectivity.

This issue was not impacting our older model AP's due to the highest firmware they are able to update to is MR 26.8.3. Those older AP's communicated over 7351 UDP.