APs showing offline

[JohnF5]

I had about 120 APs go offline on the dashboard yesterday. They cover about 3 of my locations. They are still delivering connectivity but I can't see them to manage them from the dashboard. I also can't go out to the web address on my LAN to see them either. We've tried power cycling and factory reset with no luck. I can see my other two locations no problem. 

[alemabrahao]

Have you opened a support case?

[JohnF5]

I have. I was instructed to factory reset a device. We did but no luck.

[Brash]

Is there anything in common with those 3 sites? Do they share the same interest connection? Do they all use the same template in the dashboard?

Did the AP's in those networks get a firmware upgrade?

[JohnF5]

We have 5 sites. They all share the same internet connection. When you say template what do you mean? I'm not too familiar with that.

The AP's did not get a firmware upgrade.

[JohnF5]

Under configuration templates, we have nothing setup.

[Ryan_Miles]

For that many devices to all go unreachable at the same time I'd say it has to be an upstream issue. Were any changes made on the switches or upstream layer 3 devices or firewall(s)?

Also, I took a quick look at the Support case and your dashboard. I noticed from an online AP I can ping the mgmt IP of the APs showing as offline. So, to your point they are still up & functioning, but not communicating with dashboard. 

Doing factory resets seems likely pointless to me as 120 devices don't all go bad at the same time. I'd focus on upstream troubleshooting (firewall logs, etc).

[JohnF5]

No changes were made to the firewall. I'll check on the switch on Monday. I can tell you one other site in our dashboard that is up is actually downstream from one of the sites that is down. They are on the same subnet and use the same default gateway. 

[Ryan_Miles]

Something like that would make me suspicious that an upstream NAT rule or something changed preventing one subnet from working while another is ok.

[JohnF5]

No idea. Woke up this morning and everything is magically online again. 🤔

[Ryan_Miles]

I see that. However, almost every AP that has come back now shows this.

This means UDP 7351 is not working and it's using HTTP/HTTPS. More info HERE.

[PhilipDAth]

To add to Ryan's response, I would get someone to check the firewall for blocked traffic from one of the AP IP addresses.  Anything being blocked?

[JohnF5]

Checked the firewall and traffic is being allowed.

action="accept"

dstinetsvc="Hurricane.Electric-Hurricane.Electric.Internet.Services"

dstintfrole="wan"

dstip=64.62.142.12

dstport=7351

[Ryan_Miles]

Does the firewall rule allow UDP 7351 to these 4 IPs/subnets per the firewall info page?

64.62.142.12/32

158.115.128.0/19

209.206.48.0/20

216.157.128.0/20

Checking a location of yours that didn't have the outage shows passing 7351 test to 209.206.x.x IPs. Comparing to sites with the outage those APs still show failures for 7351 checks to various 209.206.x.x IPs.

The AP's failing the test are the ones also showing the warning "Connection to the Cisco Meraki Cloud is using the backup Cloud connection."