MR34 and 28.12 major issues

[Asavoy]

I just recently updated all APs (mostly MR34 with a few MR42) and they are ALL now showing as going offline for a minute at a time multiple times a day.  123 active APs.

Issue did not happen before update to 28.12

I have 1 AP at an offsite location that is on 23.11 and has zero issues of this nature.

My predecessor, who now works for Meraki, told me to never,ever,ever try the Beta firmware.... so now what?

[Welles]

Everbody needs to get on the same page when it comes to “Beta” as fixes are rolled into Beta and take forever to hit the “up to date/stable” branch”. I’ve ran into issues that are resolved in Beta, but then other stuff breaks.  

[Asavoy]

I'm in an edu situation with 4 campuses, so I cannot afford any downtime. Again, my predecessor was recruited by Meraki and I agree with his opinion about trying the Beta.

How difficult is it to step back down to 24.8 which had no issues?

[Navjot]

If you did the firmware update pretty recently, just go back to Organization --> Firmware Upgrades, and roll back to the firmware you were previously on.

[Asavoy]

Thanks Nerdy!  Rolling back now.

[Asavoy]

Well, so that didn't work- still have the 1 minute drop issues at the one campus I rolled back. Obviously something was upgraded that doesn't roll back with a downgrade (what is this, Apple?).

Any suggestions? Nothing else changed in the network other than upgrading firmware to 24.12.

As I was typing this post, I just watched an AP go down for a minute with 1 client connected, an Apple TV that's not being used because nobody is there.

Help?

[PhilipDAth]

Is the AP actually showing as down, or are you seeing it down in some other way?

Anything appearing in the event log during the down event?

[Welles]

Kinda sounds like an issue I had with a bunch of MR32s ... did an upgrade .. saw an MR33 acting odd, then rolled back. All the MR32s appeared functional in the dashboard, then a minute later they all showed offline, had to cycle the ports(i.e. reboot) then all the APs showed back up fine. I even waited 10 minutes to see if the dashboard was just “slow”, nope.  

[Asavoy]

Something just really isn't adding up.  I don't think that what's happening is really happening, if that makes sense.

On the 17th there was a county-wide outage of our provider due to a road crew severing a fiber line.  I did the firmware upgrades on the 20th, and the issues started on the 22nd/23rd depending on the device I'm looking at.

Yesterday, I saw a lot of red (via the Meraki dashboard) at one of my campuses... but I was there on site and connected just fine to an AP that was reporting as 'unreachable for about 1 hour'.  I have alerts turned on at that site and am not getting anything that correlates to APs going down.  I got me wondering if anything was actually going down, or if the dashboard is showing erroneous info.  So, I decided to run a ping test overnight from my server to Meraki.com and to an AP.  Of 54k packets sent to Meraki, 119 were lost, which is more than acceptable, and 0 packets were lost going to the AP.  Both the main MS420 switch and that AP had 4 instances of the 1-minute No Connectivity during my ping tests.  There is nothing in the Event Logs pointing to any issues of lost connection either.

So, it looks like the equipment isn't broken, but the dashboard is severely crippled?

Anybody know what IP or URL these devices send their info to for the Meraki dashboard?

[dfurasek]

Just curious, did you see this recent announcement:

https://documentation.meraki.com/zGeneral_Administration/Other_Topics/Cloud_Maintenance_FAQ

...and adjust your firewall accordingly?

[Asavoy]

Thanks for the link.

I'll run a ping test with timestamp to an address in that range to see what happens.

If this is the solution, I'm sure glad my organization was 'proactively notified'... lol.

[dfurasek]

I was made aware of it via a banner announcement that appeared at the top of all pages in the dashboard.  I do know that those banners can be flaky, as there have been times in the past that one or more users in our office (using their own credentials) have seen announcements that others have not, so maybe you fell victim to something like that.

Also, if you are planning on just doing a standard ICMP ping, I don't know that it will suffice, as you need to allow traffic through your firewall on specific TCP and UDP ports.  As the FAQ mentions, you can refer to the Help -> Firewall Info page in the dashboard for all of the details.  I generally like to use the "Download unfiltered rules as CSV" option on that page to see the greatest level of detail on what sort of changes I might (or might not) need to make in order to allow our Meraki hardware to work properly.

[MRCUR]

@Asavoy FYI - not every address in the new range is being used yet. They're just now beginning to migrate shards over to the new space - https://bgp.he.net/net/209.206.48.0/24#_dns

[Asavoy]

@MRCUR - Do you have an idea on what I have to tell the people that administer my firewall?  I gave them the IP range and explained the issue, and gave them a few of my AP IPs to test and this is what the response was- "I ran a packet trace through the firewall from those address to the new Meraki subnet and the connection was allowed for any combination of ports, as long as the connection is initiated from your network.  I don’t believe it’s a firewall issue."

However, I took a spare MR34 that had been used temporarily at a site and plugged it into my home network, so I can confirm that the issue that I originally thought I had is not a problem.  It is just the connection to the Meraki cloud being interrupted.

[PhilipDAth]

While logged into the dashboard clock on Help (top right hand corner) and then "Firewall Info".

That will give you the firewall rules needed to allow cloud communication work for your organisation.

[MRCUR]

@Asavoy Follow the recommendation from @PhilipDAth. That way you can ensure all of the necessary rules are in place.