Multiple issues - offline clients, unresponsive dashboard, spanning tree (all at once!)

[Wade_Carroll]

I'm dying here and it's only Sunday!  (my wife and I work from home so this is an enterprise network 🙂 )

While my network is functioning, I'm unable to view clients status, experiencing time outs or long delays accessing dashboard, and seeing spanning tree and port flapping on my MS120.

Example form just now:

Mar 2 12:22:48	Meraki MS120	Ports 8, 5, 8	Switch port	MAC address flapping	mac: 7E:AC:8F:C0:D6:A0, vlan: 1, port: 8,5,8
Mar 2 12:22:28	Meraki MS120	Ports 8, 5, 8	Switch port	MAC address flapping	mac: 7E:AC:8F:C0:D6:A0, vlan: 1, port: 8,5,8
Mar 2 12:11:05	Meraki MS120	Ports 7, 8, 7	Switch port	MAC address flapping	mac: D6:ED:57:5D:72:3F, vlan: 1, port: 7,8,7

Troubleshooting-

Replaced cable between MS120 and MX75

Changed upstream DNS provider from OpenDNS, to standard upstream, to google.  No detectable difference.

rebooted primary ISP fiber modem

rebooted Meraki MX75 and MS120 at various times.

Current status:

No visibility into client status

slow performance accessing Meraki.

Nervous wife.

[RWelch]

You are not alone.

The dashboard seems sluggish to me today too and I am not able to see Network-wide > client (data) on 10 of 12 networks.  It could be the dashboard.  I submitted a ticket to see if something on the backend is goofy.

Although data is flowing on all my networks, the UI telemetry is hit/miss as well as there is quite a lot of dashboard sluggishness when navigating from org to org or network to network.

[Wade_Carroll]

Thank you for the correlation.

I continue to have the issue and no troubleshooting has resolved.  Will hope they resolve whatever issues may be happening.  

The port flapping continued so I've replaced my Meraki MS120 with an unmanaged switch (not sure if that's good or bad, haha).

I've made sure no devices (MacBook, etc.) are connected to ethernet and wifi simultaneously.

One other thing I can't seem to figure out or rule out is I installed a MR36H and applied a port profile (default) that enables all ports to join my office network.  I only have 1 physical connection to the main ethernet port on the AP (the top one) but it's a little bit of a mystery in that there are 2 ports on back with one being a 'pass thru' so hopefully I connected to the proper port.

I've also got a MR78 outdoor AP installed.  It has a long ethernet patch cord going to a jumper connector in my attic.  Hopefully that's not throwing off errors either.  I have run cable test and it checks out.

[PhilipDAth]

Spanning tree events like that are often caused by AP clients roaming.  Do ports 7 and 8 have APs plugged in?  Otherwise what is plugged into those ports?

I am on shard n85 and it seems to be working ok for me.

[Wade_Carroll]

Yes ports 7 & 8 had APs connected.  One was a MR78 and the other a MR36H.

The Mac address referenced above was my Apple Watch which would roam a lot.  

I continue to have zero client information (all shown offline) and dashboard hangs when accessing simple things like an overview of Access Points or radio settings.  Finally got to radio settings.  Usually the spinning wheels just error out as shown below.

Not sure how to tell what shard I'm on but I'll look.  

[RWelch]

You can tell what shard you are on by looking at the beginning of the dashboard URL.
In this example it's N216 - which at the moment is SUPER sluggish and I (too) like you are having difficulties with the dashboard.

My other shards are working as expected and both thankful and grateful for that. 😃 

[Wade_Carroll]

Ahh, got it.  Thanks. 

I am on the shard of n216 better known as the shard of no workie…

[AidanKamp]

Thanks for bringing this to our attention, @Wade_Carroll.

I can confirm that we are noticing issues on our side with some shards - as @RWelch has noted, your shard will be the nX.meraki.com portion of the URL in your browser address bar.

This is currently being worked on as a high-severity issue by our engineering team. I'll check in once again once I have further updates. Thank you for your patience in the meantime.

[AidanKamp]

And to elaborate, the MAC flapping will likely be unrelated. As already mentioned, devices roaming between APs can generate MAC flap events as they roam. As long as your network is continuing to function, there should be no reason to worry about your internal network.

The issues seen with our shards will affect dashboard page load times, from our perspective. This might result in some data being unavailable (e.g. clients seen as offline, when they aren't) or some pages failing to load after 30 or 60 seconds.

[AidanKamp · Accepted solution]

Thanks for your patience!

Our team has confirmed that this issue should now be mitigated - if any further issues are seen please reach out to Meraki Support and we'll be happy to help. 💚

https://meraki.cisco.com/meraki-support/overview/

[Wade_Carroll]

Thank you!  Yes, all is back to normal now.

[RWelch]

@AidanKamp the dashboard was working fine for me initially but I made a DHCP change on one network and now all orgs/networks in N216 shard are back to showing no network-wide > client (data) UI telemetry.

And I have a support ticket open - maybe I should seek a way to get off the N216 shard? 😅

[RWelch]

And after waiting for roughly 20+ min, the network-wide > client (data) UI telemetry comes back.  20 min seems like a long time.  On the bright side, the dashboard is now showing relevant data (amen).

[AidanKamp]

@RWelch would you be happy to privately message me your case number and I can take a look? 

There was a mild amount of clean up that this incident required, but this should have been completed before your comment time. If the issue is reproducible, I'd be very interested in taking a look.

[RWelch]

@AidanKamp - private message sent.