Random STP changes across network

[rabusiak]

Hi Guys,
Need some advice on figuring out why I see "Detecting a high rate of STP topology changes" alerts on random switches (on their uplink ports) at random times.
I migrated from mixed switch environment (Catalyst/HP/Aruba) to pure MS setup. Attaching topology.

Core stack is the root bridge with priority 8192, default priority is 32768 and one switch has priority 36864. Core stack is 2x MS390-24 with 10G modules. Fibers goes to access switches, one switch doesn't have direct connection with core stack (the one with lowest priority on far right side).
Setup was done some time ago and all was fine, then I started to get those random alerts. Don't recall if it was after some firmware update or a reboot. Maybe after MS390 swapped from MS firmware to CS... I'm not sure.
One core stack switch is in alerting mode because of a firmware bug - it thinks 10G ports 1 and 2 are connected to copper ports 1 and 2 and reports vlan mismatch. This was confirmed by Meraki support. According to them it doesn't have impact on network and is already fixed in one of the beta firmware versions.

Should I change root bridge priority to 0?
Should I try to enable BPDU Guard on all access ports network wide? Maybe someone is connecting a switch somewhere...
Other ideas?

[PhilipDAth]

Have you got "Opt-in to view and manage alerts in a new centralized Alert Page" enabled under Organisation/Early Access?

If so - I think this has a bug at the moment, and if you turn it off the alerts will go away (I think they are a false positive).

 

Failing that, do any of the switch ports show there is a blocked connection to another switch (indicating a loop somewhere)?

>Should I try to enable BPDU Guard on all access ports network wide?

Have you definitely got the ports configured as access ports, as opposed to trunk ports?
I don't see any harm in enabling BPDU guard if the above is already done.

[rabusiak]

I did opt-in for that... I will opt-out to verify 😉

Still it's just alerts that might by buggy... if I check those uplink ports on switch pages they also reports stp changes in last 2h:

Now I see that only one stack doesn't have any issues at all, the one which is connected with core-stack directly with fibers (not through any fiber patch panel). All the rest reports STP changes in last 2 hours.

[GIdenJoe]

It wouldn't hurt to run a longer packet capture to capture the actual BPDU's and see what is happening.

The capture filter you need is: ether host 01:80:c2:00:00:00

[PhilipDAth]

Expert above.  🙂

[GIdenJoe]

I wish 😉
I have a very weird issue between a pair of MX250's and two stacks of MS355-24X2's that is truly kicking my ass.
For some weird reason whenever I add a new VLAN downstream of the MX'es forwarding to the active MX just stops and I even get RSTP topology changes until I remove both uplinks on the active MX250 and it fails over.  I haven't been able to zero in on the actual issue.

[cmr]

@GIdenJoe what's the exact topology and versions.  Our main DC has a stack of four 355-24Xs doing the routing with a single ended pair of MX250s hanging off them that terminates all the Auto VPNs from the other sites.

[GIdenJoe]

I'm not going to fully highjack this post.  Gonna do another one perhaps with some graphics to thoroughly explain it.

[rabusiak]

I run the packet capture but probably my lack of expertise is holding me back from seeing something suspicious 😉
At random intervals there are packets like those 2 marked in red below. The only difference is that they have "topology change flag up" but bridge and anything else stays the same... 

[GIdenJoe]

I think the frame before that one should have the proposal flag on it.
In RSTP and MSTP (which also uses RSTP as convergence mechanism) whenever there is a link that comes up or a change in the BPDU's coming from the root each switch will send the updated BPDU down it's designated ports while shutting those ports down.  The downstream bridge that has it's root port there must send an agreement back to open the port up again for forwarding. It will not do that on alternate ports so those ports stay blocked.

So it might be a needle in a haystack but somewhere in your network you probably have a non edge port that is flapping or more than 1 switch vying for root bridge status.