Network Loop Issue

[Russ_B]

I had an issue this morning where a user created a loop by connecting both ports on the IP phone in the office to the switch.  Obviously that's a bad thing and will cause problems, but it essentially took the entire network down, and once I started thinking about it, that surprised me a bit.

This is a fairly large site, they have a 100Mbps WAN circuit, an MX84 and about 25 various MS series switches.  We have a total of 6 VLANs configured at the site, 3 data and 3 voice.  When I started getting reports from users on multiple VLANs, I suspected the WAN circuit, but I got low latency and no drops pinging the WAN router from my office.  Pinging the Uplink IP of the MX84 showed long strings of timeouts with occasional replies.  Same for any other IP I tried to PING at the site, regardless of VLAN.

At this point I figured it was a broadcast storm caused by a loop, so I sent someone to the site to track it down.  Once we removed the loop everything returned to normal.

At this point, I started wondering why multiple VLANs were affected by a broadcast storm, and it occurred to me that it could have saturated the trunks between the switches.  But when I started looking at the trunk links, I discovered that the switch that had the loop on it has a max 100Mbit link to the switch it is connected to, while most of the other trunk links are 1Gbps.  I would have thought the <= 100Mbps link between the switch with the loop and the other switches would have reduced the impact.

The switches are running MS 12.28, RSTP is on, and the switch the MX is plugged into is configured with bridge priority 0, all others are 32768.  I'm guessing spanning tree didn't help because of the switch in the phone, looking at the logs for the switch the phone was plugged into it looks like both ports would get disabled, then re-enabled around 4 minutes later.  That seemed pretty consistent for the entire time the loop was present.

Am I just underestimating the impact of ~100Mbps of broadcast traffic coming from a switch, or is there something else I'm missing?  

Thanks,

Russ

[cmr]

@Russ_B do you have any STP guard settings on the access ports?  It could be a spanning tree issue.  The guide here explains the options:

https://documentation.meraki.com/MS/Port_and_VLAN_Configuration/Configuring_Spanning_Tree_on_Meraki_...)

I'd try with BPDU guard as you don't want the phone switches participating in STP.

[GIdenJoe]

A broadcast storm does not respect any boundaries like VLAN's.  It will just nuke your link or make your switch CPU peg to 100% causing packets to stop forwarding.

STP will not block against a loop on a "dumb" switch or a switch even in a phone where there is a loop.

Only loopback protection can help against directly connected looped ports or ports via dumb switch.

Meraki switches should transmit these packets according to the docs.

https://documentation.meraki.com/MS/Monitoring_and_Reporting/Loop_and_MAC_Flap_Detection_on_MS

If you are running MS2x or above switches you can also use storm control to limit BUM traffic to a percentage of bw on a link to better survive these loops or even devices that are spamming the network.

[PhilipDAth]

As a matter of interest, on the phone - is the switch port and the data port on the same or different VLANs?

Aka, are the phone and PC on the same VLAN or different VLANs.

Do the phones have any options around spanning tree?

[BlakeRichardson]

Broadcast storms kill CPU which what cause massive latency.  The end result is the switch going 

[Russ_B]

@PhilipDAth  The Access ports on the switch are configured with separate voice and data VLANs, so normally the phone would be on VLAN 6 and the data would be on VLAN 5.  VLAN 5 is also the Management VLAN.

I don't manage the phones, but I downloaded the manual for the Avaya 9611G phones that we use.  There don't appear to be any options for spanning tree, the only reference I found in the manual is the statement "Spanning tree frames are always exchanged between the LAN port and the PC port in all VLAN separation modes."