Traffic flooding on every port at the stack

[theshmike]

I recently opened a case at Meraki. I am just also posting this here and hope anybody can help!

Our switches sw1, sw2, sw3 and sw4 are tied together to a stack with stacking cables.

When I do a ping from a host that is connected to any port at the stack to a host connected to any other switch at the network (outside the stack), I can see the ping requests on every port of the stack. I cannot see the ping replies.
If I ping a host on the same stack, I also can see the replies.

So, to conclude: The traffic, that ingresses on any access port of the stack seems to be flooded out to every port at the same stack.

We noticed this, because all users with a 100m connection (users that hang behind an IP phone with a 100m switch)  cannot communicate anymore when someone else with a 1G connection on the same stack started a large file transfer to/from a file server. The traffic was flooded out to every port on the stack and killed the clients.

The behaviour is reproducable and you can see the flooded packets with a packet capture on the port itself as well as with a capture at the connected clients.

[jdsilva]

Wow that's bizarre. It's as if the switch stack is unable to populate it's MAC address table and is perpetually doing unknown unicast flooding. Does the MAC address table show anything?

I wish I could help, but this one is mostly likely going to have to be dealt with by support unless someone here has encountered it before 😞

[theshmike]

---

@jdsilva wrote:

Does the MAC address table show anything?

Nope, MAC table is absolute normal to me...

[nealgs]

what switch models and firmware are they all on?

[theshmike]

They are all MS210-48FP on 11.22.

Of course, support told us to update to 11.31 before they‘ll do anything.

we‘ve scheduled the update to today’s evening, so let’s see if that helps...

[theshmike]

...did not help at all. Scary!

[jdsilva]

Yeh if I was to guess, and keep in mind I'm not an electrical engineer, it sounds like something in the ASIC microcode has gone screwy. Updating the switch firmware would only help if that particular upgrade had an ASIC microcode update inside it. You can probably up- and down-grade all over the place and it won't make a bit of difference unless you just happen to hit the right combo to force that microcode update. 

Again, total wild guess. I have no way to verify my theory, nor does it help you in any way 😞

[Gumby]

I've had that, or something very similar to that before after a firmware upgrade.  No noticeable issues that night after the update but the next day every time our incremental backup kicked off the network ground to a halt.  Had to power down the entire stack and power back up one at a time.

[theshmike]

That's really weird! I'm gonna wait if support tells me anything today. Otherwise, I'll try that this evening.

This problem caused about 30 of our IP phones (Cisco devices btw 🤣) to crash last friday. The're pretty old and it seems that the traffic flooding crashed the internal switches and now the IP phones won't come up again...

[ww]

@theshmike was there any fix?

[theshmike]

@wwWe've updated everything to the latest firmware and luckily never faced the issue again since then.

But however, I've found nothing related to this issue in the any release note...