MS issue after upgrading to 12.27 firmware

[PatR-31C]

Ran into an issue after updating a site to MS 12.27.    After the update, clients at the site reported poor connectivity to internal systems as well as internet.   

Site consists of 2 x MX100 (HA Pair), 2 x MS425  as distribution stack, and 8 x MS350 access stacks.

MXs connected to MS425 stack.   MS350 stacks connected to MS425 stack with dual fiber in aggregated pair.    MS425 stack running L3 and is the STP root.

After 12.27 upgrade, clients complaining of slow connection speeds.   Clients getting decent download speeds, but upload speeds almost zero.    Running dual ISPs to MXs.   Issue existed with both ISP links.

Finally went onsite and tested the following:

• Tested directly from ISP handoff - no issue
• Tested directly from MX port - no issue
• Tested directly from MS425 port - no issue
• Testing from MS350 access switch port - ISSUE

Once the issue came up at the access switch, I removed the aggregation on the uplink fibers and the problem goes away.   That was a bit of a head scratcher as nothing changed in our configuration other than the firmware from 11.30 to 12.27.   Those aggregated links have been on that network since it was created almost 2 years ago with no issues.   I also have another location that is similarly configured and it is not exhibiting the same behavior.    What could have caused or could be causing this?

[PhilipDAth]

I bet a power cycle of the MS350 stack would have fixed it as well.

[PatR-31C]

I thought the same many hours ago, but alas no.   MS350 stack reboot had no effect.

[PhilipDAth · Accepted solution]

Next thing I would try is deleting the LACP configuration and then putting it back in place in case it got corrupted in some way.

[PatR-31C]

Thanks for the responses.

Do you think the LACP config on the MS425 was more likely the culprit than the MS350 stacks given we're talking 1 stack vs. 8?

[PhilipDAth]

I don't think there is anyway to tell with the information available.

[Salvo]

Hi PatR-31C,

we had a similar issue in our production enviroment with 2 ms350 in stack.

Those switch are configured with L3 and have uplink to a MX84
After upgrading we experienced a 60% packet loss toward Uplink (internet) connectivity.
No problem in L3 Lan routing.
We haven't tested your solution yet (about removing channel aggregations), but we would like to know if this is a meraki certified solution or not.

Thanks in advance.

[jmalcaide]

Same issue with 2 x MS425  as distribution stack, and 2 x MS225 access stack.

50% packet loss when testing from MS225 access switch port.

If we disable one of the ports in the aggregation group, the problem goes away.

I don't know if the problem is in the MS425 stack synchronization. A reboot of the MS425 stack also fixes the problem but obviously this workaround is not the best for a distribution device.

[Kenny_PCC]

We had what sounds like the same issue, slow speeds or loss of DNS connectivity after connecting any dual aggregated pairs on our MS425 stack. Unplugging one fiber link would resolve the issue. We tried rebooting the stack but that did not seem to resolve. We didn't try deleting the LACP configuration.

Downgrading to firmware 11.31 seems to have resolved the issue for us, we've not had it happen again. We're waiting until seeing something about it in a firmware release before upgrading.