MS 17.1.4 Firmware Breaks Ports 25-52 as Uplinks on MS220-48LP Switch

[ShenzouX]

I have a lab environment setup that I vet all the firmware releases on before I roll them into production and I've found a glaring issue with the MS220-48LP Switches when you go above 16.9. so I was hoping to get some feedback here.

I realize the MS220-48LP went end of life about 6 months ago but I still have many of them in my environment across 50+ branch offices and it will take time to replace them all with something newer.

In the meantime the firmware section of the dashboard is telling me I need to update to the newest switch firmware (17.1.4) for security and bug fixes.

So in my lab I have an MR33 and MR46 AP uplinked to the MS220-48LP and then the MS220-48LP uplinked to an MX68W which is then connected to the internet.  I'm using port 48 as the uplink port to the MX on the switch.  Its as basic as you can get setup-wise.  No firewall rules or ACLs just plug and play devices with a few VLANs configured. 

As soon as I did the firmware update the access points never came back online and completely stopped talking to the dashboard.  I plugged a laptop into an access port on the MS220-48LP and it gave me absolutely no connection to anything (I couldn't even ping the switch let along get through the switch to the MX or the internet)

I got on the phone with tech support and they said they were seeing some "strange" entries in the backend logs concerning port 25-52 so on a whim I changed my uplink port to number 24 and everything started working (APs, Laptop, etc).  I was even able to plug phones, laptops, and the APs into port above 25 as access ports and everything still worked fine.  This appears to ONLY affect up linking.

It seems on the 17.1.4 firmware if you use any port ABOVE 24 as an uplink on an MS220-48LP NONE of the other ports will be able to pass traffic through the switch.

I can't have a bunch of end users moving uplink ports around at all these offices either - that's not really an option.

I setup a second test environment with a different MS220-48LP switch just to make sure it wasn't a fluke and was able to replicate the exact same behavior.

I wanted to see if anybody else is experiencing this or if it can possibly be fixed in a future update as I have a bunch of 220-48s (and 320-48s.... not sure if these are impacted but I'm scared to upgrade them now).

If firmware releases are going to brick my switches then why doesn't Meraki gate the firmware by model just like they do with APs and MXs?  

[RWelch]

I've upgraded to MS17.1.4 on MS120s, MS125s, MS130s, MS225s, MS250s, MS350s, and MS425s without any issues or problems.  I've not upgraded any of the models you mentioned above.  Hopefully if it's a bug they can remedy quickly so as to not impact a large # of switches in your environment.

[ShenzouX]

Yeah I have all those models you mentioned above as well across all the different branch offices and those all upgraded fine as well for me.  Its just the 220 48 ports (the 220 24 ports work fine with no issues).

Its like the firmware is making the 48 port switch think its a 24 port switch for the purposes of uplink ports.

[jbright]

I had similar problems with MS220 and MS320 switches. Definitely a bug there somewhere.

[ShenzouX]

Ok good glad to hear somebody else had a similar issue - i'll hold off on updating any MS220-48s and MS320-48s.

Maybe we can get some feedback from Meraki on this thread?

[ShenzouX]

Any additional word on this from Meraki support that this is a known issue?  I can't imagine that there aren't a large amount of people that still have MS220 48 ports and MS320 48 ports out there in production.

[RWelch]

You'd likely get faster resolution and feedback on if it's a known issue by opening a support ticket.  It might also give you insight on deciding to roll back to previous stable firmware until such time it can be fixed with a subsequent firmware release.

It's possible you are the 1st to discover it.

[RWelch]

 

The Product Firmware Version Restrictions page indicates MS220s and MS320s can run the latest (current) firmware release.  

[GiacomoS]

Hey awesome people from the Community,

Can you please send me a direct message if you managed to open a Support case about this problem and provide me with the case number? 

I've not spotted any known issue, so it is likely a new problem we'll need to get to the bottom of.

As always, please do make sure you grab the support data bundle before you perform any reboot/reset if possible, so we can get a snapshot of the logs during the issue (and attach it to the case).

Thank you so much for your help everyone.

Giac

[GiacomoS]

Good day folks,

Just to follow up here, I got an example of the issue from @ShenzouX (thank you for your help so far!) and we have reproduced the problem internally. 

We are investigating with our teams to get to the root of the issue. 

If you are encountering this, please do make sure you have a Support case open so we can keep you informed about our investigation. 

Many thanks!

Giac

[ShenzouX]

Any update on this GiacomoS?

[GiacomoS]

Hey @ShenzouX ,

Sorry for the delay; it is still with us at the moment, we can reproduce the problem easily, and there's a lot of data that we are collecting to pinpoint the root cause. We are progressing and hope to have better insights for you by end of this week, beginning of the next.

Many thanks!

Giac

[SNGE-Tech]

We are experiencing the same issue and have a case open as well.

[nlev]

Have you tried the new 17.2.1 version? I noticed "Fixed a bug that prevented MS120-48 switches from correctly forwarding multicast traffic on ports 25-48" in the changelog. Not exactly what you're describing but sounds somewhat similar.

[admine]

on MS320-48 17.2.1 did not fix anything, still no forwarding traffic to uplink.

Cisco just forcing people to buy new equipment by doing this

[RWelch]

@ShenzouX A New MS version is out : MS 17.2.1 

[ShenzouX]

I have MS220-48 and MS320-48 switches that need to be upgraded those are the ones i'm wondering about and I don't see anything about those in the patch notes.  Also looking at the thread for the version it looks like people are having issues upgrading 320s to that version as well.

[SNGE-Tech]

I'm going to test it in our environment in about 5 minutes.

[nlev]

Let me know how it goes. Our MS320 switches are all stuck at:

Firmware
Not running configured version
Upgrade Status
—

[SNGE-Tech]

We have the same result as you.

[nlev]

Thanks for confirming. Glad to hear I'm not alone!

[admine]

Do not upgrade! I lost traffic on MS320-48 25-52 ports.

Thanks god I have only one MS320-48 and all others are Brocade switches so I was able to replug all my desktops into Brocade.

By design it is ridiculous that I can't re-apply any old FW to restore its functionality. MS320-48 is my first and last cloud managed device ever 🤥

[ShenzouX]

Right.

I'm fine with them gating the firmware because its end of life or end of support or whatever but can they at least not have the newest firmware revision that IS going to work on these switches not break half the ports on them?

Doesn't seem like too much to ask for.