MS 14.33 Firmware Is Out

[Mloraditch]

Absolute Bevy of Fixes, including I believe Voice Vlan/802.1x and layer 3 changes needing a reboot.

Alerts

• MS390 upgrades to this version will experience a full system reload

Branch additions

• Syslog support for MS390 series switches
• SNMP support for MS390s
• RADIUS accounting support for MS390s
• Alternate Management Interface support for MS210/MS225/MS250/MS350/MS355/MS410/MS425/MS450 series switches
• QoS support for MS390s
• CoA support for MS390s
• STP anomaly detection events for non-MS390 series switches
• Adaptive policy support for MS390 series switches
• SecureConnect support for MS210/MS225/MS250/MS350/MS355/MS410/MS425/MS450 series switches
• NAC enhancements for MS120/125/210/225/250/350/355/425/450 series switches
• Critical/failed auth VLAN support for MS120/125/210/225/250/350/355/425/450 series switches

Bug fixes

• The Voice VLAN fails to stay authenticated if it is authenticated before the Data VLAN (present since 14.28)
• Client MAC addresses may fail to be learned on a port right after an access policy is applied (present since 14.28)
• SVI changes made to stacks may result in incorrect route mappings resulting in packet loss (present since MS 14.32)
• MS390 management plane can experience frequent restarts (present since MS 14.27)
• Voice VLAN bypass configuration was not being applied correctly to MS390 series switches
• Clients fail to be placed into the Critical VLAN if the re-authentication timer is configured
• In rare instances, DAI may fail to snoop DHCP transactions on stacks leading to those clients being in a blocked state
• MS390 series switches do not support forced speeds of 2.5 or 5Gbps
• Ports that are moved into a Critical Authentication state fail to be removed once the RADIUS server becomes reachable again without a reboot
• Switches in an IPv6 only network will experience reboots
• MS120 series switches may experience instability when multiple ports are modified which are configured with access policies
• MS250 series switches in rare instances could experience unexpected reboots
• SNMP IfIndex integer for MS390 series switches would start at 9 instead of 1
• ACLs are not removed on MS390s when unconfigured on dashboard
• MS120s on rare occasions will reboot (present since MS 11)
• Stack members may go offline and require a reboot to regain connectivity (present since MS 12)
• MS390 management plane can get stuck in a crash loop resulting in the switch staying disconnected from dashboard

Known issues

• MS390 switches may experience a full system restart (present since MS 14.29)
• MS390 management plane can restart which may show as brief loss of connectivity in dashboard (present since MS 12.28)
• MS390 stack members may experience a full system restart (present since MS 14.29)
• When the management plane restarts on MS390s, LACP flapping can be encountered
• Stacks with a large number of LACP bundles (10+) may fail to initialize stack ports properly requiring additional reboots to regain connectivity
• MS390s do not respond to broadcast DHCP Offers for their management IP addresses (present since MS 14.31)
• MS390 stacks may send frequent DHCP requests despite having a valid static IP
• In rare instances, the MS390 management plane will fail to restart requiring a manual reboot (present since MS 14.26)
• MA-SFP-1GB-TX modules may fail to be detected after link loss (always present)
• CDP protobuf data reports null port module ID
• Stack members may go offline after a stacking cable is reconnected requiring a power cycle (present since MS 12)
• MS120 in rare instances will not be able to perform packet captures until rebooted (predates MS 12.28)
• Packet loss is observed when pinging the MS390 management IP
• MS210/250 series switches may fail to relearn client MACs on ports that have access policies configured (present since MS 12)
• Links being established on an MS120 can result in neighboring ports to flap (present since MS 11)
• In very rare instances, MS120 series switches will reboot every 10 min (present since MS 12)
• MS390 - Port Up/Down Events Shown Across All Members
• Enabling Combined Power on MS350/355 switches results in events being logged once per minute (present since MS 11)
• Networks containing a large number of switches may encounter issues saving changes on the Switch Settings page
• Stack members are not being marked to update their configuration when changes are made on other members
• mGig switches will have an amber light for all physical ports that do not negotiate to the highest supported speed. Dashboard will continue showing a light green status for all ports above 100Mbps. Example, MS355 switch ports will incorrectly show an amber light for 1G, 2.5G, and 5G, but will show a green light for 10G.
• The list of switches to clone from fails to load when cloning a switch in an organization with a large number of switches and networks
• Cable test tool returns wrong pairs for 100Mbps connections (predates MS 12)
• MS390 stack management plane may experience recurrent loss of dashboard connectivity (present since MS 14)
• Broadcast types of traffic can leak into the Guest VLAN if a port that fails authentication has a Voice VLAN configured, and dashboard has a Guest VLAN defined (present since MS 11)
• MS120s switch ports with MAB authentication may randomly deauthenticate clients. In order to resume client authentication on that port, a switch reboot is required (present since MS 12)
• MS390 series switches do not support SM sentry
• MS390 series switches do not support loop detection
• MS390 series switches do not support warm spare/VRRP
• MS350-24X and MS355 series switches do not negotiate UPoE over LLDP correctly (predates MS 10)
• Ports with an odd-numbered MTU value fail to initialize for MS120/125 series switches (predates MS 11)
• When an SFP module is inserted/removed on MS420/425 series switches, BPDUs can be delayed leading to STP transitions in the network (predates MS 12)

[PhilipDAth]

>SVI changes made to stacks may result in incorrect route mappings resulting in packet loss (present since MS 14.32)

I got bit by this one today.  I deleted a VLAN on an MS225 and had to reboot the switch to get routing working properly again.

[KarstenI]

I am hitting this one again and again with most customers. Some will move away from Meraki because every route-change needs a planned maintenance window.

[GIdenJoe]

I've had two customers with recurring occurences of this bug.  One of those is a big one (for my scale) and he wasn't happy with the 'unstable behavior' of his MS425's.

[KarstenI]

Interesting to see that it is not just the 35X series as I thought before. I only had this on lots of these devices and not on 2XX and 425 switches. 

[Jackson_J]

Hey guys, this is a little disheartening and scary. My environment is currently on MS 12.14 and Meraki auto-scheduled an upgrade to 14.33. I was initially going to hold off, but now I'm planning to upgrade during our next maintenance window because we're starting to see a weird issue (again) where our Thin Clients that are connected to an MS350 Switch Stack running Layer 3, get randomly disconnected from their remote VM sessions without any rhyme or reason. (When users connect to these same VMs from home, this doesn't happen). This used to happen to us on MS v11 (never occurred when we were on MS v10) and I thought MS 12.14 would solve it (upgraded during the pandemic), but I guess we didn't see the issue resurface since very few people were actually in the office during the height of the pandemic. Now after seeing these posts and also seeing some of the release notes for the 15 code with "Known Issues" similar to this is starting to make me worry. I'm not sure if this is the same issue that I'm facing, since I haven't made any recent changes to any SVIs, but it sounds similar to what we're experiencing.

The issue that we're actually facing is closer aligned to this: https://community.meraki.com/t5/Switching/RDP-Brief-Disconnects/m-p/113785/thread-id/8228

The question is, what do we do??? These kind of routing/disconnect issues seem to be pretty longstanding and have been around for much longer than Cisco Meraki should even admit without feeling ashamed. I don't even know how this product could be greenlit for production at this point with all of this instability. I'm starting to be afraid to even run these Switches in our environment right now and we've got quite a few sites running these Switch Stacks in production (still have to survey the other sites, as we begin to return to office). I don't know which Firmware version to go to...I'm just at a loss right now.

[Make_IT_Simple]

This was a bad one. Meraki had multiple versions of this issue and I worked closely with Support in our LAB to help them gather data. It was affecting MS425 stack and stand-alone. They even had me test with a custom build to confirm their fix was working. At least in our environment, 14.33 did fix these abnormalities including our MS390 control plane issues which drove me crazy. 

[Jackson_J]

Hmmm...and this is coming from a Meraki employee...so what I'm hearing from you is to probably Not upgrade to 14.33 in a couple of weeks?? Is that correct @Make_IT_Simple ? Is there any Firmware version that is the most stable that you guys would recommend? I appreciate your response in advance. Thanks

Sorry about that @Make_IT_Simple , I misread your post. It looks like you're saying that in your experience, 14.33 did resolve these issues in your environment, is that correct?

My apologies for misunderstanding, if that's the case.

[TFreer]

Does anyone with 390's and L3 on it install this yet? and is it an improvement from 14.32?