New MS 16.3 beta firmware - stack routing fix (which I need) and more!

[cmr]

Switch firmware versions MS 16.3 changelog

Alerts

• MS420 devices configured for this version will run MS 15.22

General features

• ACL hit counter
• DHCPv6 and IPv6 RA guard
• Local Status Page Packet Capture

Ms12x features

• DAI support

General fixes

• Having SecurePort enabled causes a large influx of event logs for each port supporting SecurePort resulting in events to be dropped (present since MS 16)
• SecurePort may fail to successfully authenticate an MR for multiple minutes (always present)

Ms12x fixes

• Storm control significantly degrades the network by blocking valid traffic (present since MS 16.2)

Ms210/225/250 fixes

• 100Mbps devices fail to re-establish a link after a cable test is run (present since MS 14)

Ms2xx/35x/4xx fixes

• Incorrect route information may be used for ECMP resulting in packet loss for some destinations (always present)
• Stacks that have routes to virtual IPs may reach a state where the next hop port is incorrect (present since MS 14)

Ms425 fixes

• Large number of SNMP walks to switches in H.A. may destabilize the VRRP connection resulting in flapping (present since MS 15)

General known issues

• Connecting a stacking cable to a stack that is online may result in a stack member going offline (present since MS 12)
• Critical Authentication may incorrectly keep RADIUS server status as unreachable after a failed connection resulting in all subsequent authentication attempts failing (always present)
• Cross-stack LACP groups may fail to fully initialize, resulting in one or more ports staying offline (present since MS 16.1)
• In rare instances, the port status will fail to update to dashboard until the system reboots (present since MS 14)
• Switches move LACP ports to an active forwarding state if configured. This can cause loops when connecting to an MS390 or other Catalyst switches unless the bundles are configured on the MS390/Catalyst switches first. All non-catalyst ports are configured in passive LACP mode so that loops do not occur between Meraki switches (always present)

Ms120 known issues

• Switches in extremely rare instances will experience reboots every few minutes (present since MS 11)
• Switches may fail to provide PoE power to legacy access points (always present)

Ms125 known issues

• The local status page cannot be accessed from the management port (always present)

Ms12x known issues

• Ports with an odd-numbered MTU value fail to initialize (predates MS 11)

Ms210 known issues

• Stack ports have an orange LED instead of green (present since MS 11)

Ms2xx/35x/4xx known issues

• IPv6 multicast packets tagged for a VLAN may not be filtered correctly resulting in packets egressing into other VLANs (present since MS 14)
• IPv6 solicited-node packets bypass port isolation (present since MS 14)
• Rebooting a stack member may result in internal stacking packets to be sent out into the LAN (present since MS 15)
• SVI default route can be incorrectly removed from the routing table after L3 changes are made on dashboard. Removing and re-adding the route is required to resync the stack (present since MS 15)
• Stack routing tables can get out of sync when L3 changes are made. Making an additional change to L3 can resolve the sync issue (present since MS 15)
• Switch stacks will learn MAC addresses from ports in the STP blocking state which can trigger a constant flood of MAC flaps in the event log

Ms355 known issues

• In rare instances, stack ports fail to initialize after an upgrade (always present)

Ms35x known issues

• Enabling Combined Power results in events being logged once per minute (present since MS 11)
• UPoE does not negotiate over LLDP correctly (always present)
• mGig switches will have an amber light for all physical ports that do not negotiate to the highest supported speed. Dashboard will continue showing a light green status for all ports above 100Mbps. For example, MS355 switch ports will incorrectly show an amber light for 1G, 2.5G, and 5G, but will show a green light for 10G.

Ms425 known issues

• 1Gbps ports may randomly fail and go down until the system is rebooted (present since MS 16.1)
• Stacks in rare instances will start dropping DHCP traffic on trusted ports while DAI is enabled until rebooted (present since MS 12)

Ms4xx known issues

• When an SFP module is inserted/removed, BPDUs can be delayed leading to STP transitions in the network (predates MS 12)

[RaphaelL]

If 

• SecurePort may fail to successfully authenticate an MR for multiple minutes (always present)

Why wasn't it included in MS15.22 . Don't get it

[cmr]

@RaphaelL it is called the UEF, or upgrade encouragement factor 😉

[RaphaelL]

haha ! Had never heard of that one yet 😂

[Kyle_Smith]

Good question! As MS 15 is currently in its General Availability (GA) phase, it is crucial to exercise caution when considering additions for subsequent GA releases. The utmost priority is to avoid introducing any changes that might inadvertently destabilize the system.

In this particular case, the severity of the issue appears to be relatively low, as the devices eventually authenticate despite the challenge. Therefore, the risk versus reward assessment leans towards a more cautious approach.

It is worth noting that the most recent beta/rc (release candidate) releases tend to include a higher number of fixes. This is due to the slightly higher tolerance for potential bugs that may emerge while attempting to address existing issues.

Rest assured, our team is committed to maintaining a robust and stable product, and we appreciate your understanding and support as we strive to continuously enhance its performance.

[RaphaelL]

Wow ! Thanks a lot for that explaination. Let's say we are still on MS14.33 which is rock solid for us and that MS15 is in a GA phase , should we wait to jump to MS16 instead ? We have around 4K MS which we have to pilot and test a LOT of things and features. Trying to avoid to deploy MS15 and having to re-do all the testing for MS16 which sounds like is coming to GA late 2023.

[Kyle_Smith]

I understand your concern about the potential impact on your organization's testing and deployment efforts. It's important to consider the specific needs and priorities of your organization when making decisions about software updates.

While MS 14.33 may be rock solid for you at the moment, it's worth noting that newer releases include important security patches and bug fixes. Waiting until MS 16 becomes GA might mean missing out on these critical updates for a longer period of time.

One approach you could consider is to evaluate the potential impact of upgrading to MS 15.22 by initiating testing of this version to simulate your organization's specific use cases and scenarios. This will help you determine whether the benefits of the new release outweigh the potential challenges of retesting and redeploying. While this may initially seem like a duplication of effort vs waiting for MS 16, it's important to recognize that testing MS 16 will build upon the foundation laid during the testing of MS 15. By thoroughly testing MS 15, you gain valuable insights into the potential impact of future updates, such as MS 16. This allows you to proactively address any issues, reducing the overall effort required when transitioning to MS 16.

Ultimately, the decision to upgrade depends on the specific needs and risk tolerance of your organization. Taking a cautious and well-planned approach to testing and deployment can help minimize disruption and ensure a successful transition to a newer release.