Firmware 9.32

[Adam]

Looks like we are going to have a strongly encouraged stable firmware push soon.

Upcoming scheduled maintenance

The new firmware includes support for the following features:

• Support for MS225/MS250 platforms
• Support for URL Redirect and Walled Garden
• Support for PIM-Sparse mode
• Support for Multi-Domain Authentication
• Support for specifying a range of ports in a QoS rule
• Various performance and stability improvements

Your switches will continue to serve clients while upgrading until the minute or so it takes to boot into the new version. If you need to reschedule the upgrade, you may do so by visiting the network-wide settings page.

For more information, please refer to the Cisco Meraki Cloud Management Manual. If you have any questions, please contact Cisco Meraki Technical Support at support@meraki.com or +1 415 632 5994.

IMPORTANT - SECURITY UPDATE

The MS 9.32 firmware contains a security update that addresses an issue identified in CVEs 2017-13704 and 2017-14491 to 2017-14496.

You can choose to self-upgrade at an alternate time.

Deferring/cancelling this update may present a security risk and is strongly not recommended.

[PhilipDAth]

9.32 is already declared as a "stable" release.

I've been using it standard for clients for the last couple of weeks now.  I have only discovered one bug related to using the DHCP server blocking feature when using non-Meraki DHCP forwarders between VLANs.  This feature is off by default, so 99.99% of users wont experience the issue.

[Adam]

Many of my sites were at 9.30. We are starting the process of converting them to 9.32 gradually. Any other issues anyone has experienced?

[mac]

I've had to rollback to MS 9.27 from MS 9.32 on an MS220-8P today - after the firmware update the device experiences 2-5 minutes of normal operation and then locks up, resetting interfaces and getting stuck with an orange indicator LED. Rollback immediately restores operations.

The logs for the MS only show stp events so it's difficult to diagnose. I'll try to upgrade again tonight and see if it was a glitch in the upgrade process, or if it's something that is reproducible.

[Adam]

Had another site that had the issue when going from MS 9.27 → MS 9.32.  But this particular site/switch has had the issue for every firmware update I can remember.  This is the third site that has experienced the issue out of about 20 so far. 

[PhilipDAth]

I have a sick feeling in my tummy.

I have a site running 13 switches.  MS225's and MS425's.  Everything is running 9.32.  I had a massive outage.  Simultaneously, all 13 switches stopped forwarding packets.  Workstations could not even ping other workstations in the same VLAN on the same switch.

I am 99.999% confident it is related to spanning tree.  There are a small number of old Cisco Enterprise switches plugged in at this site.  The network has a lots of loops for redundancy (on purpose).

What really got me is the switches wouldn't even forward local traffic any more.

I eventually managed to get everything going by deliberately shutting down one MS425 core switch (which removes the redundant loops) and giving everything a restart.

This is the first time I have ever had an issue with 9.32.  But it is also also the first time I had a network also using very old Cisco Enterprise switches with a lot of deliberate loops.

I have re-configured the Cisco Enterprise switches to use mst instead of their proprietary RSTP+.  In my case, I believe this action will mitigate the issue I am having (will be testing it hopefully in the next week by power on the redundant MS425).

So in short, I strongly discourage the use of Cisco RSTP+ on the Cisco Enterprise switches when connecting with Cisco Meraki switches.  I think I'll write a bigger article next month about it.

[mac]

I don't think my issue was spanning-tree related (I mentioned it because the logs v/ the website only show stp events on port down/up). The symptoms were identical though- the switch wasn't forwarding any local traffic. I know the interfaces reset as the clients on the network attempted to renew DHCP leases and failed. Bouncing the MS220 brought the network back up temporarily as mentioned.

I had the issue occur in a home-office environment with a flat network with the MS220 being the only switch installed - single VLAN, no loops, all ports configured as access ports including one to an AIR-CAP3702i in Autonomous mode.

[Adam]

Mine definitely isn't spanning tree related.  I've had the issue in one of my networks that had a single Meraki switch with a single MX.  No other switches.  

[ww]

@PhilipDAth did you have vlan 1 on all switch-switch links?

[PhilipDAth]

 No.  Some switches use a different native VLAN between them - but it matches on both sides.

[ww]

Did you verify the stp topology especially at the switches connecting to the cisco switches . I think i have seen both switches being root when not adding vlan1 to your trunks when connection meraki <> cisco. 

https://documentation.meraki.com/MS/Deployment_Guides/Advanced_MS_Setup_Guide

PVST/PVST+

This is a Cisco proprietary protocol on Catalyst/Nexus switches that is compatible with Spanning tree (802.1D). It is important to note however that because PVST/PVST+ is a multi-VLAN spanning tree protocol, in order for the MS series switches to participate in spanning tree a spanning tree instance must be running on VLAN 1 of all switches and VLAN 1 is allowed on all trunk ports running PVST+ so that BPDUs are seen by the Meraki switches in the topology. Connecting an MS series to an existing switch fabric running PVST+ will force the MS series switch(es) to run in legacy mode (STP) which can increase convergence time.

Rapid-PVST

This is a Cisco proprietary protocol on Catalyst/Nexus switches that is compatible with Spanning tree (802.1D) and RSTP (802.1w). It is important to note however that because Rapid-PVST is a multi-VLAN spanning tree protocol, in order for the MS series switches to participate in spanning tree a spanning tree instance must be running on VLAN 1 of all switches and VLAN 1 is allowed on all trunk ports running Rapid-PVST so that BPDUs are seen by the Meraki switches in the topology.

also some good info on connection to a cisco switch where mstp is recommended over pvst 

https://www.ifconfig.it/hugo/post/pvst-interoperability/

[PhilipDAth]

@ww when all the Meraki switches go offline simultaneously and stop forwarding packets and stop checking into the cloud it is very hard to verify the spanning tree topology.

You don't mention it in your links, but I personally recommend using MST on Cisco Enterprise switches connecting to Meraki switches.  This is because you can configure a single instance of spanning tree, which results in RSTP being used under the hood and using the same calculation as the Meraki switches.

The problem with RSTP+/PVST/PVST+ is they can tunnel their packets over a Meraki switch (invisibly to the Meraki switch) and form a completely different spanning tree for those VLANs.  Dangerous.

[lmhost]

Anyone else experienced some outages? After Update to MS9.32 
I had  massive Problems with my Meraki 425 Stacks in the past and still have some.
So in the past switches have decided to not Forward Traffic anymore during Production times.
It has ended up with freezing the Firmware Version to one More stable one. Actually we run now 9.16 stable.
However if we plug out one switch of the stack on MS 9.16 MS425 Core Cross Etherchannels stop Work. So we are not happy at all.
We have one Nutanix infrastrucktur attached to the core which goes compleatly mad if one of the core switch members would go down.

So i have still a bad feeling to upgrade to MS 9.32 because i may experience even more Problems as before.
Support told me that freeze the version to 9.16 is not one option so they force us to upgrade to the new Version.
In staged Upgrade Process it is still not Possible to say that i want to upgrade one of the acces switches to the latest beta for test purposes You can only upgrade the Whole Network instead of only 1 Switch. In other Words if You select Beta in Your main schedule that will affect also the staged ones.
I hope that this Version will be really stable and i will not experienc outages after the Upgrade.
Maybe somebody can also make some practice out of life recomendation to Upgrade the 425 Stacks as Painless as Possible.
Maybee shut down 3 out of 4 and do the upgrade only on one and switch after that 3 online again ? Or somebode have one better idea?

[ww]

If you want to test firmware on a single switch in your network you should contact meraki support. I would have someone standby on the customer site when upgrading any stack and do this after production hours so you be able to hard reset/reboot a switch. There are so many fixes in the newer software relating to stacking and channels, you do not want to stay on 9.16.

[PhilipDAth]

@lmhost I have also had massive problems.  I am 99.99% confident it is related to a spanning tree bug in the Meraki switches when interacting with non-Meraki switches and their are deliberate loops in the network (which spanning tree should knock out).

Do you by chance have other devices in your network that speak spanning tree?  Other switches or routers?

[VictorC]

If interoperating with PVST, make sure you check the following:

PVST - No compatibility with STP/RSTP

PVST+ - Compatibility with STP, make sure VLAN 1 is allowed on the non-Meraki switch

RPVST+ - Compatibility with RSTP, make sure VLAN 1 is allowed on the non-Meraki switch

[PhilipDAth]

Hi @VictorC.  That really provides a false impression of comfort - as it incorrectly makes people think it will work.  It will only work in the simplest of cases.

If you have this:

<cisco enterprise switch> -- <meraki switch> -- <cisco enterprise switch>

And you make the Meraki switch the spanning tree root, and if you are using PVST+ or RPVST+ then Cisco enterprise switches will correctly converge only for VLAN1.  However for every other VLAN they will tunnel the spanning tree packets through the Meraki switch (so the Meraki switch does not participate) and will elect one of themselves as the root for those spanning tree instances.

If you add in an extra loop, then you can quickly get to the case of having spanning tree broken because two difference systems have not calculated the same loop free paths.

It really is only safe to use MST on the Cisco Enterprise switches with a single instance.

[VictorC]

@PhilipDAth Thanks for providing more context.

We definitely recommend using MST as that provides full backwards compatibility. 

[lmhost]

Hi All

Actually the network outages we have experienced especially with MS425 Stacks are not related to Other Vendors Equipements STP whatever. In our case we can Plug out one of the core Stack member and we will loose connections an all Cross switch Etherchannels.

The network is one Meraki only Network.

[ww]

@lmhost

Your firmware is way behind, I have seen lots of improvement in stability of stacks and channels in later firmware.

[PhilipDAth]

9.32 is not way behind.  It is the current stable train for Meraki switches.

[ww]

@PhilipDAth he is running 9.16.

[Adam]

Out of about 20 of my networks, 5 of them had the issue where a switch needed to be manually rebooted to get it to be un stuck.

[BrothersTM]

The firmware issues is definitely not resolved.  On this round of updates to my MS320 switches from 9.30 to 9.32, all but 5 of my switches had to be manually rebooted post deployment in order for the firmware to update.  I thought last Friday's snow day was a great day to update all the switches.  I was very wrong, since that required me to come in anyways and go closet t closet pulling power to switches until they were all updated and back online.

[VictorC]

Thank you to everyone for being so active in the Meraki Community.  

We have identified the issue which has been causing MS22, MS42, MS220 and MS320 switches to require a physical reboot during an upgrade.  The issue is present on switches currently running MS 9.0 and above.  We hope to rectify this issue and provide a fix shortly. 

In the interim, we have discovered a reboot of the switch via dashboard prior to performing an upgrade will prevent the switch from becoming unreachable.

As always, if you have any further questions/concern, please do not hesitate to reach out to our Cisco Meraki Support Team.

[mac]

Hi Victor,

Why would Meraki leave this version available with a bug like this present? I wouldn't count this as acceptable even on a product where upgrades were 100% manual, having this on a product that automatically upgrades is reckless at best.

These are production appliances that Cisco is heavily pushing and I'd expect better from a product line with a lease/subscription-style cost model.

[PhilipDAth]

Well lets just say the prior versions had a lot of defects, and this is the lessor of two evils.

[Kujoh]

So how do I do a self-upgrade at an alternate time? I want to be around when this happens, not automatically in the wee hours of the morning.

[PhilipDAth]

Organization/Firmware upgrades - and then schedule a time.