Switch firmware 12.14 being pushed out and 12.16 released

[cmr]

A busy day, as another poster noticed, switches on 11.31 are being pushed to 12.14 in 3 weeks or so as it is now a stable release candidate.  However 12.16 has been released and below is a combined fix list for 12.15.and 12.16, I think we might want to go straight to 12.16!

12.16 Bug fixes

• Filtered multicast streams cannot forward on LAG interface on MS120/125 series switches

12.15 Bug fixes

• MS390 LAG configuration can fail when selecting module ports and module is not installed
• Device local status page missing details about LAN IPv6 uplink
• Cable test results on MS210/225/250 series switches inconsistent on 100mbps links
• Routing to a particular next-hop can fail under specific conditions for non-MS390 switch stacks

[Nick]

Yes there is a few nasty issues in there... 

[PhilipDAth]

I had a nasty experience that cost me 4.5 hours with 12.14.  The existing network was already running 12.14.  A switch (an MS225) was newly added to the network.

The switch showed as being online, but failed to take any config changes (the time in the dashboard would keep incrementing).  On the local status page it reported that communication to the Meraki Dashboard on tcp/7430 was being blocked.  The thing is - this was an additional switch being added to an existing network that already had many Meraki switches.

Regardless, I spent much time going through the firewall, taking packet captures, etc.  Tried factory reset.

Eventually called support.  Actually I tried many times.  But the DTMF tones were not working, and I could not key in the customer number or case number, and there is no way to speak to a human without doing this.

Eventually, I had the idea to make an international phone call and try support in another country - which worked.  So it was just the support phone number in my country that was broken.

Support found the version of firmware shipped on the new switch was unable to upgrade directly to 12.14, and the switch had gone into an infinite upgrade loop which was preventing it from taking new config (but allowed it to show as online in the dashboard).  The switch was communicating fine on tcp/7430.

Support had to pin the switch to an interim firmware version and let it upgrade to that, and then remove the pin to allow it to do the normal upgrade to get it working properly.

[Nick]

Now that is painful.....

At that point I am sure you wanted to give up and become a carpenter 😁

[cmr]

Well, this evening I upgraded a stack of three MS210s that have L3 routing for about 10 VLANs.  After going from 12.12 to 12.16, only 2-3 of the VLANs routed outside of the switch stack through the MX HA pair and over the AutoVPN to other sites.  All nodes could be pinged from the dashboard but not from beyond.  The APs also reported DNS and other errors, showing 50% orange.  I left it for a couple of hours to see if it sorted itself out, rebooted the MXs and in the end downgraded to 12.12.  Everything sprang back to life, so be careful with 12.16, on L3 210 stacks anyway, might well work with L2 or non stacked L3. 

Now where did I put that saw and plane?

[Gumby]

I had an interesting experience...

Previous firmware was back in the 10.x due to failures at the last firmware attempt.  Went to 12.14 after many weeks of me cancelling automated update windows.

Started the usual staged rollout, set to deploy to my test switches first... it did the entire network all at once... in the middle of the day...

As expected after the restart the L3 stack of MS250's couldn't decide who was master and randomly decided if it would route or give TTL expired errors.  Called support with their suggestion being a hard reboot of the stack.

Not the first time I've had issues updating this L3 stack and routing post reboot.  End result is I've ditched the stack to save my sanity.

[Nick]

@cmr pass me the hammer....

Now that is distinctly not pleasant. Thanks - i'll make sure we don't fall into 12.16 accidentally!

@Gumby that even worse.... 

[Banfield75 · Accepted solution]

Intresting why they push out the software? I had no issues during my update. I have a stack 210 but no layer 3.

[cmr]

12.17 is out now but the only listed fix is for the MS390 series:

Bug fixes

• MS390 DHCP server may stop working after DHCP fixed assignment or custom option configurations

[sebas]

And still no news about : 

• ARP entry on L3 switch can expire despite still being in use (predates MS 10.x)

Anyone has a clue when this will be sorted ?

[PhilipDAth]

I'm now having regular repeat issues with customers with switch stacks (multiple models) being automatically upgraded to 12.14.  Most of the issues are service disrupting, preventing customers from being able to work.

This is turning out to be a very bad release.

[cmr]

@PhilipDAth the stack that had issues when I upgraded earlier to 12.16 (from 12.12), was okay when I upgraded to 12.17 last weekend.  It is a stack of three 48 port 210s with 15 routed interfaces.

[cmr]

12.18 out now...

[jwwork]

This is annoying that they do this.  When I upgraded from 10.45 to 11.30 (waited until the ARP table stacking issues were resolved) it went very badly.  Basically to the point where we had a total network outage for a day while support tried to figure out what was going on and they pointed me to this in the release notes:

Routing to a particular next-hop can fail under specific conditions for non-MS390 switch stacks

It seems like there were two causes of this issue and 11.30 only fixed one of them.  Now they are pushing out new firmware with what in the release notes?

Routing to a particular next-hop can fail under specific conditions for non-MS390 switch stacks

I'm starting to give up on the idea that these will ever be ready for work in any type of even medium sized organization.

[MarcAEC]

It is concerning how often I see the same items listed in the known issues release after release. 

[cmr]

@MarcAEC  I think that is just confirming the unfixed bugs and quite normal.  However I agree that some to seem to stay around for an awfully long time!

[MarcAEC]

I'm a programmer.  When there are unfixed bugs in my projects, I fix them.

[cmr]

Well 12.22 is out now (obviously 12.20 and 12.21 didn't make the cut) but the bugs remain unless you have a 390 where some more are fixed:

Bug fixes

• Traffic handled by a static route can fail to be routed properly on non-MS390 switch stacks
• MS390 firmware upgrade progress can reflect as 0% on local status page despite progressing normally

Also added is that the MS390 upgrade process will now take a more Meraki time as opposed to an IOS-XE time, although this is from 12.10 onwards... 😎

[mcvosi]

This thread makes me never want to upgrade any of my stacks. I'm new to Meraki (old school Cisco here). I would never want to leave some sort of vulnerability open though. My switches are currently on 11.31, and haven't seen any real issues that force me to upgrade. I just see that "Update available" message and think I should.