Interesting double firmware release, MX15.42 for current devices and MX14.55 for old models

[cmr]

MX15 looks like it is about to become a release firmware as MX14.x version MX14.55 is ONLY for older models such as the MX90, se below.  No major fixes, just stability reliability improvements, already being pushed out to replace our MXs running 15.41 despite only being released today:

Security appliance firmware versions MX 15.42 changelog

Important notice

• This is a late-stage beta version for the next major MX release. While we believe this release is able to provide a quality experience in most circumstances, we recommend taking additional caution before upgrading critical systems. Where applicable, MX 14 releases will provide a more stable upgrade alternative.

Legacy products notice

• When configured for this version, Z1, MX60, MX60W, MX80, and MX90 devices will run MX 14.55.

Bug fixes

• Stability improvements to Z3(C) appliances.

Known issues

• After making some configuration changes on MX84 appliances, a brief period of packet loss may occur. This will affect all MX84 appliances on all MX firmware versions.
• Some stability-impacting issues present in MX 14.19 that affect a small population of MX250 and MX450 devices still exist.
• Some stability-impacting issues present in MX 14 that affect a small population of MX67(C,W) and MX68(W,CW) appliances still exist.
• Some stability-impacting issues present in MX 14 that affect a small population of Z3(C) appliances still exist.
• Please note that until certification has been obtained, the Z3C will not be supported on Verizon's network.
• World-wide device SKUs of the MX67C, MX68CW, and Z3C units cannot be deployed in North America and North America device SKUs of the MX67C, MX68CW, and Z3C units cannot be deployed outside of North America.
• When deployed in warm spare / high availability (HA), MX67C and MX68CW do not support using their cellular connectivity to pass client traffic. In this deployment, the cellular connectivity can only be used for device monitoring or network troubleshooting. This is an expected limitation for these platforms.
• MX67C, MX68CW, and Z3C units must be connected to the Meraki Dashboard initially to retrieve an update to allow for proper use of the integrated cellular connectivity. This is most likely to be an issue when bringing the units online for the very first time.
• On the MX67(C,W) and MX68(W,CW) platforms, when the MX is providing PoE to a connected device, this information will not be reflected on the Meraki Dashboard.
• Once a Z3 has been updated to this firmware version it can only run MX 14.31 or MX15.8 and higher. This is an expected result of updates to the device booting mechanisms and this limitation will not be resolved in future releases.
• The DES encryption algorithm is no longer supported for use in formation of VPN tunnels.
• Creating VPN tunnels using aggressive mode IKE is no longer supported.
• Due to MX 15 regressions, USB cellular connectivity may be less reliable on some modems.
• Due to an MX 15 regression, the management port on MX84 appliances does not provide access to the local status page.
• Client traffic will be dropped by MX65(W), MX67(C,W), and MX68(W,CW) appliances if 1) The client is connected to a LAN port with 802.1X authentication enabled and 2) The VLAN ID of the port is configured to 16, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, or 240.
• Due to issues still under investigation, Z3(C) appliances are more likely to experience a device reboot when the wireless functionality is being utilized.

Security appliance firmware versions MX 14.55 changelog

Supported products notice

• This firmware version is only available for Z1, MX60(W), MX80, and MX90 appliances. Any other devices configured to this firmware version will not upgrade.

Bug fixes

• Fixed a rare issue that could result in MX appliances not providing DHCP leases after multiple, quick HA failover events (aka "flapping") had occurred.
• Corrected an additional issue that could result in MX and Z appliances being unable to successfully fetch a configuration from the Meraki Dashboard without Meraki Support’s intervention.
• Corrected an MX 14.32 regression that may have resulted in some Z1 appliances connected to the Internet only using a USB cellular modem to be unable to successfully upgrade to firmware versions newer than 14.31.
• General stability and security improvements

Known issues

• All known issues only relate to current models that this firmware does not support so I have deleted them for clarity

Other

• Added an Event Log entry for when all of the IP addresses designated for the Client VPN subnet have been exhausted.

[CptnCrnch]

I received this later today a few minutes after upgrading to 15.41 🤣

Upgrading from .41 to .42 was flawless though 😉

[PhilipDAth]

The sooner 15.x is the new stable the better.  Then 16.x can become beta, and AnyConnect can become available.

[AnythingHosted]

I noticed this morning that our MXs on 15.42 now states “release candidate”.

[nsingh]

I am specifically posting my experience with Meraki Support and Meraki latest "Stable" release 15.42.1 

I upgraded our "X network" MX box on this sunday at 4.30pm PST.

To start with, it's been a very poor code quality since Meraki released 15.42 and 15.42.1.

Taking some packet captures, revealed that there is connectivity and routing issues between Site-to-Site Meraki peers, so while on one location client VPN, you cannot access the resources of the another location, there are intermittent packet loss.

Since the same day we started experiencing the connectivity issues in our Client VPN. These are the behaviours.

1). I connect to our "X network" client VPN on Meraki. Connects fine. Then within one minute the VPN disconnects automatically with no error message nothing.

2). I connect to our "X network" client VPN. Connects fine. No internet works on the VPN, internal and external. I can see packets going out, but no return traffic.

3). I connect to our "X network" client VPN. Connects fine. Internal traffic does not work, external traffic works.

4). I try to connect to our "X network" client VPN. It gives me Authentication failed for the same exact credentials that are saved in my VPN profile which was previously authenticating without any issues.

I have downgraded the "X network" MX box on 14.53, and now everything works fine. This happened on both 15.42 and 15.42.1.

Here is a sneak peak of other blogs of users facing similar issues - https://www.reddit.com/r/meraki/comments/n5hygl/mx_15421_breaks_routing_somehow/

[cmr]

@nsingh that is odd, I'm guessing it is only related to the client VPN as we don't use that, have had 20+ MXs on 15.42 since it came out and don't have any routing or pocket loss issues on them.

Having said that, we do have one site with a L3 core of a 355 stack (most are Cisco 3850 stacks) and if they run 12.x we get packet loss over the WAN, but if we run 14.x we don't.  I only noticed it when I downgraded it to troubleshoot what turned out to be a couple of knackered printers as we normally use the latest code.

What cores do you have, are they Meraki, or do you do all the routing on the MXs?

[nsingh]

@cmr  - So we have few locations where we are using Meraki Cores, and this particular upgrade was done where we are using MX for routing.

[cmr]

Thanks @nsingh what switches are connected to it and are they a stack, dual connected or single?  Is it a pair of MXs or just one?

[nsingh]

@cmr  - Thanks for your reply. Here is our simple infrastructure.

ISP -> MX250 standalone -> MS350-48 (acting as core) -> MS350-48FP -> APs and Clients

The switches are single and MX is also single at this point.

[cmr]

@nsingh What firmware version are the MS350s running?

The only packet loss issue we saw was when we briefly downgraded a stack of MS355s to 12.x code from the 14.x code we were running.  Both 12.28 and 12.33 had the same problem so we went back to 14.x and are now running 14.21. 

The problem we thought we were seeing turned out to be multiple connected endpoint devices failing in a short period rather than a switch issue...

[nsingh]

@cmr - Ah! maybe that could be it. We are running 12.28 on both switches, however there was no mention of switch OS in the 15.42.1 release document for compatibility.

So you saying the "beta" OS 14.21 is good to upgrade?

[cmr]

Absolutely, we have 5 sites with Meraki switches covering 355s, 225s, 220s, 210s and 120s.  All sites are running 14.x code, some on 14.16 (L2 switches) one on 14.20 (L3 210 stack) and two on 14.21 (L3 355 stack and a single L2 220).

We haven't had any issues with 14.x but we don't use port aggregation and there has been an issue reported on this community regarding that.

Meraki beta code is simply the next major version that hasn't been used by many customers yet.  In the release notes you will sometimes see early stage beta (MX 16.x currently has this) and I wouldn't usually put that on a core/hub until proven on our own other sites/switches

[nsingh]

@cmr  - Yeah, no luck after 14.21 upgrade, and I checked we are not using aggression. 

Well, now my Client VPN disconnects automatically in few seconds.

But thanks for the advise.

[cmr]

No worries and hopefully someone else here will get you sorted 🤞

[CharlesIsWorkin]

@nsingh 

Any changes? We use client vpn here alot and so I don't want to upgrade at all if this is an issue.

[AlexP]

If anyone's confused at why this happened, please refer to the following KB: https://documentation.meraki.com/General_Administration/Firmware_Upgrades/Product_Firmware_Version_R...

Quoting the relevant detail here:

"Z1s, MX60/Ws, MX80s and MX90s cannot upgrade to MX 15 firmware, or any newer major release versions. If networks containing these devices are configured to run newer firmware revisions, they will run the newest MX 14 maintenance release instead."