Security appliance MX 18.211 Release

[TyShawn]

Security appliance firmware versions MX 18.211 changelog

Important notice

• USB modems with MX/Z series devices running firmware MX 18 or newer will be limited to best effort support and will not be receiving any future firmware fixes or improvements.

Bug fixes

• Resolved an MX 18.2 regression that resulted in the WAN2 being unable to pass traffic if 1) WAN1 was not in use and 2) cellular was enabled.
• Fixed inconsistencies with the cellular active uplink feature. WAN 2 cannot be used as a functioning WAN interface when cellular active uplink is enabled.
• Fixed a MX 18.2 regression that resulted in MX75, MX85, MX95, MX105, MX250, and MX450 appliances being unable to successfully establish IPSec VPN connections when NAT-T was required to establish the connection.
• Corrected an MX 18.2 regression that resulted in MX75, MX85, MX95, MX105, MX250, and MX450 appliances failing to form AutoVPN or teleworker tunnels with other peers via their LAN interfaces.
• Resolved an issue that made MX75, MX85, MX95, MX105, MX250, and MX450 appliances more likely to rewrite the source port of traffic being NAT'ed out a WAN interface.
• Fixed an issue that resulted in the VPN status information for non-Meraki VPN peers being shown incorrectly on the VPN status page in Dashboard.
• Fixed a rare issue that could result in the AnyConnect VPN process becoming unresponsive on MX75 and MX85 appliances.
• Resolved an issue that could result in AutoVPN tunnel instability on both MX uplinks when packet loss and intermittent connectivity occurred on one uplink.
• Corrected an issue that could result in Z4C appliances being unable to successfully pass cellular traffic when using a Telstra SIM.
• Fixed an issue that resulted in MX75, MX85, MX95, MX105, MX250, and MX450 appliances tracking information about upstream WAN addresses as if they were local clients if 1:1 or 1:M NAT were configured.
• Resolved an issue that resulted in uplink connectivity tests for IPv6 being routed incorrectly.
• Fixed an issue that could result in an increased level of jitter and latency for AutoVPN traffic on Z3(C) appliances. That would specifically occur during periods of low and infrequent AutoVPN traffic.
• Stability improvements for MX75, MX85, MX95, MX105, MX250, and MX450 appliances.
• Resolved an issue that could result in MX appliances with adaptive policy configured encountering frequent connectivity state changes for AutoVPN tunnels.
• Corrected a MX 18.2 regression that resulted in the SIM and APN configuration being shown on the device local status page for devices without integrated cellular modems.
• Reduced the potential for existing traffic flows to be disrupted from configuration changes on MX75, MX85, MX95, MX105, MX250, and MX450 appliances.
• Resolved a rare issue that could result in the AnyConnect client VPN process crashing.
• Corrected a rare issue that could result in an IPv6 delegated prefix not being visible in Dashboard.
• Fixed a MX 18.2 regression that could result in MX appliances not performing ARP for virtual IP addresses, 1:1 NAT IP addresses, and 1:M NAT IP addresses when 1) the MX was configured in high availability and 2) had WAN1 disconnected or disabled.

Legacy products notice

• When configured for this version, Z1 devices will run MX 14.56.
• When configured for this version, MX400 and MX600 devices will run MX 16.16.9.
• When configured for this version, MX64(W), MX65(W), MX84, MX100, and vMX100 devices will run MX 18.107.10.

Known issues status

• This list is being reviewed and updated.

Known issues

• In rare cases, MX67C, MX68CW, and Z3C appliances may fail to enter into a "Ready" state despite being able to register to a cellular network and obtain an IP address for the modem.
• The Non-Meraki VPN service may fail to properly establish IKEv2 tunnels when the MX appliance is acting as the IKEv2 responder and many allowed subnets are configured.
• Due to an MX 18.2 regression, the link light LED for WAN2 on MX75 appliances will not light up if WAN2 is the only wired interface in use.
• Due to an issue with no known method of reproduction, the IDS and IPS process may unexpectedly restart.
• When a WAN failover occurs, Non-Meraki VPN tunnels will persist on the backup, non-primary uplink after a failback to the primary WAN interface if the WAN interface uses IPv6.
• Due to an issue still under investigation, MX appliances may experience an unexpected reboot when ThreatGrid is enabled.
• MX AutoVPN tunnels fail to generate new connections when the AutoVPN flow has been blocked or filtered unidirectionally by an upstream or intermediary device. This prevents appliances from automatically working around this partially connected state.
• MX appliances may experience unstable eBGP connections when 1) the MX appliance is configured in Routed mode and 2) the MX learns a large number of routes from its eBGP neighbor. This may result in eBGP-learned routes being inaccessible.

Other

• Improved AutoVPN failover times for VPN connections between MX appliances running MX 18 or higher.

[CptnCrnch]

That's a lot of fixes! 😎

Upgrading just now.

Edit: looking good so far!

[cvsmith122]

Do you do any VPN or 1:1 I have rescheduled my update because the last time Meraki did a auto update it broke VPN and 1:1. 

[plumdev]

I updated two networks from 18.208 to 18.211 last night and one of them lost autovpn site to site connectivity completely. Rolled it back this morning.

[cmr]

Still a lot of stability issues, hopefully the fix list won't shrink...

[RaphaelL]

MX18.2.11 was supposed to be the first 'stable' version. Wonder what happened.

[jimmyt234]

It needs to reach at least 15% global network saturation prior to being promoted to stable (Meraki Firmware Release Process - Cisco Meraki Documentation). We've seen at least a dozen customer orgs have an auto-upgrade scheduled for this version overnight, so it seems Meraki will push this one out quite aggressively to reach this limit in order to get it to stable.

[RaphaelL]

Yes you are right. forgot about that part. 

Which also means that MX18.2.11 will be the only stable version.  Other fixes will only be included in patches ( 18.2.11.X )

[jimmyt234]

Yes 😄

18.210 has been pretty solid for us, hoping for the same from .211!

[FRover]

We still have a large number of devices that are 4G only on 16.16.9 due to mobile connectivity issues on 17.xxx 

Is that resolved here or does it fall under this known issue?

• In rare cases, MX67C, MX68CW, and Z3C appliances may fail to enter into a "Ready" state despite being able to register to a cellular network and obtain an IP address for the modem.

[DMD]

I updated one of my test sites that had this similar issue on 18x running the 4G SIMs.  Are you using static or dynamic APNs?  I updated mine to 18.211 using a static APN and associated with the proper location and worked perfect.  the 18.210 broke this 

·         West: WE01.VZWSTATIC

·         Midwest: MW01.VZWSTATIC

·         Northeast: NE01.VZWSTATIC

·         South: SO01.VZWSTATIC

[jbright]

I had a customer that had to rollback to 18.210 this morning because 18.211 broke site-to-site AutoVPN.

He told me Meraki TAC told him that they have a number of customers experiencing issues with this version today.

I installed it on all of my company's MX and it is working fine for us and we have one AutoVPN connection that is also working

fine. My advice is to test it on non-production networks first to see if your environment is impacted.

[cmr]

I've installed it on an MX68 that is acting as an AutoVPN spoke and it seems to work okay, being stable and performing normally.  It is only running the enterprise feature set, so issues might be related to advanced licensing?

[cockroach_c]

It looks like MX86 isn't part of the issues with AutoVPN. MX75, 85, 95, 105,250, and 450 are the ones having issues. 

[BHC_RESORTS]

We YOLO'd this update to our "test" site which is our corp office. MX95. So far so good...we got burned hard on the last "stable" RC so we're taking a bit of a gamble here but I liked the fix list.

[TyShawn]

For me, SNMP stopped working with this release. 

[Lauzon_C]

What version of firmware were you using prior? I heard mention of possible snmp issues here. We are on 18.210.

[Lauzon_C]

Had to roll back to 18.210, had vpn issues on the site I tested it on.

[TyShawn]

I was on 18.210 and many other older versions. 18.211 is the version that broke SNMP for me. I have a ticket opened up on this version. 

[bdeen]

I have a scheduled upgrade for 4 sites for this weekend, are there any known issues with this present release MX 18.211 apart the one mentioned above. Thanks 

[cmr]

@bdeen from the release notes the below are all known issues, hopefully there aren't too many more:

• In rare cases, MX67C, MX68CW, and Z3C appliances may fail to enter into a "Ready" state despite being able to register to a cellular network and obtain an IP address for the modem.
• The Non-Meraki VPN service may fail to properly establish IKEv2 tunnels when the MX appliance is acting as the IKEv2 responder and many allowed subnets are configured.
• Due to an MX 18.2 regression, the link light LED for WAN2 on MX75 appliances will not light up if WAN2 is the only wired interface in use.
• Due to an issue with no known method of reproduction, the IDS and IPS process may unexpectedly restart.
• When a WAN failover occurs, Non-Meraki VPN tunnels will persist on the backup, non-primary uplink after a failback to the primary WAN interface if the WAN interface uses IPv6.
• Due to an issue still under investigation, MX appliances may experience an unexpected reboot when ThreatGrid is enabled.
• MX AutoVPN tunnels fail to generate new connections when the AutoVPN flow has been blocked or filtered unidirectionally by an upstream or intermediary device. This prevents appliances from automatically working around this partially connected state.
• MX appliances may experience unstable eBGP connections when 1) the MX appliance is configured in Routed mode and 2) the MX learns a large number of routes from its eBGP neighbor. This may result in eBGP-learned routes being inaccessible.

[scc_sysadmin]

Ever since the upgrade to this version, all our Meraki switches' DNS entries were changed to 1.1.1.1 and 8.8.8.8. Causing intermittent disconnections across our switches now. Anyone have similar issues or how can I revert back to the older version?

[ESMichal]

After the upgrade to 18.211 large portion of our clients were not able to use internet because of DNS problems on the MX gateway: Client made a request to the DNS server, but it did not respond (no_dns_response).

We use DNS passthrough option: Client -> MX Gateway -> Google DNS. There are two gateway in hot-standby mode

We rolled back to 18.208 and the DNS issue seems to be gone.

Edit: We do have one high priority traffic shaping rule enabled.

[jimmyt234]

Release notes have been updated with:

Known issues - may 19th update

• Due to an MX 18.211 regression, networks that have traffic shaping rules configured with a "high" priority may incorrectly drop traffic being routed between VLANs or AutoVPN.

[cmr]

Wow @jimmyt234 that is a bit of a major bug!  Glad we'd only updated a home user Z3 so far that doesn't have traffic priority rules set 😬

[cockroach_c]

I have my downgrade scheduled in the am so I will follow up and let everyone know if it worked, but we started experiencing what seemed like intermittent route drops in inter vlan routing. We have been ripping out hair out ever it. The client does have traffic shaping going on. I will find out if rolling back worked in the morning.

[cockroach_c]

UPDATE: I tried to revert traffic shaping rules to normal to see if issue resolves. No progress yet. I don't have a downtime window to reboot appliances right now, so I don't know if a reboot would help with the workaround. I will most likely find out post upgrade (in the AM) if this was the issue. 

I did notice that the mx updated to 18.211 the same morning issues occurred. 

[cockroach_c]

The downgrade ended up working. Everything is up and running. I did have to do some micromanaging on the servers ... i.e. backtrack the troubleshooting steps that were done before the ticket was escalated to me. Therefore, I don't know if adjusting the traffic shaping rules to normal actually worked or not. But, it is all fixed now with rolling back the update. 

[GiacomoS]

Hey awesome people in the Community,

Thank you for the discussion and feedback here!

We have also provided some guidance regarding the issue with traffic being dropped in this post. 

Please tag us here if there's anything else we can assist with, and make sure you raise a case with Support for tracking 🙂

Thank you!

Giac

[TyShawn]

The SNMP issue was caused by the routing issue as well according to the support team. 

@GiacomoS If you need my ticket number I can DM you.

[GiacomoS]

Hey @TyShawn ,

Please do drop me a message, I'll have a look and check if anything else is required.

Many thanks!

Giac

[GiacomoS]

Folks, 

We have released a hotfix, 18.211.0.1, which addresses the problem with traffic being dropped when you have traffic shaping rules with either a high or low priority set. If you are impacted by that, I would recommend upgrading. 

Many thanks!

Giac

[TyShawn]

So far 18.211.0.1 looks to have resolved my issue. I will report back if things change.

[Andrew_Allston]

Did this also impact the default shaping rules or only custom rules? We are just going back to make sure what we were seeing around that time was in fact due to this issue or not. Thanks in advance!

[Cyrus777]

I upgraded to this ver and caused too many issues with my AnyConnect Clients so I rolled back. Now we have a new stable release  MX 18.211.2 which I'm afraid of doing upgrade again and facing the same issues. Do you guys have tried MX 18.211.2 in prod environment yet? is there any thing we should know before upgrade?

[cvsmith122]

Do you mind me asking if your using site to site VPN and 1:1 Network assignments ? Last time we did 18.211.1 it broke both of these. I keep pushing the deployment back to make sure this does not effect us again. 

[Cyrus777]

I do have site to site VPN across the org and using hub only for mesh. We don't do 1:1 nat though. I saw one site failed due to being spoke after upgrade and I had to make that one hub as well to resolve the issue. Even Meraki support couldn't figure out what caused the issue and I had to roll back on my corp MX that is used for AnyConnect. now with 18.211.2 I'm not feeling confidant to perform the firmware upgrade anymore. 

[dcatiller]

I just updated two MX250 HA pairs at two sites and all looks good so far. 

[Abechara]

So for MX 84 no more update ?

[cmr]

There will possibly be security updates like 18.107.11 etc., but essentially that's it for MX84 and MX100.

[Abechara]

and the appliance will remain working ?

[cmr]

Yes, guaranteed until the end of life, which is October 31st 2026.  So almost two and a half years to go, with no updates...

[Abechara]

we already baught licenses for 3 years how they sell us licences like that 

[cmr]

You are supported and they should get security fixes until then, we have plenty and aren't looking to replace them yet.

[bkrautner]

Hi,

As regards the following item:

• In rare cases, MX67C, MX68CW, and Z3C appliances may fail to enter into a "Ready" state despite being able to register to a cellular network and obtain an IP address for the modem.

Seems it may also need to include Z4C's as well, as seems to have been allocated IPv4 and IPv6 addresses, but not showing as Ready:

Seems to be up, but not really 100% sure.