New MX 19.1.8 stable release candidate: more fixes than Bob the builder

[cmr]

Security appliance firmware versions MX 19.1.8 changelog

Important notice

• As of MX 19.1, Cisco Meraki will no longer support USB-based Cellular Failover on the MX and Z platforms.

Legacy products notice

• When configured for this version, Z1 devices will run MX 14.56.
• When configured for this version, MX400 and MX600 devices will run MX 16.16.9.
• When configured for this version, MX64(W), MX65(W), MX84, MX100, and vMX100 devices will run MX 18.107.12.

Executive summary

• MX 19.1 has crossed several key adoption thresholds having already been deployed on more than 20% of eligible devices. Based on this and its performance in the field, we are anticipating promoting MX 19.1 to Stable around the late summer to early fall time frame. We would strongly encourage all customers to start evaluating MX 19.1.
• This is the latest generally available maintenance release for the MX 19.1 release. It contains a substantial amount of valuable fixes across many different areas. We anticipate almost all devices having valuable fixes in this release.
• Customers that have experienced issues with unexpected device reboots on MX75 appliances are strongly encouraged to evaluate this release. A private preview of the fixes included here have shown significant improvements in many cases.
• There are also a variety of fixes related to cellular connectivity; AutoVPN; for MX75, MX85, MX95, MX105, MX250, and MX450 appliances generally; and much more. Please read through the full details below.

Bug fixes - general fixes

• Fixed an issue that could result in duplicate retrospective “malware download detected” emails being erroneously sent. This resolves known issue MX-30111.
• Corrected a rare issue that could result in a URL persisting in the content filtering allow list if the URL was reconfigured from being included in the allow list to being included in the block list.
• Corrected a very rare issue that could result in web traffic being incorrectly denied when URLs were defined in the custom pie chart option through the Network-Wide>General section of the Meraki Dashboard.
• Fixed an issue that resulted in MX appliances incorrectly advertising their IPv6 uplink subnets to AutoVPN peers through IBGP when there were not IPv6-enabled VLANs participating in VPN on the MX.
• Fixed an issue that could result in Non-Meraki VPN tunnels failing to form when the peer was configured for an FQDN and only an IPv6 DNS response was available for that FQDN.
• Fixed an issue that resulted in MX appliances failing to properly install a 0.0.0.0/0 route into the routing table when it was learned through BGP over an IPSec VPN connection.
• Fixed an issue that could result in traffic sourced from clients in non-VPN-joined VLANs being incorrectly routed over an IPSec VPN connection. This could occur when the destination for that traffic matched a subnet that the MX appliance had learned about through an EBGP session with an IPSec VPN peer.
• Updated the AnyConnect VPN service.
• Corrected an issue that could result in a sustained decrease in available system memory after continually connecting and disconnecting client VPN clients.
• Fixed a case that could result in an unexpected reboot when AutoVPN was in use. This was most likely to occur on VPN hubs running MX 19 firmware with many AutoVPN peers also running MX 19 firmware.
• Resolved an MX 18.1 regression that resulted in MX appliances improperly routing traffic in the following configuration: 1) the spoke is configured in passthrough mode, 2) the spoke is configured with at least two AutoVPN hubs, 3) OSPF route advertisement is configured, and 4) there is at least one route being advertised from both hubs.
• Corrected an issue that could result in TCP flows being removed from flow tables sooner than expected under high volumes of network flows.
• Resolved an MX 18.211.5.1 regression that could result in excessive CPU utilization when SD-WAN policies for Internet traffic were enabled and at least 8 tracking destinations were configured.
• Corrected an issue that caused traffic to the Thousand Eyes Cloud to be incorrectly dropped If an MX appliance had a WAN uplink with an MTU lower than 1500 bytes.

Bug fixes - limited platform fixes

Z4(C)

• Fixed an issue on Z4(C) appliances that resulted in unicast traffic between an 802.1X authenticated client and other devices on the same VLAN being incorrectly dropped.
• Resolved an MX 19.1.5 regression that could result in Z4(C) appliances failing to provide PoE power to connected devices. This resolves known issue MX-34938.
• Fixed an issue that could result in an unexpected reboot could occur on Z4(C) appliances when Wi-Fi was in use.

Z3C, MX67C, MX68CW

• Fixed an issue that could result in the cellular modem remaining active on Z3C, MX67C, and MX68CW appliances, even after it was disabled through the Meraki Dashboard.
• Resolved an issue that could result in Z3C, MX67C, and MX68CW appliances failing to get an IP address from their integrated cellular modems, despite having carrier connectivity.

Any C

• Corrected an issue that could result in Z3C appliances failing to successfully form cellular connections using the integrated cellular modem.
• Corrected an MX 19.1 regression that resulted in traffic failing to route over AutoVPN when the only active uplink was a cellular connection. This resolves known issue MX-35703.
• Resolved an MX 18.2 regression that resulted in the WAN2 being unable to pass traffic if 1) WAN1 was not in use and 2) cellular was enabled.
• Fixed inconsistencies with the cellular active uplink feature. WAN 2 cannot be used as a functioning WAN interface when cellular active uplink is enabled.

SoHo

• Performance improvements for Z3(C), Z4(C), MX67(C,W), MX68(W,CW), and MX75 appliances.
• Resolved an issue that resulted in various types of unexpected device reboots on MX75 appliances.

1Gb+

• Corrected an issue on MX75, MX85, MX95, MX105, MX250, and MX450 appliances that could cause HTTP traffic transferred across AutoVPN to fail when HTTP Content Caching was configured.
• Fixed an MX 19.1.7 regression that could result in MX75, MX85, MX95, MX105, MX250, and MX450 appliances servicing many clients to experience intermittent periods of heightened packet drops and latency. This resolution to known issue MX-36338 was also fixed in MX 19.1.7.1.
• Corrected an issue that could result in high device utilization or an unexpected device reboot when the pool of ports available for NAT translation were exhausted on MX75, MX85, MX95, MX105, MX250, and MX450 appliances. This was also resolved in MX 19.1.7.2.
• Fixed an issue on MX75, MX85, MX95, MX105, MX250, and MX450 appliances that could result in the ping live tool erroneously reporting 100% loss for IPv6 addresses.
• Fixed an issue that resulted in MX75, MX85, MX95, MX105, MX250, and MX450 appliances experiencing an unexpected device reboot when VPN NAT was configured. This resolution to known issue MX-36180 was also resolved in MX 19.1.7.2.
• Corrected an issue that could result in network performance metrics for AutoVPN tunnels being calculated incorrectly on MX75, MX85, MX95, MX105, MX250, and MX450 appliances. This may have affected the reliability of performance-based uplink selection policies for AutoVPN traffic.
• Resolved an MX 18.211.4 regression that could result in IBGP sessions failing to form between AutoVPN peers when 1:M VPN NAT was configured on MX75, MX85, MX95, MX105, MX250, and MX450 appliances. This resolves known issue MX-36231.
• Fixed an issue that caused MX75, MX85, MX95, MX105, MX250, and MX450 appliances to send more ICMP pings than intended for uplink statistic monitoring.

Multigig

• Resolved a very rare issue that could result in MX95, MX105, MX250, and MX450 appliances incorrectly broadcasting frames that were destined to the MX itself.
• Fixed an issue that could result in MX95, MX105, MX250, and MX450 appliances incorrectly forwarding CDP frames.

Datacentre

• Corrected an issue that could result in an unexpected device reboot when there were many teleworker VPN peers connected to an MX appliance. This was most likely to occur on MX250 and MX450 appliances given the scale of VPN connections they typically support.
• Resolved a rare issue that could result in MX250 and MX450 appliances becoming temporarily unresponsive after making VLAN configuration changes.

vMX

• Corrected an issue that prevented taking LAN packet captures on vMX appliances from the Meraki Dashboard.
• Resolved an issue that resulted in AnyConnect VPN not functioning for vMX appliances when a DDNS certificate was used.

Known issues status

• This list is being reviewed and updated.

Known issues

• During the upgrade process, MX appliances upgrading from version prior to MX 19 may experience a failure to properly classify traffic. This issue will be resolved once the appliance has completed the upgrade to MX 19. (MX-36307)
• Due to an issue under investigation, MX appliances may incorrectly route traffic destined to subnets learned through eBGP over a Non-Meraki VPN connection. (MX-34803)
• Due to an issue under investigation, making certain configuration changes to WAN interfaces (such as disabling or enabling an interface) can cause the IDPS process to fail. This issue may also cause high device utilization. The issue can be worked around by rebooting the MX appliance or disabling and then re-enabling IDPS. (MX-34504)
• When failover is configured between non-Meraki VPN tunnels, the Route Table page on Dashboard may incorrectly show the route for the primary VPN tunnel is inactive. (MX-36316)
• During the upgrade process, MX appliances upgrading from versions prior to MX 19 will experience a failure to connect to non-Meraki VPN peers if any VPN peer names contain a space. This issue will be resolved once the appliance has completed the upgrade to MX 19. (MX-36312)

Other

• Introduced an updated version of the fix for known issue MX-35210. A fix was initially included in MX 19.1.7, but was temporarily disabled in MX 19.1.7.2. Under conditions of heavy network traffic and system load, MX appliances will begin to more actively remove clients that have not been recently active from its databases. Additionally, MX appliances can limit the number of total concurrent clients that can be supported during these periods of high system load in order to maximize network performance and stability.
• MX appliances will now expire or age out DNS flow more rapidly than flows for other types of network traffic.

[DarrenOC]

Can he fix it?

[cmr]

He's added some much needed separators now 😈

[ww]

Need him to fix MX-34504

[cmr]

@ww just turn it off and on again.  It's worked for Microsoft for years 😉

[RaphaelL]

Yep that update looked promising until I saw that MX-34504 wasn't fixed. Won't upgrade

[PhilipDAth]

Someone has been doing a lot of work.

[rhbirkelund]

When you realize you've been doing "git commit" for a while, but hadn't done "git push".

[FRover]

The old issue of MX67C appliances losing their 4G network connectivity should be resolved now? I am dying to upgrade from 16.16.9

[Chad_CSB]

We've been running 18.211.x for a while without issue. Have you tried upgrading? 

8x Mx67c using cellular failover w/o issue. Users don't even notice internet outages now.

[TyShawn]

Updated my MX75 to this release to test the stability issues. I will be pushing this load out over the next few days to a small group of sites I manage.

p.s.

I'm bbaaccck :).

[CarolineS]

👋 Hi @TyShawn! We missed you! 

[TyShawn]

Moved over a group of Z3/4's to this release so far things are stable.

[cmr]

Bit of a performance hit with 19.1.8 vs 19.1.7.2, see the graph below:

[Chad_CSB]

Similar observation on our pilot device. Side note, we DID enable a Cisco XDR integration though immediately post upgrade. We upgraded on the 7th at 5:00pm. MX67C utilization screenshot. 

18.211.5.1 to 19.1.8

[cmr]

No changes at all on the MX75 I upgraded above...

[RaphaelL]

Yikes. 

Looks exactly the same hit we got when we went from 18.107 to 18.211 on small-medium MXs.

[cmr]

Indeed, I did have the same when jumping from 18.1 to 18.2, so skipped straight to 19 which has been good (for me) until now.  I'm leaving it for the moment as it is stable.

[Fabian1]

Same here with MX67C