New MX 18.107.7 patch firmware - many VPN, reboot and other fixes, a fair few issues as well!

[cmr]

Security appliance firmware versions MX 18.107.7 changelog

Important notice

• USB modems with MX/Z series devices running firmware MX 18 or newer will be limited to best effort support and will not be receiving any future firmware fixes or improvements.

Bug fixes

• Fixed a data validation issue with PPPoE authentication on the device local status page that could result in devices getting into an invalid config state when invalid data was entered for the PPPoE username.
• Fixed a race condition that could result in the integrated cellular modem on Z3C, MX67C, and MX68CW devices getting stuck being unable to detect the SIM card.
• Resolved an issue that resulted in client VPN clients being unable to communicate properly if they were connected to an MX appliance configured to operate in passthrough mode.
• Resolved an issue that could result in the loss graph from the “historical device data” section of the appliance status page incorrectly reporting 100% packet loss.
• Fixed an issue that resulted in some MX67(C,W) and MX68(W,CW) appliances experiencing difficulty upgrading to MX 18.2 from their factory default firmware.
• Added support for configuring 10 Gbps full duplex from the local status page for WAN interfaces on MX85, MX95, and MX105 appliances.
• Various cellular fixes to increase the reliability of integrated cellular modems.
• Expanded the range of conditions that Z3C, MX67C, and MX68CW appliances can automatically recover from the integrated modem becoming unresponsive.
• Corrected an issue that could result in MX appliances not failing over to a backup cellular connection after the WAN interfaces had been disabled from Dashboard.
• Resolved a rare issue that could result in unexpected VRRP transitions when MX appliances were configured in high availability (HA) and content filtering was enabled.
• Fixed an issue that could result in a device reboot when content filtering was enabled.
• Corrected a case that could result in the AnyConnect process crashing.

Legacy products notice

• When configured for this version, Z1 devices will run MX 14.56.
• When configured for this version, MX400 and MX600 devices will run MX 16.16.9.

Known issues status

• This list of issues is currently being maintained and there may be new updates in the future.

Known issues

• Due to unknown causes, the NBAR traffic analysis engine may fail to classify traffic in some cases.
• Due to conditions under investigation, MX appliances often fail to initialize a service required for encrypted communication with Umbrella.
• MX250 and MX450 appliances may incorrectly forward LLDP and BPDU messages received on the LAN out their WAN interface(s) during the bootup process.
• In very rare circumstances, MX appliances may report the incorrect interface IP address to the VPN registry. In some circumstances, this can interfere with the proper functioning of AutoVPN and teleworker VPN tunnels.
• Due to a rare issue with no known method of reproduction, MX appliances may reboot unexpectedly.
• MX64W and MX65W appliances may experience unexpected device reboots for reasons currently under investigation.
• MX67C, MX68CW, and Z3C appliances may erroneously detect a SIM card as missing. This state can be cleared by rebooting the device.
• MX67W and MX68(W,CW) appliances may experience unexpected device reboots for reasons currently under investigation. A potential cause may be oversized wireless packets.
• In rare circumstances the intrusion detection and prevention process may crash and restart. In some circumstances this can cause a minor disruption to network traffic. This issue is expected to be resolved through an update to the IDS/IPS container rather than the MX firmware version.
• Clients using an older version of the AnyConnect client may not be able to successfully perform Duo multi-factor authentication. This can be resolved by updating the AnyConnect client to 4.10.05085 or higher.
• Due to unknown reasons, MX64W and MX65W may experience unexpected device reboots. This is most likely related to the wireless subsystem.
• MX67C, MX68CW, and Z3C appliances may encounter an issue where they are unable to communicate with the integrated modem. This state can be cleared by rebooting the device.
• Due to a rare issue with no known method of reproduction, MX appliances have been documented to fail to fetch an updated device configuration for several days.
• In rare cases, MX67(C,W) and MX68(W,CW), MX75, MX85, MX95, and MX105 appliances with intrusion prevention configured may erroneously block SIP traffic from client VPN clients. This is most likely related to an issue with IP fragmentation and reassembly.
• In rare cases, MX67(C,W) and MX68(W,CW), MX75, MX85, MX95, and MX105 appliances with intrusion prevention configured may result in increased latency for Citrix. This may be related to an issue with IP fragmentation and reassembly.
• MX67C, MX68CW, and Z3C appliances may fail to apply custom APNs.
• Due to a rare issue under investigation, MX67C and MX68CW appliances may unexpectedly fail to detect some working SIM cards.
• In rare cases, large numbers of routes can cause network instability during AutoVPN connectivity changes.
• In rare cases, MX67C, MX68CW, and Z3C appliances may fail to enter into a "Ready" state despite being able to register to a cellular network and obtain an IP address for the modem.
• When MX67C, MX68CW, and Z3C appliances are repeatedly unable to communicate with the integrated modem, they will attempt to reset the modem to restore connectivity. In some cases, this reset procedure may fail, requiring the appliance to be physical power cycled to restore connectivity with the modem.
• Due to an MX 17 regression, the integrated cellular modem on MX67C, MX68CW, and Z3C appliances may fail to acquire an IP address via DHCP. This can be resolved with a physical power cycle of the appliance.
• MXs appliances incorrectly modify the source IP address of ICMP time-to-live exceeded messages when routing them between VLANs.
• When using a cellular active uplink with the primary uplink configured as cellular, the Dynamic DNS hostname will not function properly.
• MX67W and MX68(W,CW) appliances may experience a crash of the wireless subsystem that results in a device reboot.
• Due to architectural changes to support content filtering powered by Talos, MX devices will no longer report the category that caused a URL to be blocked by content filtering when in full list mode.
• Due to a rare issue with no known method of reproduction, MX95, MX105, MX250, and MX450 appliances may encounter unexpected device reboots.
• Due to reasons still under investigation, MX85 appliances may be more likely to encounter an unexpected device reboot on this version.

Other

• Added support for configuring 10 Gbps full duplex from the local status page for WAN interfaces on MX85, MX95, and MX105 appliances.

[CptnCrnch]

That‘s a darn long list of known issues 🤔

[TyShawn]

Sheesh cell connectivity feels a bit like "Under Construction" at this point based on the known issues.

[RaphaelL]

Can't quite understand that : 

BUG FIXES

• Fixed a race condition that could result in the integrated cellular modem on Z3C, MX67C, and MX68CW devices getting stuck being unable to detect the SIM card.

KNOWN ISSUES

• Due to a rare issue under investigation, MX67C and MX68CW appliances may unexpectedly fail to detect some working SIM cards.

So I guess that SIM cards can randomly fail on MX68CW/MX67C and it is still not 100% fixed ?

[jbright]

Why is Cisco calling this a Stable release when you have statements for it like this one:

• Due to reasons still under investigation, MX85 appliances may be more likely to encounter an unexpected device reboot on this version.

This seems far from stable to me...

[Mloraditch]

I would hazard a guess that other Cisco hardware has issues list that are similar, it's just not nearly as easy to read and digest them. As an example: https://www.cisco.com/c/en/us/td/docs/security/asa/asa918/release/notes/asarn918.html#reference_ygr_...
has this bug: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwh58090
And the list of known issues is about the same length and seem roughly similar in severity.

I would say this is all pretty normal and they weigh the qty and severity of the bugs when publishing for download.

It would help if they'd be clear when the bug was first detected and what versions it's been seen on. Maybe one day we'll get Bug Toolkit integration!

[Fabian1]

So we better stay on MX 18.107.2?

Those pached firmwares are more confusion than releasing release candidates...

Stable should be stable in my opinion

[CptnCrnch]

My experience (and also for a a few customers) 18.107.7 has been perfectly OK.

[RaphaelL]

No point on staying on 18.107.2 , you are missing on all the fixes.

[mikabrownrm]

This is still an issue??

• In rare circumstances the intrusion detection and prevention process may crash and restart. In some circumstances this can cause a minor disruption to network traffic. This issue is expected to be resolved through an update to the IDS/IPS container rather than the MX firmware version.

[RaphaelL]

For those running 18.107.7 and lower : 

Known issues - january 25th update

• Due to an MX 18.107.7 regression, MX appliances that 1) have Mandatory DHCP enabled and 2) are rebooted, can encounter severe disruptions to network traffic. We recommend customers with Mandatory DHCP enabled do not upgrade to this firmware version.
• The Non-Meraki VPN service may fail to properly establish IKEv2 tunnels when the MX appliance is acting as the IKEv2 responder and many allowed subnets are configured.
• Due to an MX 18.1 regression, VPN status information about WAN2 is not properly reported. This will result in the information in the VPN status page being incorrect.

[MichelSchuurman]

Using mand. DHCP in a lot of VLAN's in our network, experienced major issues.

Had to roll back to 18.107.6 as advised by Meraki Support.

Upgrading to 18.107.8 also resulted in an unstable network, switches, camera's, AP's no longer coming online.

Latest update from support: 

"we have an issue with mandatory DHCP on firmware 18.107.7 and above, and it is still unresolved"

[Janny_Rupus]

is this solved by now? or with the FW 18.2?