New MX 18.107.6 stable patch - lots of fixes, and admission that this will change and lots of issues

[cmr]

Security appliance firmware versions MX 18.107.6 changelog

Important notice

• USB modems with MX/Z series devices running firmware MX 18 or newer will be limited to best effort support and will not be receiving any future firmware fixes or improvements.

Bug fixes

• Fixed an issue that resulted in the AnyConnect VPN and IPSec client VPN services restarting when an MX appliance had a change to IPv6 uplink information, even when these services were not using or providing any IPv6 functionality.
• Corrected an issue that could result in devices connected to MX68(W,CW) and MX85 appliances being unable to negotiate 802.3at power levels from PoE.
• Fixed an issue that resulted in the AnyConnect VPN client appearing to hang for 2 minutes if the user hit cancel on the login page of the client.
• Resolved an issue that could result in MX appliances failing to free IP addresses from the client VPN subnet pool after IPsec client VPN clients disconnected.
• Corrected an issue that resulted in AutoVPN tunnels briefly dropping and re-establishing after configuration or WAN connectivity change if 1) the MX was configured in high availability (HA) mode and 2) both AutoVPN peers were running MX 18.
• Resolved an issue where MX appliances could encounter latency issues when 1) the appliance was configured in high availability, 2) the appliance was acting as an AutoVPN hub, and 3) IPv6 traffic was traversing AutoVPN.
• Corrected an issue that resulted in MX84, MX250, and MX450 appliances incorrectly returning a MAC address of 0 via SNMP for their WAN1 and WAN2 ports.
• Updated the protocol pack used by NBAR to version 66.
• Resolved an issue that could result in NBAR prematurely reaching its peak capacity for the amount of concurrent flows that it can track. When this occurred, the classification of traffic may have been less accurate. This change will increase the reliability of NBAR and its traffic classifications.
• Corrected an issue that could result in content filtering being bypassed if certain capitalization was used in the HTTP header of the GET request.
• Resolved an MX 18.107.3 regression that could result in longer device boot times for devices with many AutoVPN peers.
• Corrected an issue that resulted in MX67W and MX68(W,CW) platforms being unable to automatically select a new wireless channel for use.

Legacy products notice

• When configured for this version, Z1 devices will run MX 14.56.
• When configured for this version, MX400 and MX600 devices will run MX 16.16.9.

Known issues status

• This list of issues is currently being maintained and there may be new updates in the future.

Known issues

• After making some configuration changes on MX84 appliances, a brief period of packet loss may occur. This will affect all MX84 appliances on all MX firmware versions.
• Due to an MX 15 regression, the management port on MX84 appliances does not provide access to the local status page.
• MX appliances will now properly validate that DBD packets conform to the appropriate MTU size. If the MX’s OSPF peer has an improper MTU configured, it may cause the OSPF adjacency to fail to properly form. The updated behavior properly conforms to RFC. Please ensure these settings are properly configured on any MX’s OSPF peers to avoid disruption after upgrading to MX 18.1.X
• AutoVPN transmissions between peers which are running firmware versions 18.1 and later will consume an additional 4 bytes of overhead, totaling up to 68 bytes.
• Due to reasons still under investigation, MX85 appliances may be more likely to encounter an unexpected device reboot on this version.
• In very rare circumstances, MX appliances may report the incorrect interface IP address to the VPN registry. In some circumstances, this can interfere with the proper functioning of AutoVPN and teleworker VPN tunnels.
• MX appliances with content filtering enabled may encounter unexpected device reboots. This is more likely to occur if it has been a long time since the MX appliance was last rebooted.
• MX appliances may not failover to a backup cellular connection after the WAN interfaces have been disabled from Dashboard.
• MX67W and MX68(W,CW) appliances may experience unexpected device reboots for reasons currently under investigation. A potential cause may be oversized wireless packets.
• In rare circumstances the intrusion detection and prevention process may crash and restart. In some circumstances this can cause a minor disruption to network traffic. This issue is expected to be resolved through an update to the IDS/IPS container rather than the MX firmware version.
• Due to unknown reasons, MX64W and MX65W may experience unexpected device reboots. This is most likely related to the wireless subsystem.
• Due to a rare issue with no known method of reproduction, MX appliances have been documented to fail to fetch an updated device configuration for several days.
• In rare cases, MX67(C,W) and MX68(W,CW), MX75, MX85, MX95, and MX105 appliances with intrusion prevention configured may erroneously block SIP traffic from client VPN clients. This is most likely related to an issue with IP fragmentation and reassembly.
• In rare cases, MX67(C,W) and MX68(W,CW), MX75, MX85, MX95, and MX105 appliances with intrusion prevention configured may result in increased latency for Citrix. This may be related to an issue with IP fragmentation and reassembly
• Due to a rare issue under investigation, MX67C and MX68CW appliances may unexpectedly fail to detect some working SIM cards.
• In rare cases, large numbers of routes can cause network instability during AutoVPN connectivity changes.
• In rare cases, MX67C, MX68CW, and Z3C appliances may fail to enter into a "Ready" state despite being able to register to a cellular network and obtain an IP address for the modem.
• MXs appliances incorrectly modify the source IP address of ICMP time-to-live exceeded messages when routing them between VLANs.
• MX67W and MX68(W,CW) appliances may experience a crash of the wireless subsystem that results in a device reboot.
• Due to architectural changes to support content filtering powered by Talos, MX devices will no longer report the category that caused a URL to be blocked by content filtering when in full list mode.
• In rare cases, MX appliances may report severely incorrectly data for the loss and latency graphs visible on the Appliance Status page.
• Due to a rare issue with no known method of reproduction, MX95, MX105, MX250, and MX450 appliances may encounter unexpected device reboots.

[PhilipDAth]

That is a good number of AnyConnect improvements.

[CptnCrnch]

Whoa. More Known Issues than fixes 😳

[PhilipDAth]

The golden question - how many unknown bugs are there ...

[ww]

I always wonder.. are these bugs on 18.107.6. Or is it also present in all/some previous fw.

It should say, present since fwxx

[Mloraditch]

I don't know how far back it goes, but they did update at least some older versions and included this note:

• This list of issues has been provided as part of a one-time update. The list is current as of October 23, 2023 and is no longer being updated with new information. Please see the latest available patch release for new, active updates.

[RaphaelL]

With my luck and the complexity of our setup pretty sure I'm going to hit every single "Due to a rare bla" and "Due to unknown reasons"  haha !

[jdestef17]

"In rare circumstances the intrusion detection and prevention process may crash and restart. In some circumstances this can cause a minor disruption to network traffic. This issue is expected to be resolved through an update to the IDS/IPS container rather than the MX firmware version."

Our network has been experiencing these issues, and it's concerning that this hasn't been more widely communicated. I'm quite surprised by this. We've been told that the Snort process is crashing and have been informed that it requires a patch from Meraki support. However, the patch seems to be hit or miss. When can we expect this issue to be addressed?

[cmr]

Is this what is causing me a 15-30 second outage every 20 minutes or so?  I've been having a go at my ISP...  it still might be them as I'm using a very lightly loaded MX75.

[jdestef17]

Disable for an hour and see if it clears up. Ours was roughly every 20-30 minutes when there was traffic on the network. Extremely frustrating. Also, if you're pinging your local router's interface and seeing drops, that's the culprit. If you're not dropping packets to your router internally, it's most likely ISP. When Snort crashes, it locks up the MX so it cannot process anything in/out, which would affect all local and egress traffic. 

[cmr]

Disabled just now...  This is what I was seeing and nobody was using the network in the gap...:

[RaphaelL]

From what I understand this should ( could , must ? ) be present on any MX 18.1 firmware since this is mostly coming from SNORT anyway ? 

What IPS/IDS settings are you running ? I will compare with mine , not sure if we are hiting that issue ( I hope not lol )

[jdestef17]

MX 18.107.2

AMP = Enabled

ID&P = Prevention & Security

We had to call support to have them apply a patch that downgraded us from SNORT v3 to SNORT v2. It's been about 5 hours with no dropped packets. 

[cmr]

18.107.5 and 18.107.6

AMP = Enabled

ID&P = Prevention & Balanced

Last outage was when I changed the setting, monitoring now and will confirm...

[cmr]

Looks like it might be the ISP, yellow highlight shows when I turned off ID&P

[RaphaelL]

New list of changes : 

Resolved an issue where MX appliances could encounter latency issues when 1) the appliance was configured in high availability, 2) the appliance was acting as an AutoVPN hub, and 3) IPv6 traffic was traversing AutoVPN.

changed to :

• Resolved an issue where MX appliances could reboot or experience a period of degraded functionality after an undetermined period of time when 1) the appliance was configured in high availability, 2) the appliance was acting as an AutoVPN hub, 3) the MX established AutoVPN connectivity to a large number of peers, and 4) there were frequent AutoVPN connectivity changes between the MX and its AutoVPN peers. This issue has been most frequently observed on AutoVPN hubs supporting more than 500 AutoVPN peers.

^^^^^^^^^ was so fun to troubleshoot 🙂 

Added : 

• MX64W and MX65W appliances may experience unexpected device reboots for reasons currently under investigation.
• MX67C, MX68CW, and Z3C appliances may erroneously detect a SIM card as missing. This state can be cleared by rebooting the device.
• MX67C, MX68CW, and Z3C appliances may encounter an issue where they are unable to communicate with the integrated modem. This state can be cleared by rebooting the device.
• When MX67C, MX68CW, and Z3C appliances are repeatedly unable to communicate with the integrated modem, they will attempt to reset the modem to restore connectivity. In some cases, this reset procedure may fail, requiring the appliance to be physical power cycled to restore connectivity with the modem.
• Due to an MX 17 regression, the integrated cellular modem on MX67C, MX68CW, and Z3C appliances may fail to acquire an IP address via DHCP. This can be resolved with a physical power cycle of the appliance.

[RaphaelL]

I wish the firmware bot would post the news changes everytime it gets updated 🙂 

[cmr]

It was @MeredithW who introduced me to the firmwarebot, @AmyReyes would you know who in Meraki developed the code and could it be modified to update us when release notes get updated?

[AmyReyes]

Great question! I'll take this back to the Community team and see if anyone knows. Tagging in @Leinad77 for visibility as this falls into his domain of platform management 😊

[AmyReyes]

In the chaos of my last week before leave, I forgot to follow up on this! The code was developed by a Meraki SE, and unfortunately, he made it publish everything that it can. Posting updates and changes as they happen is outside the scope of what it can do and would require a manual effort from someone. But that is definitely worth our consideration for the future as something to pursue!

[RaphaelL]

Updated 2023-11-30

Added :

• Due to conditions under investigation, MX appliances often fail to initialize a service required for encrypted communication with Umbrella.
• Due to unknown causes, the NBAR traffic analysis engine may fail to classify traffic in some cases.

• Due to a rare issue with no known method of reproduction, MX appliances may reboot unexpectedly.

[RaphaelL]

Appending the latest changes to the bottom + timestamp would also help !

[RaphaelL · Accepted solution]

Someone is listening !!!

This was added to the current changelogs : 

Known issues - november 20th update

[...]