New MX 17.10.4 stable firmware - more performance and reliability fixes than we could have hoped!

[cmr]

Security appliance firmware versions MX 17.10.4 changelog

Important notice

• While Meraki appliances have traditionally relied on UDP port 7351 for cloud communication and TCP ports 80 and 443 for backup communications, with MX 16 we are beginning a transition to using TCP port 443 as the primary means for cloud connectivity. In order to ensure proper connectivity to the Meraki cloud after this upgrade, please ensure that all “Meraki cloud communication” traffic specified in the Help > Firewall Info page is allowed through any firewalls or security filtering devices that may be deployed upstream of your Meraki appliances. These requirements have been updated on Nov 2022, so it’s important that you review them.
• HTTP proxy, which allows default management traffic from MX appliances to be sent through a proxy, is deprecated on MX 16 and higher firmware versions.
• The transition to Cisco Talos intelligence for our content filtering services means that some URL categories have changed names, some categories are no longer available, and multiple new categories are now available. Please review your configuration after upgrading to ensure content filtering is effectively tailored to your needs and deployment environment.

Bug fixes

• Fixed an issue that resulted in​​ fragmented packets that were received by an MX appliance and transmitted across AutoVPN getting fragmented after encryption took place.
• Corrected a rare issue that could result in MX84, MX100, and vMX appliances being unresponsive after rebooting.
• Fixed an issue that resulted in wireless clients experiencing degraded performance for HTTP and HTTPS traffic when content filtering was enabled.
• Reduced the amount of time before a URL classification request performed by the content filtering service would be considered to have failed. This may resolve latency caused by content filtering in cases when classification requests failed and needed to be retried on a consistent basis.
• Resolved an issue that could result in connectivity checks to Meraki cloud infrastructure that are performed prior to allowing an upgrade to be scheduled to erroneously fail.
• Fixed a rare issue that could result in the WAN interfaces for MX appliances incorrectly transitioning to a down state for a brief period of time.
• Corrected an issue that resulted in intrusion detection rule update events not being logged to the event log.
• Resolved an issue where an MX appliance could stop responding to LAN traffic when a frame was received with a source MAC address matching the MX’s own LAN port. This issue was most likely to occur in complex switching topologies, or when there were forwarding issues within the network.
• Fixed an issue of incorrect fragmentation handling for IPv6 packets.
• Fixed a rare issue that could result in disruptions to DHCP in High Availability configurations.
• MX appliances will now drop additional types of erroneous traffic received from AnyConnect VPN clients.
• Resolved a rare case that could result in non-Meraki VPN traffic being incorrectly forwarded when MX appliances were configured in passthrough mode.
• Fixed several rare cases that could result in a device reboot.
• Corrected an issue that resulted in client traffic being will be dropped by MX65(W), MX67(C,W), and MX68(W,CW) appliances when 1) The client was connected to a LAN port with 802.1X authentication enabled and 2) The VLAN ID of the port was configured to 16, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, or 240.
• Resolved an issue that caused Z3(C) devices to be described as “Security Appliance” as opposed to “Teleworker Gateway” on the device local status page.
• Corrected an issue that resulted in the device local status page incorrectly showing a configuration to convert the WAN1 port to operate as a LAN port.
• Fixed an issue that could result in MX appliances replying to ARP messages for an incorrect IP address when 1) The MX was configured to operate as the standby/spare device in a high availability configuration and 2) the MX appliance was configured to operate in passthrough mode.
• MX appliances will no longer attempt to perform content filtering on URLs with some invalid characters.
• Fixed an issue that caused packet loss between client devices communicating within the same VLAN when both client devices were connected to ports configured for 802.1X port authentication
• Corrected an issue that could result in an inconsistent connection to the Cisco Umbrella service when Umbrella Protection was configured
• Resolved a case where the device local status page could still be accessed, even after it had been disabled via the Meraki Dashboard.
• MX appliances will now always connect to Active Directory servers using DCOM version 5.7. This should help resolve errors when communicating with Active Directory servers using more recent versions of software.
• Corrected an issue that could result in the Dashboard reporting of the Active Directory server status always showing a failed connection.
• Corrected an additional issue that could result in AutoVPN connectivity failing to form when cellular active uplink was configured.
• Fixed an issue that could result in ESP packets being incorrectly fragmented on MX67(C,W), and MX68(W,CW), MX75, and MX85 appliances.
• Resolved a rare issue that could prevent Z3(C) appliances from upgrading to MX 17.X firmware versions when AutoVPN was enabled.
• Stability improvements for Umbrella Meraki SD-WAN Connectors.
• Resolved a rare issue that could result in traffic being misclassified by NBAR.

Legacy products notice

• When configured for this version, Z1 and MX80 devices will run MX 14.56.
• When configured for this version, MX400 and MX600 devices will run MX 16.16.9.

Known issues

• After making some configuration changes on MX84 appliances, a brief period of packet loss may occur. This will affect all MX84 appliances on all MX firmware versions
• Due to an MX 15 regression, the management port on MX84 appliances does not provide access to the local status page

Other

• MX appliances will now use 64 as the CurHopLimit value in IPv6 Router Advertisement messages.

[Brash]

Wow!

Engineering must be doing a spring cleaning of their old defects. Better late than never as they say 😊

[rhbirkelund]

Corrected an issue that resulted in client traffic being will be dropped by MX65(W), MX67(C,W), and MX68(W,CW) appliances when 1) The client was connected to a LAN port with 802.1X authentication enabled and 2) The VLAN ID of the port was configured to 16, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, or 240.

That's been there for a while. 😄

[ww]

This bit must be stuck in de code

00000000

[rhbirkelund]

Yeah... I'm guessing some Dev is cutting corners or doing something "clever" in binary conversion.. 😉

I'd love to see the patch on that one tnough!

[AlexP]

I naturally cannot go into more details about the nature of the defect, but I will say, I was the one who scoped that internally, and I got quite the laugh when I determined the scope.

"Wired 802.1X client traffic dropped when port VLAN ID < 256 && VLAN ID % 16 == 0" is certainly one of the more absurd bug subjects I've ever submitted 😁

[cmr]

I've applied it to an MX64, MX65 and pair of MX100s.  So far, so good...  The MX84 pairs will be the real test.

[Henrik_]

We updated our MX84 HA yesterday. Everything looks fine so far. 

[SouthPaw2020]

Does this address the WAN speeds issue when IDP is turned on?

[NJNetworkGuy100]

I can confirm the "Fixed an issue that resulted in wireless clients experiencing degraded performance for HTTP and HTTPS traffic when content filtering was enabled." worked on a network using a MX68W for firewall/wifi. 

They were complaining of slow wifi speeds off the MX since being upgraded to 17.10.2 a month ago, and this patch fixed their complaints on wifi speeds.  

[Jedijrobbie]

Can anyone confirm if this helped correct the below error with Anyconnect? We've been experiencing this and other odd errors ever since we moved from 16.6.6 to 17.10.

Solved: Re: Cisco Anyconnect - SAML using OneLogin for MFA - The Meraki Community

[anibalgg]

Hi What about URL categories? Are there some big changes? Regards

[cmr]

We have hit one issue with this update.  On a site with an HA pair of MX100s that are running in multiple VLAN mode serving DHCP IP addresses on most of them, it stops working for multiple of them and we see the below:

You can see that the device with MAC ending 43 seems to receive an offer in the same way that the device below did, yet it then complains that it doesn't have one.  The device does not get an IP and falls off the network.

We tried rebooting the MXs but the issue reoccurred within 5 minutes, so we rolled back.

We don't have any other HA pairs of MXs that serve DHCP and the single MXs we have that do serve DHCP are lesser models such as MX84, MX67 etc. and they all seem okay.

Has anyone else upgraded and HA pair to this release that is acting as a DHCP server for multiple subnets?

[cmr]

This fix might have caused it as 17.10.2 worked fine...

• Fixed a rare issue that could result in disruptions to DHCP in High Availability configurations.

???

[Anoraknid]

Yes we have just run into this today.  Our current work around is to swap roles between primary and spare. DHCP service is resurrected.  How successful that might be long term is as yet unknown. We have multiple MX100 HA deployments and it has only affected one so far.  All template builds so rolling back one means rolling back many. Not keen.

[Eric-Fretz]

We've run into this same problem with all of our MX100 deployments, as well...  I built a quick & dirty script to dig through event logs in the past 5 minutes looking for DHCP failures, then swap the active/standby units, then reboot the standby.  I have one pair of MX100s at a large-ish site that gets failed over & rebooted several times per day.  *sigh*

[Anoraknid]

We have 10 sites with HA MX100 .  This "DHCP problem extra: no_offers_received, vap: 0"  bug affected only 1 out of the 20 devices that were upgraded  to 17.10.4. So at one site swapping the primary for the spare solved issue. However we didn't want to take the reliability hit and have somewhat reluctantly rolled back to 17.10.2

[Anoraknid]

Rolling back to 17.10.2 and then forward again to 17.10.4  has resolved our DHCP issue.

[Eric-Fretz]

"Fixed an issue that resulted in​​ fragmented packets that were received by an MX appliance and transmitted across AutoVPN getting fragmented after encryption took place."

We are still seeing this issue with 17.10.4 on a MX67W.  Specifically, any SSID advertised on this unit with RADIUS authentication (or wired splash pages with RADIUS authentication) fail due to the MX dropping fragmented RADIUS response packets.  The last version of MX to handle this correctly was 16.16.8, which we had to rollback to when 17.10.4 broke RADIUS once again.

For anyone else with similar problems, our MX67W is advertising a 802.1x authenticated wireless SSID and wired VLAN using RADIUS and the RADIUS servers are across the autoVPN in remote datacenters.  We have had case 08991820 open since 12/3/2022 for this issue.

[NJNetworkGuy100]

We ran into this exact same Radius failure issue with the 17.10.2 firmware (and still with the 17.10.4 patch).  We have a couple of branch locations using the MX wifi and Radius over the AutoVPN to a data center.  We rolled those MX's back to 16.16.8.  

For those couple of MX's at those branch offices, since no patch is forthcoming and we want to update the MX's, we've shipped them an AP to use instead of the built-in MX wifi.  

We opened a ticket with Meraki ourselves just to keep it on Meraki's radar.  

[Nathan_McNair]

Tacking onto what NJNetworkGuy100 said, in our lab network, we upgraded an MX68W to 18.106 and RADIUS fails there as well.

[CapnDoody]

Can anyone verify that they are still seeing the issue with access to the status page via Mgmt port? I have an MX84 that we can't get to the status page on, but another one (going to be an HA pair) right next to it that works fine. The one not working has not been online yet, because it needs a static for our ISP. 

[AET-Tech]

FYI, All computers behind MX can no longer install OfficeSetup.exe. I was told it was HTTP Content Caching issue. Will need to upgrade to MX 17.10.5