New MX 17.6 stable release candidate firmware - no longer a beta, many fixes, AnyConnect for MX64/5

[cmr]

Security appliance firmware versions MX 17.6 changelog

Important notice

• While Meraki appliances have traditionally relied on UDP port 7351 for cloud communication and TCP ports 80 and 443 for backup communications, with MX 16 we are beginning a transition to using TCP port 443 as the primary means for cloud connectivity. In order to ensure proper connectivity to the Meraki cloud after this upgrade, please ensure that traffic using TCP port 443 between 209.206.48.0/20 is allowed through any firewalls that may be deployed upstream of your Meraki appliances.
• HTTP proxy, which allows default management traffic from MX appliances to be sent through a proxy, is deprecated on MX 16 and higher firmware versions.
• The transition to Cisco Talos intelligence for our content filtering services means that some URL categories have changed names, some categories are no longer available, and multiple new categories are now available. Please review your configuration after upgrading to ensure content filtering is effectively tailored to your needs and deployment environment.

Legacy products notice

• When configured for this version, Z1, MX60, MX60W, MX80, and MX90 devices will run MX 14.56.
• When configured for this version, MX400 and MX600 devices will run MX 16.16.

New features

• Added AnyConnect support for MX64(W) and MX65(W) appliances.

Bug fixes

• Fixed a link negotiation issue that could occur when connecting Ethernet ports on MX100 appliances to SFP ports on other devices.
• Update the AnyConnect VPN service
• Corrected an issue that caused MX appliances to drop IPv6 packets with unknown protocols.
• Extended IPv6 extension header validation to also include Hop-by-Hop, Destination Options, Fragment, and Routing headers.
• Resolved an issue that resulted in MX appliances incorrectly dropping link-local pings to itself when the hop limit was set to zero.
• Corrected an issue where MX appliances were not discarding ICMP6 “packet too big” error messages specifying an MTU of less than 1280 bytes.
• Fixed a rare issue that could result in SSIDs configured for WPA PSK authentication being broadcast as open SSIDS on MX67W, and MX68(W,CW) appliances.
• Added logic to drop IPv6 packets that arrive at the MX and are out of place. An ICMPv6 "destination unreachable" message will be sent when these packets are dropped.
• Subnet-router anycast addresses will now be configured for each VLAN interface on MX appliances.
• MX appliances will now send ICMP6 error messages when IPv6 traffic with an incorrect header is received.
• Fixed an issue that could result in IPv6 packets with extension headers being dropped.
• Corrected an issue which resulted in MXs not fragmenting packets whose length exceeds the WAN MTU before encrypting them and sending them over a non-Meraki VPN tunnel.
• Resolved an MX 17 regression which could result in content filtering URL lookups failing when AMP was also enabled.
• MX appliances will now ignore DHCPv6 advertisement messages if those messages do not have prefixes available.
• Fixed an MX 16 regression that resulted in incorrect port LED behavior on MX64(W) appliances.
• Fixed an issue that could result in wireless clients being unable to connect to SSIDs broadcast by MX64W and MX65W appliances that are configured for WPA PSK authentication.
• Resolved an MX 17 regression that resulted in MX appliances configured to operate in VPN concentrator mode being unable to form EBGP connections.
• Corrected inaccuracies in the SNMP ifTable data on MX95/105 appliances.
• Updated the IBGP route behavior for IPv6 routes to mimic the behavior of IPv4.
• Fixed an issue that resulted in IPv6 local network routes not being installed into the IBGP routing table for MX appliances configured in passthrough mode.
• Resolved an issue that could result in the ping live tool not receiving ICMP responses when sourced from a VLAN interface and destined to an IPv6 address.
• Fixed an issue that could result in traffic being improperly routed when LAN clients were using link-local addresses.
• Corrected an issue where duplicate address detection (DAD) failed to detect and mark an address as duplicate when a DAD Neighbor Solicitation message was received.
• Improved performance on MX250 and MX450 appliances when Intrusion detection or prevention were enabled.
• Fixed a rare case where non-Meraki VPN connections would not attempt to form when devices were configed in a warm spare / HA topology.
• Resolved a case that could result in IBGP sessions resetting when using IPv6 and OSPF settings were changed.
• Resolved a case that could cause WAN uplinks to disconnect and reconnect intermittently (“flap”) on MX95, MX105, MX250, and MX450 appliances when an SFP module was plugged into the WAN2 port without anything connected to the module.
• Performance improvements on the MX 95, 105, 250, and 450 appliances.

Known issues

• After making some configuration changes on MX84 appliances, a brief period of packet loss may occur. This will affect all MX84 appliances on all MX firmware versions
• Due to an MX 15 regression, the management port on MX84 appliances does not provide access to the local status page
• Client traffic will be dropped by MX65(W), MX67(C,W), and MX68(W,CW) appliances if 1) The client is connected to a LAN port with 802.1X authentication enabled and 2) The VLAN ID of the port is configured to 16, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, or 240.
• Due to an MX 17 regression, USB cellular modems cannot currently be utilized for primary or backup connectivity on MX84, MX100, MX250, and MX450 appliances.
• MX84 and MX100 appliances have significantly reduced VPN throughput.
• There is an increased risk of encountering device stability and performance issues on all platforms and across all configurations.

Other

• Updated the range of port settings available for MX95 appliances on the device local status page.
• Updated the IP spoofing Event Log messages to be more intuitive and human readable.

[Aaron_Wilson]

"MX84 and MX100 appliances have significantly reduced VPN throughput"

Eek!!!

[HScar]

This is exactly what is holding me back as well. They had this in the beta of 16 and eventually fixed it. I'll be very happy to drop Brightcloud and move to Talos. They're much more on top of it.

[Arsham]

My MX64W does not show any config for Anyconnect

[CptnCrnch]

Its being rolled out in batches (as far as I understood) and not as a one-shot operation worldwide. Just give it a little time. 👍

[ecotter]

I upgraded on Monday, opened a ticket with Support on Tuesday asking why the option isn't there...we're waiting on devs to get back to Support on why the option is absent

edit: think my Support engineer is in the thread 🙂

[AlexP]

Support team forwarded a bug about that this morning - we're looking to get answers ourselves on why it's not visible yet.

[Arshamfa]

Stupid question, do I have to have the licenses for it to show in the dashboard?

[JDomagala]

"MX84 and MX100 appliances have significantly reduced VPN throughput."
What does significantly mean? 5%, 35%?
Please test and provide specific information on VPN throughput limit for those platforms on this firmware.

[naterator]

I've had cellular connectivity problems (via USB modems and built-in modems) on several MX devices after upgrading them to MX 17.6 and created a support case, but am curious if others are experiencing similar odd cellular-related issues on MX 17.6.

Here are the details of my cellular issues, if anyone's curious:

- I upgraded a MX68CW from MX 17.5 to MX 17.6, which broke cellular functionality. The built-in cellular modem always says "connecting", but never gets to "ready" or "active". I rolled back the MX68CW to MX 17.5 and cellular functionality was restored. This device is using the built-in cellular modem.

- I upgraded a MX64W from MX 16.16 to MX 17.6, which broke cellular functionality. The USB modem is not even detected. I rolled back the MX64W to MX 16.16 and cellular functionality was still broken (the "Cellular" section on the appliance page just shows a spinner that never changes). This device is using an Inseego SKYUS DS USB modem.

[harmankardon]

Interesting. I had the opposite result with built-in cellular on our MX67C devices, they wouldn't properly take APN override on MX 15.44 (among other cellular connectivity issues) but when we upgraded to MX 17.6, the APN override worked right away and it resolved a few other cellular issues.

For USB modems, it does say (in the 17.6 patch notes) that certain models don't work with USB cellular but it doesn't list the MX64W, strange. It does seem that Meraki's support for USB cellular modems seems to be lacking ever since they released the built-in models and the MG line. 

Here's a snippet from https://documentation.meraki.com/MX/Cellular/3G%2F%2F4G_Cellular_Failover_with_USB_Modems

"New USB modem approvals are currently on hold until further notice. All ongoing evaluations for USB modems will be updated on this page as soon as completed.  We recommend to upgrade to the integrated/MG models for cellular connectivity and reach out to your sales contact."

That doesn't sound very promising....

[PaulMcG]

Have one customer where 7 MX68CW were upgraded to 17.6 and now 3 are having the same "connecting" issue you describe.  Trying a rollback on one to see if it fixes the issue.

[naterator]

For the MX68CW, we wound up rolling it back to 16.16, then re-upgrading to 17.6 again. At that point, the built-in cellular started to work as expected again.

In our case with Meraki for the MX64W, it definitely does not work on 17.6, no matter how many times you roll it back and re-upgrade it to 17.6. After spending some time with support, they marked it as unexpected behavior and are sending it up to the product team.

[PaulMcG]

Good to know.  This location is actually moving next week and I was hoping to have LTE up and running for the move.  It will give me a chance to try upgrade to 17.6 again to see if I get better results.

[MSchwark]

I have to strongly agree.  How can a release with significantly reduced VPN throughput as a known bug be a stable release candidate?  This statement is concern enough to cause pause. We saw issues with MX16.16 and the SPF issue, reverted all our sites to MX15.44 and then updated one site, low risk, to MX17.6.  I would like to know what significant means before updating our major sites.

[cmr]

@MSchwark MX84 and MX100 are both end of sale, hence the stable release candidate with qualifications state.  I'm guessing we will get more of this as time goes by.

[JDomagala]

Now that is just ridiculous. No matter how many years it will be supported or sold for, what matters is amount of that devices in production. This is most heavily used MX in my organization of 2k+ networks.

So yes, Meraki should spend a couple of days of testing to measure the VPN throughput limitation, probably the meaning of VPN peers/clients has on that limitation too.
Can that be arranged?

[cmr]

@JDomagala I'm a customer too and hopefully you are right.  Our SD-WAN consists of MX84s, MX100s, one pair or MX250s and a smattering of small home user devices.  I won't be moving any of those to 17.x until it is either fixed (🤞) or we get a better idea of the level of reduction.... 

I wasn't excusing Meraki, just explaining how I think it can be classified as it is.  I am running it on an MX84 at a large site as the public internet edge, where the VPN isn't therefore in use, and it has been stable there since 17.2 👍

P.S. in case you weren't sure, the green stars indicate Meraki All Stars, who are all either customers or resellers etc., not employees.  Meraki employees have the green M ✅

[ww]

I used Bonjour forwarding to use chromecast services  between vlans . But it looks broken in 17.x fw. Anyone else have this problem?

[CJHarms]

Our MX250 seems to be ignoring the WAN Bandwith Limits we set for the different WAN Uplinks - anyone else seeing this Problem with the 17.6 Firmware?

[CGRE]

We've just seen issues with 17.6 on MX's with RADIUS based wifi authentiction, it broke the auth piece and we've had to roll back a load of sites as it caused client connection issues, we have a ticket logged on the issue.

I do wish you could opt out of release candidate when on auto update to only get stable version code.

[SJB1]

I have the same issue with an MX68W.  Is there a way to rollback the firmware?

[CGRE]

Yes, you can do this from the dashboard UI easily under firmware if within so many days of the update, if past that just manually force the stable version firmware to the MX.

[CGRE]

Further issue with another customer who was moved to 17.6 now, this time related to content filtering categories changing between Brightcloud and Talos systems, this is mentioned it can change but whats frustrating is I can find no documentation to show a matrix or difference between old Brightcloud and new Talos databases, we've had to roll back a customer due to them complaining of it causing them issues.

[PaulMcG]

Been expecting that to be an issue with certain customers.  Went through the same situation with the updates to Layer 7 when going from 15.x to 16.x.