New MX 16.16 stable firmware released - lots of fixes!

[cmr]

Security appliance firmware versions MX 16.16 changelog

Important notice

• While Meraki appliances have traditionally relied on UDP port 7351 for cloud communication and TCP ports 80 and 443 for backup communications, with MX 16 we are beginning a transition to using TCP port 443 as the primary means for cloud connectivity. In order to ensure proper connectivity to the Meraki cloud after this upgrade, please ensure that traffic using TCP port 443 between 209.206.48.0/20 is allowed through any firewalls that may be deployed upstream of your Meraki appliances.
• HTTP proxy, which allows default management traffic from MX appliances to be sent through a proxy, is deprecated on MX 16 and higher firmware versions.

Legacy products notice

• When configured for this version, Z1, MX60, MX60W, MX80, and MX90 devices will run MX 14.56.

New feature highlights

• Added support for Cisco AnyConnect client VPN on Z3(C), MX67(C,W), MX68(W,CW), MX75, MX84, MX85, MX95, MX100, MX105, MX400, MX600, MX250, and MX450 appliances.
• Added Network-Based Application Recognition (NBAR) integration.
• Added support for using a cellular uplink concurrently with a wired uplink, as opposed to the cellular uplink only being available for failover.
• Added firmware support for SD-Internet functionality

Bug fixes

• Corrected several cases where disabled ports on MX95, MX105, MX250, and MX450 would be in a disconnected state as opposed to fully disabled. This could cause devices connected to disabled ports to show the connection as active.
• Corrected several cases where wireless SSIDs would reset during a WAN or cellular failover event for MXW appliances that were configured to operate in warm spare / high availability.
• Corrected an issue that resulted in the user name being displayed incorrectly on the Clients page in Dashboard for AnyConnect VPN clients connected via SAML authentication.
• Further upgrade reliability improvements for MX64(W) and MX65(W) appliances.
• TCP and UDP Performance improvements in cases where many clients are transmitting traffic.
• Resolved an issue that could affect the reachability of subsets of AutoVPN routes if two MX appliances were joined to the AutoVPN topology using the same IP address.
• Fixed a rare case that could result in a device reboot when ICMP traffic was sent from a ClientVPN client to a non-Meraki VPN peer.
• Performance improvements on the MX 95, 105, 250, and 450 appliances.
• Corrected an issue that could cause SFP links to fail to establish if the modules were hotplugged into MX95, MX105, MX250, and MX450 appliance ports.
• Resolved a case that could cause WAN uplinks to disconnect and reconnect intermittently (“flap”) on MX95, MX105, MX250, and MX450 appliances when an SFP module was plugged into the WAN2 port without anything connected to the module.
• Fixed a rare case where non-Meraki VPN connections would not attempt to form when devices were configed in a warm spare / HA topology.
• Corrected an issue that resulted in traffic not being properly routed when 1) An MX was configured with a specific ECO-only static routing configuration and 2) a WAN uplink had a failover event and subsequently recovered from the failover (failback).
• Corrected inaccuracies in the SNMP ifTable data on MX95/105 appliances.
• Fixed an MX 16 regression that resulted in incorrect port LED behavior on MX64(W) appliances.
• Resolved an issue which resulted in MXs not fragmenting packets whose length exceeds the WAN MTU before encrypting them and sending them over a non-Meraki VPN tunnel.
• Improved the reliability of application classification decisions made by the NBAR traffic engines.
• Stability improvements for all platforms

Known issues

• After making some configuration changes on MX84 appliances, a brief period of packet loss may occur. This will affect all MX84 appliances on all MX firmware versions
• Due to MX 15 regressions, USB cellular connectivity may be less reliable on some modems
• Due to an MX 15 regression, the management port on MX84 appliances does not provide access to the local status page
• Client traffic will be dropped by MX65(W), MX67(C,W), and MX68(W,CW) appliances if 1) The client is connected to a LAN port with 802.1X authentication enabled and 2) The VLAN ID of the port is configured to 16, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, or 240.

Other

• Added the ability to set static speed and duplex settings on LAN SFP ports on MX85 appliances.
• Updated the IP spoofing Event Log messages to be more intuitive and human readable

[CharlesIsWorkin]

Anyone with an MX84 give this a try?

[jrsilvius]

We upgraded all 4 of our MX84 devices. Seems to be okay. But we have very vanilla configurations on them. Other than HA, we don't have much to break. Our MX65s on the other hand seem to be having and issue timing out. We have an applications that request data over or SD-WAN connections and they connect request the data, and will get the first bit of it, then time out. So we are looking to rollback the firmware to see if that is the root cause, because it started happening after the upgrade to 16.16

[cmr]

We have deployed it to a variety of models from Z3 to MX100, but nearly all are in single LAN mode so we haven't experienced an apparent L3 rule issue.

[OVERKILL]

This moves 16.xx into GA, so for those of us using AnyConnect through the Beta and RC, I know we have to buy AnyConnect licenses, which I have my wholesaler working on now, but how much of a grace period is there, assuming there will be license enforcement now? 

[Lefteris]

Hello

After upgrading on 16.16 firmware update, all users lose connectivity to azure app server (ERP program) we receive an error from app that no messages can received on port [8110]. Something changed to default policy.

Problem temporary solved with rollback.

Product model MX100

[IanJSaul]

Deployed on an MX100 @ 4:30AM - by 5:40AM repeated 10-12 minute outages - almost every single hour.

Rolled back and testing now.

[OVERKILL]

May not be related to your situation, but I had a site that was on 16.14 that was fine until last week and then they started experiencing outages. Upgraded them to 16.16 the night before last to see if it would help, had another outage again today. Outage apparently lasts 5 or 10 minutes. 

Checked the logs and it flapped ALL the active interfaces:

It's connected to a stack of 2960S switches and its logs show the link, fluttering at 9:27 then going down at 9:35 then coming back up at the time we see there in the Meraki log. The flutters don't show in the Meraki log, only the result of what appears to be all of the active interfaces resetting including the WAN links. 

I opened a support case this AM, we'll see what's going on. 

[BB97]

Same issue on one of my MX100s, and had to roll back to 15.44. The other is not dropping like this, but is not routing properly, is blocking VNC intermittently, and Teamviewer is dog slow. Wanted the update for AnyConnect, but that isn't working properly either, so will roll back. I have an open ticket with Meraki, but it isn't looking promising.

[jtranchina]

Was on 16.16 a few months back on a pair of MX250s. Had both WAN VIPs experiencing high latency and packet loss. Rolled back to 15.44. This week, we upgraded to 16.16.4 and are experiencing the same issues on both WAN VIPs. Did your ticket with Meraki yield any results? 

[cmr]

@jtranchina we have two MX HA pairs running 16.x with excellent low latency and no packet loss.  They are running the enterprise licence.   One is running 16.16.3 and is in routed mode,  the other is running 16.15 and is in single arm concentrator mode.   End to end latency from devices on one site to the other is 0.4-0.8ms.  They are connected by two layer 2 MPLS networks and are about 1-2 miles apart (though of course the networks follow a longer path).  The WAN ports use Cisco GLC-T modules and on the routed par we are using the 1Gb/s RJ-45 ports for the LAN.

We have no firewall rules on them and a couple of QoS rules as they aren't Internet facing,  below is the VPN chart:

 

[jtranchina]

95% of the time we have very minimal latency/packet loss. It appears to spike randomly on both circuits at the same time and requires a reboot of the MX to resolve.

[mwiater]

I had a couple networks get NBAR blocks in the event logs, internal VPN traffic between Avaya IP Offices at different sites and internal as well as external DNS requests as a result of upgrading to 16.16

[MerakiSA]

Is there current sizing guidance on the MX400 for client vpn using AnyConnect?

[OVERKILL]

I believe it's the same as whatever it was for the Meraki VPN client, so if that was 500, then that's what it is for AnyConnect. 

[Armelin]

Meraki MX 16.16 has still a lot of bugs. This Firmware has changed a lot the way how the NBAR categorize the traffic.  We are facing problems with traffic being denied by Layer 7 Firewall Rules, which was allowed on the previous firmware. I have open a case to support and they confirmed that there are problems with it but they do not provide any solution. Should we start now to change the configuration of each network one by one in order to fix the problem with MX 16.16?

[Chad-CLE]

We are seeing the same thing - Layer 7 firewall rules are denying internal traffic on the site-to-site VPN.  It broke connectivity between our Mitel / ShoreTel switches and a LOB application that uses .net TCP services.  Appears to be NBAR ID 1889 with a Classification of Statistical Peer-To-Peer.

[DylanO]

Did you ever figure out a solution to this? We're having the issue when trying to use our remote support tool. 

[Chad-CLE]

The solution for us was disabling Layer 7 inspection--specifically we were blocking Peer-To-Peer traffic.  For whatever reason, Meraki changed it so that Layer 7 inspection is now occurring on VPN traffic where it wasn't before.

[DylanO]

That's where we're running into the issue as well - Peer to Peer blocked over VPN when using a shadowing app. Was hoping for a resolution other than to disable that layer 7 Rule but looks like nothing yet.

[Armelin]

Try to Upgrade to 17.9. Meraki says it has fixed a lot of problems with the new Firmware. I have upgraded my MX to this version and it seems that NBAR is working perfectly.

[DylanO]

Thanks, yeah after reading the release notes it sounds like that might be the fix. I'll wait until it's in stable vs stable release candidate due to impact on operations for us but this is good to know. Thank you! 

[Jordan1]

Anyone know if these NBAR issues with 16.16 have been resolved in 16.16.1? 

[Midway1]

With my last support ticket for the NBAR issues we had after upgrading to MX 16.16 on our MX 450, I was advised by Meraki support to stay on MX 15.44. In anticipation of not being able to better control NBAR by allowing allow rules or exceptions, I am researching replacement firewalls for our infrastructure.

[Jordan1]

Got it. Thanks for the update. 

[jrsilvius]

We are rolling back to 15.44 .16.16 broke so many things on our network. I don't think it is ready for release.

[Armelin]

Firmware 16.16 still present a lot of issues to Meraki Appliances (mostly on the MX450 and MX100). Until now we have defined three main problems:
- High Device Utilization.
- NBAR blocking traffic because of Layer 7 Firewall rules.
- Instability on the VPN connectivity (Meraki Auto VPN Site to Site and the VPNs created when MX is used as Wireless Concentrator). There are a lot of packet loss and high latency for the traffic that is going through VPN.

Solution that has been tested until now is to downgrade to 15.44 which is the most stable firmware for the Security Appliances (Up to 15-25% reduce of Device Utilization, no packet loss and high latency through the VPN).