We upgraded all 4 of our MX84 devices. Seems to be okay. But we have very vanilla configurations on them. Other than HA, we don't have much to break. Our MX65s on the other hand seem to be having and issue timing out. We have an applications that request data over or SD-WAN connections and they connect request the data, and will get the first bit of it, then time out. So we are looking to rollback the firmware to see if that is the root cause, because it started happening after the upgrade to 16.16
We have deployed it to a variety of models from Z3 to MX100, but nearly all are in single LAN mode so we haven't experienced an apparent L3 rule issue.
This moves 16.xx into GA, so for those of us using AnyConnect through the Beta and RC, I know we have to buy AnyConnect licenses, which I have my wholesaler working on now, but how much of a grace period is there, assuming there will be license enforcement now?
After upgrading on 16.16 firmware update, all users lose connectivity to azure app server (ERP program) we receive an error from app that no messages can received on port . Something changed to default policy.
Problem temporary solved with rollback.
Product model MX100
May not be related to your situation, but I had a site that was on 16.14 that was fine until last week and then they started experiencing outages. Upgraded them to 16.16 the night before last to see if it would help, had another outage again today. Outage apparently lasts 5 or 10 minutes.
Checked the logs and it flapped ALL the active interfaces:
It's connected to a stack of 2960S switches and its logs show the link, fluttering at 9:27 then going down at 9:35 then coming back up at the time we see there in the Meraki log. The flutters don't show in the Meraki log, only the result of what appears to be all of the active interfaces resetting including the WAN links.
I opened a support case this AM, we'll see what's going on.
Same issue on one of my MX100s, and had to roll back to 15.44. The other is not dropping like this, but is not routing properly, is blocking VNC intermittently, and Teamviewer is dog slow. Wanted the update for AnyConnect, but that isn't working properly either, so will roll back. I have an open ticket with Meraki, but it isn't looking promising.
Was on 16.16 a few months back on a pair of MX250s. Had both WAN VIPs experiencing high latency and packet loss. Rolled back to 15.44. This week, we upgraded to 16.16.4 and are experiencing the same issues on both WAN VIPs. Did your ticket with Meraki yield any results?
@jtranchina we have two MX HA pairs running 16.x with excellent low latency and no packet loss. They are running the enterprise licence. One is running 16.16.3 and is in routed mode, the other is running 16.15 and is in single arm concentrator mode. End to end latency from devices on one site to the other is 0.4-0.8ms. They are connected by two layer 2 MPLS networks and are about 1-2 miles apart (though of course the networks follow a longer path). The WAN ports use Cisco GLC-T modules and on the routed par we are using the 1Gb/s RJ-45 ports for the LAN.
We have no firewall rules on them and a couple of QoS rules as they aren't Internet facing, below is the VPN chart:
95% of the time we have very minimal latency/packet loss. It appears to spike randomly on both circuits at the same time and requires a reboot of the MX to resolve.
I had a couple networks get NBAR blocks in the event logs, internal VPN traffic between Avaya IP Offices at different sites and internal as well as external DNS requests as a result of upgrading to 16.16
I believe it's the same as whatever it was for the Meraki VPN client, so if that was 500, then that's what it is for AnyConnect.
Meraki MX 16.16 has still a lot of bugs. This Firmware has changed a lot the way how the NBAR categorize the traffic. We are facing problems with traffic being denied by Layer 7 Firewall Rules, which was allowed on the previous firmware. I have open a case to support and they confirmed that there are problems with it but they do not provide any solution. Should we start now to change the configuration of each network one by one in order to fix the problem with MX 16.16?
We are seeing the same thing - Layer 7 firewall rules are denying internal traffic on the site-to-site VPN. It broke connectivity between our Mitel / ShoreTel switches and a LOB application that uses .net TCP services. Appears to be NBAR ID 1889 with a Classification of Statistical Peer-To-Peer.
The solution for us was disabling Layer 7 inspection--specifically we were blocking Peer-To-Peer traffic. For whatever reason, Meraki changed it so that Layer 7 inspection is now occurring on VPN traffic where it wasn't before.
That's where we're running into the issue as well - Peer to Peer blocked over VPN when using a shadowing app. Was hoping for a resolution other than to disable that layer 7 Rule but looks like nothing yet.
Try to Upgrade to 17.9. Meraki says it has fixed a lot of problems with the new Firmware. I have upgraded my MX to this version and it seems that NBAR is working perfectly.
Thanks, yeah after reading the release notes it sounds like that might be the fix. I'll wait until it's in stable vs stable release candidate due to impact on operations for us but this is good to know. Thank you!
With my last support ticket for the NBAR issues we had after upgrading to MX 16.16 on our MX 450, I was advised by Meraki support to stay on MX 15.44. In anticipation of not being able to better control NBAR by allowing allow rules or exceptions, I am researching replacement firewalls for our infrastructure.
Firmware 16.16 still present a lot of issues to Meraki Appliances (mostly on the MX450 and MX100). Until now we have defined three main problems:
- High Device Utilization.
- NBAR blocking traffic because of Layer 7 Firewall rules.
- Instability on the VPN connectivity (Meraki Auto VPN Site to Site and the VPNs created when MX is used as Wireless Concentrator). There are a lot of packet loss and high latency for the traffic that is going through VPN.
Solution that has been tested until now is to downgrade to 15.44 which is the most stable firmware for the Security Appliances (Up to 15-25% reduce of Device Utilization, no packet loss and high latency through the VPN).