A new beta appliance firmware available - MX19.1.4

[rhbirkelund]

Important notice

• As of MX 19.1, Cisco Meraki will no longer support USB-based Cellular Failover on the MX and Z platforms.

What's new

• Added support for configuring eBGP over non-Meraki site-to-site VPN connections.
• Added support for failover (and failback) between non-Meraki VPN tunnels
• MX appliances can now integrate with the Cisco XDR network security product from the Meraki Dashboard
• Significantly expanded the troubleshooting capabilities available from the device local status page
• Improved traffic classification with SD-AVC (Software-Defined Application Visibility and Control)

Supported products notice

• VMX-XL appliances are not yet supported in the MX 19.1 beta. VMX-XL appliances that are configured for this version will not upgrade.

Legacy products notice

• When configured for this version, Z1 devices will run MX 14.56.
• When configured for this version, MX400 and MX600 devices will run MX 16.16.9.
• When configured for this version, MX64(W), MX65(W), MX84, MX100, and vMX100 devices will run MX 18.107.11.

Bug fixes

• Fixed an issue that could result in AutoVPN being disabled when 1) the MX appliance was configured in passthrough mode and 2) DNS caching size parameters were modified in the device configuration.
• Resolved an issue that resulted in jitter and latency spikes on Z3(C) appliances when there was minimal AutoVPN traffic.
• Fixed an issue on MX75, MX85, MX95, MX105, MX250, and MX450 appliances that resulted in AutoVPN traffic being sent to the incorrect destination uplink when 1) the MX appliance was configured to operate as a hub, 2) the MX was configured in passthrough mode, 3) the destination AutoVPN spoke had enabled Active-Active AutoVPN, and 4) the AutoVPN traffic was spoke-to-spoke communication being routed through the AutoVPN hub.
• Resolved an issue that could result in extended periods of incomplete route table updates when a WAN outage occurred if 1) the MX appliance was operating as an AutoVPN hub, 2) the MX appliance was configured in NAT mode, 3) OSPF route advertisement was enabled, and 4) a large number of VPN peers were connected.
• Fixed an additional issue that could degrade the performance of traffic destined to and sourced by MX appliances when 1) IPv6 was enabled, 2) BGP was enabled, and 3) there were over 1024 AutoVPN peers.
• Fixed an issue that could result in the Route Table page failing to display correctly. This was caused by BGP routes with very long AS paths. MX appliances will now only report the first 50 AS paths to the Route Table.
• Corrected an MX 18.2 regression that resulted in MX75, MX85, MX95, MX105, MX250, and MX450 appliances failing to form AutoVPN or teleworker tunnels with other peers via their LAN interfaces.
• Fixed an issue that could result in EBGP peering instability when 1) the MX appliance is configured to operate in routed mode and 2) many EBGP routes are received.
• Fixed several rare cases that could result in MX75, MX85, MX95, MX105, MX250, and MX450 appliances improperly dropping traffic destined to or from a VPN peer.
• Resolved an issue that could result in users encountering failed connections or connections taking an excessive amount of time to complete when using SAML authentication to authenticate an AnyConnect client VPN session.
• Corrected a very rare issue that could cause the AnyConnect VPN process to crash.
• Corrected an issue that could result in group policies failing to apply to AnyConnect VPN clients on MX75, MX85, MX95, MX105, MX250, and MX450 appliances.
• Resolved an issue that would cause intermittent AnyConnect authentication failures when SAML authentication was in use.
• Resolved an issue that could result in client VPN not working when 1) the MX appliance had FIPS mode enabled and 2) the device connecting to the client VPN was a Windows PC.
• Fixed a rare issue that could result in the AnyConnect VPN process becoming unresponsive on MX75 and MX85 appliances.
• Corrected an issue that could result in MX appliances failing to properly form IKEv1 VPN tunnels.
• Corrected an issue that resulted in Non-Meraki VPN tunnels persisting on the backup, non-primary uplink after failing back to the primary uplink when the WAN interface used IPv6.
• Fixed an issue that resulted in the VPN status being reported incorrectly for Non-Meraki VPN peers connected via FQDN.
• Corrected an issue that resulted in MX75, MX85, MX95, MX105, MX250, and MX450 appliances being unable to successfully establish IPSec VPN connections when NAT-T was required to establish the connection.
• Corrected an issue that resulted in the primary MX appliance in a warm spare configuration advertising an incorrect VRRP priority when cellular active uplink was enabled.
• Resolved a rare issue that could result in Z3(C) appliances failing to advertise any SSIDs.
• Cellular reliability improvements for Z3C, MX67C, MX68CW appliances.
• Resolved an issue that could result in MX67C and MX68CW appliances failing to correctly establish cellular connectivity after swapping SIM cards.
• Corrected a rare MX 18.2 regression that could result in MX67C, MX68CW, and Z3C appliances failing to establish a cellular connection using the integrated cellular modem.
• Resolved an issue MX67C, MX68CW, and Z3C appliances may fail to apply IPv4-only custom APNs.
• Fixed an extremely rare issue that could result in an unpredictable set of firewall rule sets to fail to allow traffic as configured. This would be consistently reproducible as long as the firewall configuration remained identical. Since the likelihood of encountering this is extremely low, any modification to the firewall rule set, including changes that would produce no functional change to network operations, would remediate the issue.
• Fixed an issue that could result in MX75, MX85, MX95, MX105, MX250, and MX450 appliances failing to appropriately block or allow IPv6 traffic that relied on L3 FQDN firewall rules.
• Corrected a very rare issue that could result in MX68(W,CW) appliances failing to supply full 802.11at power through PoE.
• Fixed an issue that caused disabled ports to still provide PoE.
• Corrected a rare issue that could result in MX67(C,W), MX68(W,CW), MX75, and MX85 appliances failing to forward traffic on a subset of physical ports. This was most likely to occur after extended periods of uptime.
• Corrected an issue that resulted in MX appliances failing to authorize ports when 1) 802.1X port authentication was configured and 2) the configured RADIUS server sent the Tunnel-Private-Group attribute with a string value.
• Fixed an issue that could result in MX appliances responding to client ARP requests using the WAN MAC address instead of the LAN MAC address.
• Resolved an issue that caused MX appliances to not reply to IPv6 ICMP requests when they were addressed to one of the MX’s VLAN interfaces.
• Resolved a rare issue that resulted in MX appliances failing to block websites when the TLS initialization messages were segmented across multiple packets.
• Fixed an issue that resulted in devices not reporting the category that caused a URL to be blocked by content filtering when in full list mode.
• Fixed a rare issue that could result in a device reboot when IDS/IPS was enabled.
• Fixed an issue that could result in a device reboot when very large or very complex firewall rules were configured in the L3 firewall or the site-to-site VPN firewall.
• Fixed an issue that resulted in MX95 appliances being unable to boot up properly when more than 370 VLANs were configured. Configuring this number of VLANs is not recommended.
• Resolved an issue that could cause MX appliances upgrading to MX 18.2 from very old, unsupported firmware versions to hang during the upgrade process.
• Corrected an issue that could result in brand new MX67(C,W) and MX68(W,CW) appliances taking significantly longer than expected to perform a firmware upgrade.
• Corrected an issue that could result in MX75, MX85, MX95, MX105, MX250, and MX450 appliances failing to forward NTP responses to LAN clients.
• Resolved a flow timeout issue that could result in IPv6 inter-VLAN traffic being incorrectly dropped on MX75, MX85, MX95, MX105, MX250, and MX450 appliances. This was most likely to occur in client communication patterns where one side of the inter-VLAN conversation was “silent” for extended periods of time.
• Fixed an issue that could result in packets being routed incorrectly after 1) a static route was deleted and 2) a VLAN was created using the same subnet as the now-deleted static route. This issue could have been worked-around by rebooting the MX after the configuration change.
• Resolved a rare issue that could result in traffic being improperly dropped by MX75, MX85, MX95, MX105, MX250, and MX450 appliances.
• Resolved an MX 18.211 regression that resulted in MX75, MX85, MX95, MX105, MX250, and MX450 appliances failing to properly forward Radius Authentication traffic when configured as a Wireless Concentrator for SSID Tunneling or Layer 3 Roaming. This caused clients to fail to connect to wireless SSIDs.
• Corrected several packet routing issues that could result in MX75, MX85, MX95, MX105, MX250, and MX450 appliances continuously operating at 100% device utilization.
• Resolved an issue that resulted in the ThousandEyes agent being unable to send traffic over IPsec VPN connections.
• Fixed an issue that could result in MX75, MX85, MX95, MX105, MX250, and MX450 appliances reporting an erroneous spike in network traffic usage.
• Resolved an issue where wireless MX and Z appliances could erroneously report an 802.1x authentication even in the event log when clients connected to an open SSID.
• Corrected an issue that resulted in MX appliances always setting the NAS-IP attribute in RADIUS Access-Request packets to 127.0.0.1. The NAS-IP will now be the IP address of the MX’s lowest numbered VLAN.

Known issues

• There is an increased risk of encountering device stability and performance issues on all platforms and across all configurations.

Other

• When key processes crash, MX appliances will now make a greater effort to restart and recover those processes before initiating a device reboot as a last resort to restore the device to a healthy state.
• Added support for enabling and disabling PoE via the device local status page.
• Reinstated a device reboot that will occur after MX appliances lose connectivity to the Meraki Dashboard for 8 hours. These reboots will continue every 8 hours until Dashboard connectivity is restored.
• The SIM PIN can now be configured via the device local status page without having to first set a custom APN.
• Added SIM configuration information for AT&T FirstNet and TRUE-H cellular operators for MX67C, MX68CW, Z3C, and Z4C appliances.
• Z4(C) appliances will no longer perform a device reboot after FIPS mode is enabled or disabled.

[rhbirkelund]

Long list of bug fixes, but these two new behaviours mentioned in the end are a doozy.

These are really going to mess up customers, who don't expect their MX to randomly reboot because Dashboard Connectivity was lost.

• When key processes crash, MX appliances will now make a greater effort to restart and recover those processes before initiating a device reboot as a last resort to restore the device to a healthy state.
• Reinstated a device reboot that will occur after MX appliances lose connectivity to the Meraki Dashboard for 8 hours. These reboots will continue every 8 hours until Dashboard connectivity is restored.

[RaphaelL]

Came here to comment that. 

What. The. Hell. 

[rhbirkelund]

My thought exactly.

[DarrenOC]

Er! How is that deemed a good idea?  

[thomasthomsen]

I have decided, after thinking about it, that I think this is a good idea.

8 hours without internet connection is a long time.

And we actually had a scenario yesterday where an local ISP did "something" so a lot of this customers Z3C and Z4 gateways on remote locations needed a reboot - this could have solved that problem.

[rhbirkelund]

I can see scenarios where this may be a good thing, but I can definitely also see ones where this may potentially wreck havoc.

[RaphaelL]

Cloud connectivity != Internet connection

Just remember the next dashboard outage and when your MX will reboot every 8 hours , you might not enjoy that.

[thomasthomsen]

I know ..... but have the dashboard ever been away for 8+ hours ?

I mean, this is of course now in "Cisco / Merakis hands" never to have the dashboard be offline for more then 8 hours.

[Mloraditch]

I've definitely seen internet connections where despite the ISP being restored the firewall (of whatever vendor) doesn't come online without a restart.

Outside of an over 8 hour dashboard outage, which certainly could happen, but has not in my memory, I'm curious what other scenarios folks think of that could cause issues? At least with my customers no internet means no operations.

[Brash]

Here to play devil's advocate on these:

• When key processes crash, MX appliances will now make a greater effort to restart and recover those processes before initiating a device reboot as a last resort to restore the device to a healthy state.

It's not uncommon for network equipment to do this, similar to other Linux appliances. In the event of a process crash, a watchdog will attempt to restart it. If it crosses a threshold of multiple crashes within a timeframe or the process cannot be restarted, the watchdog triggers a device crash and restart. I'd rather that occur than certain components silently not working (Eg. firewall rules or IPS not applying, or logging suddenly stopping).

• Reinstated a device reboot that will occur after MX appliances lose connectivity to the Meraki Dashboard for 8 hours. These reboots will continue every 8 hours until Dashboard connectivity is restored.

Despite the fact that I don't necessarily agree with this, I believe it's standard from version 17 and above:
"On firmware versions MX 17+ the MX will reboot after 8 hours without dashboard connectivity to support self-healing" - Behavior during Connection Loss to Cisco Meraki Cloud - Cisco Meraki Documentation

[RaphaelL]

I'm 100% fine with the first point. However , how is rebooting the MX every 8 hours is helping at all ? I don't understand 

[Brash]

Agreed, I'm not a fan of it either.

Just pointing out that I don't think it's new functionality specific to 19.x

[rhbirkelund]

Linux does try to recover services and watchdogs by restarting them. So fair point that it's not uncommon. But I have yet to see a Linux machine reboot completely by itself in order to recover from a services stopping.

[Brash]

True about Linux machines in general.

I was thinking more to Nexus switches which I have managed a lot of in the past. They would reboot with a reboot reason of a hap reset if there was a critical system process that had crashed and the watchdog could not recover it.

Usually this would be related to a bug causing the process to crash which got resolved in later versions.

[rhbirkelund]

And sorry for hijacking your usual good work, @cmr 😉

[cmr]

No worries @rhbirkelund, you did a fine job whilst I was distracted 😎

[RaphaelL]

On a side note , this looks very similar to my old post. We had issues with MX85 not loading up when a large amount of vlans were configured.

• Fixed an issue that resulted in MX95 appliances being unable to boot up properly when more than 370 VLANs were configured. Configuring this number of VLANs is not recommended.

https://community.meraki.com/t5/Security-SD-WAN/Maximum-vlan-limit-on-a-MX/m-p/235244

[DarrenOC]

Who needs 350+ VLANs?

[RaphaelL]

Long story short. "Security" asked to split all IOT devices in groups. HVAC , Cameras , card scanners and so on PER vlan and PER floor. We have a building with 80 floors.  6 groups ( vlans ) x 80 = insane amount of empty vlans 😀. Ugly design that I have to live with for another 12 months.

[DarrenOC]

😂🤣 

I bet those firewall rules were fun as well!

[Brash]

Oh man that's disgusting!
I've been through a similar scenario, and absolutely hate that kind of design.

[cmr]

Oh my, that is just insane! 😮 

[ITSDigital]

I think the auto-reboot function would be great providing:

* You can turn the functionality off in the dashboard with a tickbox per network.

* The reboot reason and crash data is uploaded to the cloud once connectivity is restored.

[thomasthomsen]

I think we can all agree that this is a good idea 🙂

[GIdenJoe]

So let me get this straight.  They start to support an advanced feature like having eBGP over a non Meraki VPN tunnel but they still don't support something basic like a route based non Meraki VPN?  I do question what features get priority...

How about:
- More rule granularity (L7 allow rules), IPS rules only on specific L3/4 rules
- STP and LACP support on the LAN ports
- Bouncing VPN sessions
- Viewing queue status and explicit configuration of the realtime queue.
- Viewing and killing of open firewall sessions.

[RaphaelL]

Wish I could kudo x1000

[cmr]

I wish they had something like Sophos used to, where you could suggest a feature and the community voted on it.  The ones with the top votes got considered and reasonably often got added.  @AmyReyes could we do something here? 🙏

[GIdenJoe]

I'm not throwing stones at Meraki here but I do want to give my opinion.
The place I work at does several vendors for networking and security.
I'm a Cisco only guy, but I can impart my general networking knowledge to my colleagues doing other vendors.
The issue is that the security people also work alot with another vendor that starts with F and ends in T and find it very weird that things they can accomplish easily has no support in Meraki.  And yes this is quite hard to defend it sometimes.

Funny thing was that on my last Cisco live I had a meet the engineer session and he told me they didn't expect that there would be so many people would be asking about LACP support on the MX and that it was pretty low on their priorities list and perhaps has to be reevaluated.  Over half a year later, it's still not on the list...

[cmr]

I always think of F......t as more like FirePower or even SRX, which all need a networking expert.  Whereas Meraki is much easier for generalists, but doesn't have all the bells and whistles that some people want.  Personally I've never had an MX as an enterprise edge device, but they are great for SD-WAN, outbound protection, client VPN, data centre core and all those sorts of things.

[GIdenJoe]

True, I don't expect all the bells and whistles.  However I do expect all basic must haves to finally come on MX'es so I don't have to excuse it all the time.  Without LACP and STP support there are still weird issues happening that can't be helped.

Without more customization on the site 2 site VPN side you can't always integrate with other solutions.

Having no firewall rules on site 2 site non Meraki VPN's is definitely a big no no.

[cmr]

I agree that STP support really is a must, but I've never really missed LACP as the MXs can't generally run faster than a single 10Gb connection anyway and the new MX650 has 25Gb ports to cater for it's improved performance.  But I do agree it should be supported.

[GIdenJoe]

LACP is not used for the speed.  LACP is used so you can make simple bundels of 2 LAN ports to go to two members of the connected stack without having blocked ports.
The whole Meraki design is about simplified campus architecture by using stacks everywhere thus eliminating blocked links.  However the MX appliance does not play ball here.

If we uplink another vendors firewall to the access or distribution stack (yes in smaller networks the firewalls get connected directly to the distribution switch). We can use 1 port channel per firewall and each of them have a port on both stackmembers.  This is the most stable design and does not suffer from transient loops that occur on MX250's in our case each time we add or remove a VLAN.

[cmr]

Do you dual connect the MX250s?  I'd generally just connect one MX to one switch and the other MX to another in the stack.  If STP were supported I would dual connect.

[cmr]

MX75 upgraded 3 days ago, just crashed and rebooted, hopefully not a sign of things to come, 19.1.3 was rather stable for me...🤔

[thomasthomsen]

I had a weird thing happen to me as well.

"Shut / No shut" WAN 2 on my MX85, and it took down the network for, what felt like, a couple of minutes. 😕

WAN2 is connected to a MG, and WAN1 is of course just the normal internet connection, and WAN1 of course has the priority (just to be clear 🙂 )

[rhbirkelund]

A change on a WAN interface incurs a reset on all WAN interfaces. E.g. if you change the IP address on WAN2 - even if it's not connected - will reset both interfaces.

It's expected behaviour - https://documentation.meraki.com/MX/Monitoring_and_Reporting/Appliance_Status/MX_Uplink_Settings#Sec...

[thomasthomsen]

I never noticed that in the documentation.

Oh well .. not a bug ... but a "feature" then ... 🙂

Lol and also the LAN interfaces I see ... what kind of horror is this :

State (enabled/disabled) of any LAN interface

[rhbirkelund]

Neither did I, untill I was preparing a WAN port for a Internet circuit switch to a tagged VLAN, and my MX went Offline. I opened a case with Meraki, who then pointed me towards the behaviour. 🙂

I wasn't impressed either, to say the least.

[AnythingHosted]

Think I will wait before upgrading my MX75 

[jimmyt234]

Security and SD-WAN (MX,Z) Features Directory - Cisco Meraki Documentation

There is also now public documentation on the new features in the beta release, this info was lacking until recently.

[bpar]

Once "eBGP over non-Meraki site-to-site VPN connections" is configured on Hub MX, can those 3rd party prefixes/routes learned on MX Hub be propagated to other MX branches over Auto VPN?