Hello. I have two MX250 firewalls set up in a NAT HA failover pair, using the network-connected design for VRRP heartbeats.
Both MX250s have one link connected to WAN1 in the same subnet and I'm using the Virtual-IP for client traffic headed to the internet.
The problems start when I disconnect MX250-Primary-Master's WAN1: the MX250-Spare takes over the master role within seconds. However most clients and switches do not regain internet connectivity- the switches go offline and clients connected to switches have no internet, BUT with the exception of the root switch MS225-24P-2K. The root switch regains internet connectivity and clients behind root switch can also access the internet. But rest of the switches and clients are offline- can not even ping the gateway (gateways are in the MX250). I have included two illustrations of the working setup and the nonworking setup after MX250 failover. I also have an open case with Meraki but no solution yet.
Any ideas what is wrong? Thank you.
Best regards
Heiki
I like where Adam is going with this because that's where my mind went at first too. But if you think about it, the LAN doesn't change when you disconnect the WAN link, so by all rights the STP topology should not be changing either. On the 8 port switch port 9 should still be forwarding, and port 10 should still be blocking. This makes the path to the gateway kinda wonky, but it _should_ still work. So I suspect something else is at play here (Maybe the now standby MX isn't forwarding ARP replies from the now active MX to the 8 port switch???).
I have a couple very similar topologies built, but we use the Meraki recommended practice of connecting the MX's directly.
I am not really a fan of this as the MX's don't participate in STP, but it does seem to work and fails things over correctly. While not providing an explanation to what you are seeing it should be a simple way to stabilize things so you actually have a viable failover.
Yes, previously I had the MXs directly-connected as Meraki documentation seems to be best practice.
However then I had other problems: when disconnecting any switches primary uplink (port 25) the switch and clients behind it lost connection to internet. What I found weird was that the root switch had one port in STP-blocking state which should not happen on a root switch (all ports should be deisgnated-forwarding, unless a loop exists). Googling around I found out that MXs do not participate in spanning tree, thus the MXs caused a spanning tree loop which causes the root switch to block one of its ports. I see that as bad design and suspected that it could cause problems (although spanning tree was working, loops were blocked). After removing the direct-link between MXs the problem was solved- root switch had both it's ports designated-forwarding and no more problems when removing a switch's primary uplink.
The problem of removing MX250-Primary's WAN1 uplink existed in both cases- directly connected and network-connected design.
Actually I have 5 switches connected to MX firewalls via 10Gbit links (all trunks with same VLANs), so the directly connected link actually achieves nothing (most likely at least one of five switches has both uplinks connected to Primary and Spare MX to transfer VRRP heartbeats in each VLAN). The direclty connected link would only be 1Gbit which would be a bottleneck if a switch's uplink to primary-active MX goes down ( solvable with a 10G twinax, although extra cost).
All switches have manually configured STP bridge priorities (of which none are equal: each switch has unique priority).
For some reason the traffic does not flow from nonroot-switch -> MX-Prim (offline) -> Root-switch -> MX-Spare-master.
One more diagram- if I disconnect a switch's primary uplink port9 (root port) then port10 goes from ALT-> Root port and the switch regains connectivity to cloud/internet. This does not explain much.
I am thinking of powering down the entire Meraki network of switches and MXs and then booting them up. Maybe it will resolve some quirks. I did have switches and MXs firmwares recently upgraded but I believe the firmware upgrade process rebooted each device.
Thank you all for input.
@heiki I'd forgotten about this thread! Oddly enough, I've also been revisiting the topic of MS-MX Internet Edge this week and I have a discovery that I think might be related here.
I have observed that when an MX is in standby mode it WILL NOT forward frames that have the destination MAC address set to the address of the VRRP virtual router.This effectively black holes all traffic that is sent to the default gateway.
All other traffic seems to forward normally. You should be able to ping both switches, and the other clients from your client that cannot reach the Internet. At least I could in my testing. If you can't I'd be interested to know that.
Side Note: Meraki does not follow the RFC and uses the physical MAC address of the Primary MX as the MAC address of the virtual router. This means that when the Standby MX becomes Master it responds to the physical MAC of the Primary MX. You can observe this on the clients in the ARP table as the MAC for the GW will be the MAC of the Primary MX, and will not change when you initiate a failover.
**Edit** Meraki actually uses a Meraki OUI based virtual MAC that both MX share.
So, where does this leave us? Well for me I'm going to change the way I connect this stuff up such that there is never a path to the Active MX that is through the Standby MX. This will eliminate the black holing of traffic that occurs when traffic has to go through the Standby MX. I think that this topology is our best bet going forward:
I will likely open a case on this too since this is incorrect behavior. I'm less concerned about the improper VRRP operation than I am about the Standby MX not forwarding traffic for one specific MAC address. You can't run VRRP with another device anyway so it's not that big a deal that Meraki has played fast and loose with the RFC (there's other things they didn't follow as well).
@heiki There's another way you can manipulate this... When there's a failure, if you were to swap port 25 and 26 on the Root Bridge, then things would start working again. By swapping the ports you are forcing the disconnected switch to break the tie for it's root port in favour of the path through the secondary, and now active, MX.
I realize this is likely not useful at all... But I noticed this in my testing so I thought I'd mention it to you as a last resort to get things working if there was an outage.
@PhilipDAth I am not a fan of a topology where multiple switches uplink to both MX's. I think that introduces the potential for STP to become unstable with a topology that has a mesh of neighbours connected over PtP interfaces. I get the sense that's one of those things that would work fine... Until it doesn't. Just my thoughts though.
@PhilipDAth Crap. Good point.
I see the easiest way out here (in the topology I put forward) to simply stack the distribution switches. However, between here and Reddit I've read too many stacking horror stories to comfortably move forward with that option...
Unfortunately, there's real problems with the dual attached option as well, as illustrated by @heiki in this thread. If you ever have a failure such that traffic must pass though a Standby MX to reach the Master MX that traffic is dead in the water.
@paulholloway201 wrote:
You're right buying MS425-32 switches to support all the 10Gbps links from other switches isn't the smartest idea. I think you should read through this link https://www.willette.works/mx-warm-spare/ if you're considering redesign of a setup.
Nope. That design is broken and should not be used.
Hi
We (as an MSP) went through the same questions as you are describing here. Basically the thing bothering us is the loop you create if you follow the Meraki documentation regarding NAT Warm Spare.
Then we followed this: https://willette.works/mx-warm-spare/
And it's basically the missing piece in the documentation. What we did after reading the blog is:
* Shut down the link between the MXs (which was a trunk, allowing all VLANs and since the MX don't do STP basically creating a loop).
* Then create a VLAN on the MX with /30 addressing (like 10.1.1.0/30) and configure the link between the MXes as a pure access port (not trunk) with that VLAN (let's say it's VLAN 161)
Now this VLAN is purely for VRRP communication between the MXes, nothing else. It serves as a way to avoid dual-active scenarios.
* Then prune that VLAN 161 from all other ports, especially those connecting to the switches.
* Prune this VLAN on the switches too connecting to the MXs (just to be sure).
* Then we re-enabled the access port connecting both MXs directly (port 10)
Downstream Switches are usually a stack in our setups unless there's only one single switch of course. We experienced a few problems with stacks too in the earlier firmwares (like 9.32 for example), but with the current stable version (9.36) everything's working as expected, 10+ not so much. That switch stack is usually the root unless the design warrants other switches to be STP root.
If configured that way you see no blocked ports on the switches (ports 47 and 48 are the uplinks to each MX):
Important note:
To avoid having outages in case a single ISP fail, always connect both ISPs to both MXs. This requires at least 2 IP addresses in the same subnet, or even better: use virtual IPs, but you would need at least a /29 per ISP to do it.
The only issue with this design is when you have a problem with the link from the master MX to the switch downstream or from one single MX to the ISP. I think this is very rare but would like to know more if you have concerns regarding this design.
IMHO the Meraki documentation should mention this (prune VRRP VLAN, don't create a loop). In theory a loop should work too since STP would block one of the ports, but it creates all sorts of quirky issues with such a design according to our experience with Meraki.
This setup (without a loop, but a dedicated VRRP VLAN between the MXes) works best for us currently.
HTH 🙂
