Solved
There are issues with Willette's topology, and I would not suggest using it. The first is that VRRP heartbeats are sent on all VLANs, so creating a dedicated "heartbeat VLAN" is not actually possible. Second, what you can do is use this dedicated VLAN for DHCP database synchronization... But because there are no actual knobs in the dashboard to configure this the MX will use the link that comes up first. That's great if your dedicated VLAN comes up first, but if not then it could be any of the other links to the switches making this non-deterministic.
IMHO this architecture is incorrect and falsely gives people the impression they are controlling something they cannot actually control.
@PhilipDAth's suggestion is the best topology to use here.
@GreenMan Yup, I'm with you on the simple path part. I'm just saying that the dedicated VLAN over a dedicated link for "heartbeats" is flawed thinking as VRRP doesn't work that way, and you can't deterministically predict where the DB sync traffic is going.
What is needed to complete this setup is a way to flag the heartbeat VLAN as the heartbeat VLAN. Right now there is no such control on the MX.
...And, Meraki has officially changed their documentation on this. The heartbeat cable is no longer a recommended configuration.
Yay!
That page was not updated due to an oversight I believe. They are aware now that it is outstanding and hopefully it'll get changed soon.
In my experience the issues caused by creating the loop at L2 on devices that do not participate in STP are far more detrimental than having VRRP frames pass through one switch between MXes. I would agree that you don't want your VRRP to take the scenic route through your switch fabric to get to the other MX. But if you have problems getting VRRP through a single switch before the dead timer expires then you really have much bigger problems that you need to be looking at.
I think this actually is detrimental to your network. You are creating a "shortcut" path that is not representative of the path your clients will use. This dedicated VLAN heartbeat cable leaves you wide open to the scenario where your clients lose connectivity to the active MX, but the active MX does not relinquish control to the secondary thereby taking your entire network down. You want VRRP to move over the path your clients actually use.
Not to necro this old thread but does any of this discussion change with the recent addition of the cellular MX models?
Yeh that's a nasty little gotcha. I hadn't seen that mentioned anywhere else yet. Good to know.
There is this snippet here on this document:
https://documentation.meraki.com/MX/Deployment_Guides/NAT_Mode_Warm_Spare_(NAT_HA)
Cellular Failover Behavior
Meraki does not currently support any cellular failover with a high availability (HA) pair; as we do not perform connection monitoring on cellular uplinks (as of MX 10.X+), which is necessary for HA uplink failover. At this time, if a cellular uplink is used in an HA pair, the following will occur in order:
Primary MX WAN 1+2 fails > fails over to Secondary MX
Secondary MX WAN 1+2 fails > fails over to Primary MX Cellular
Primary MX cellular fails > fails over to Secondary MX Cellular
While it is possible to use cellular failover as described above, it is not officially supported by Meraki.
I have two MX67C right now and I was testing them. I don't have anything plugged into them yet (no LAN etc.) So I had to do the direct-connect cable between them on port 5. Before I connected them, they both showed as 'Current Master'.
Once I plugged the cable into the spare, it changed to 'Passive; Ready' status.
Running 14.34
Hi Everyone,
@JasonCampbell, @jdsilva, @PhilipDAth @MoBrad
I manage our documentation. Hopefully I can clarify a few things here. First:
@GreenMan wrote:Whilst it's not related to the cabling, please note that, at this point in time, the new MX67 and MX68 models do not yet support VRRP (i.e. warm standby).
This is not correct. MX Cellular models DO support VRRP/warm standby, just not using cellular. You can absolutely set up MX Cellular models in an HA pair configuration, we just recommend doing so without LTE.
This is what your recommended failover options look like:
On the left, you can use MX Cellular models just like any other MX model in a standard HA pair. The left-side design is the recommended HA design for ALL MX models. Notice that the topology matches the documentation. The design on willette.works is not an official Meraki design and is not recommended.
So you might be thinking: "The design on the left is using cellular MX models but isn't using LTE failover at all. What's the point in using cellular models?" - If your priority is HA failover, you may not want to use an MX Cellular model. If your priority is LTE failover, we recommend the design on the right side. If you happen to have an MX Cellular model on hand and WANT to do HA failover, the design on the left side is officially supported and recommended.
The design on the right side is how we officially recommend using LTE failover. On a single MX Cellular device, not in an HA pair.
Alright so, the next question is: "Alright, well what would happen if I DID set up LTE in an HA pair?"
This is answered in the HA documentation, as @NolanHerring pointed out.
Cellular Failover Behavior
Meraki does not currently support any cellular failover with a high availability (HA) pair; as we do not perform connection monitoring on cellular uplinks (as of MX 10.X+), which is necessary for HA uplink failover. At this time, if a cellular uplink is used in an HA pair, the following will occur in order:
Primary MX WAN 1+2 fails > fails over to Secondary MX
Secondary MX WAN 1+2 fails > fails over to Primary MX Cellular
Primary MX cellular fails > fails over to Secondary MX Cellular
What does that actually look like? Here's a diagram to clarify:
This diagram shows what happens when you use LTE in an HA pair. Note the RED TEXT at the top saying that this isn't officially supported or recommended. This is just to clarify current device behavior.
Cheers.
@CameronMoody Great post! Thank you very much for the clarification.
@CameronMoody wrote:
The design on willette.works is not an official Meraki design and is not recommended.
I'm also very glad to see that. I'm not a fan of that configuration at all.
What @PhilipDAth said.
Scroll down and you'll see real world fail over with cellular being used.
https://nolanwifi.com/2018/10/25/you-down-with-l-t-e-yeah-you-know-me-raki/
@CarolineS I actually went the wrong direction again at first. But that road didn't go very far, so I reread the post and got on the right track. After the last one @Adoos found I figured out what you were doing 😉
@LuigiJuve wrote:I tested many scenarios but the best to get this to work is to have a designated vlan set on the MX for the heartbeat for example vlan 1111 ip 1.1.1.1/30.
Can you please show me exactly how you designate a VLAN as heartbeat?
like this
Hi @LuigiJuve ,
No where in there did you designate that VLAN as heartbeat. You simply created a VLAN, assigned it a subnet. Can you please clarify what makes this VLAN a heartbeat VLAN different from all the other VLANs?
How do you control which VLAN is used for hearbeats?
You can't designate. Heartbeats for VRRP are sent out across all VLANs by default, and there is no way to configure them to do otherwise. At this point that designated VLAN is probably redundant logically, maybe not physically if you separated it out (not sure if that is worth it either).
@nickydd9 wrote:I think @jdsilva understands this entirely, he's just trying to get @LuigiJuve to arrive at that answer on his own.... 😃
Shhh! 😉
Hi @LuigiJuve,
I'm not saying it won't work. What I'm saying is that it's not possible to configure a "heartbeat VLAN" as described in that document. It's a completely false pretense to think that this supposedly magical VLAN is any different from any other VLAN you have configured. As @NolanHerring pointed out, VRRP hellos (the "heartbeats") are sent out on every VLAN. Further, Meraki's implmentation of VRRP has elected to use a single Virtual Router with multiple Virtual IP addresses as opposed to a Virtual Router for each VLAN, meaning, that as long as a VRRP heartbeat makes is to the standby unit on ANY vlan the current active master will remain the master for all VLANs. It's not possible for only one VLAN to fail over. This is truly and Active/Standby only implementation.
So please forgive me if this sounds harsh as it's not meant to, but your 15 installations are not working because you followed that guide, and they also are not working the way you think they are working.
This is actually my objection to this configuration, and Aaron Willette's design. It's based on a falsity, and that falsity has been propagated to many corners of the Internet. Quite often people chime in here in the Community and reference that design, and nearly every time the person making the reference has no idea that it's simply not possible to actually designate a VLAN as a "Heartbeat VLAN".
My preferred way to do Meraki Warm Spare, which has come from my own experience deploying Warm Spare for numerous customers, and through the very insightful advice of @PhilipDAth, is to NEVER EVER directly connect the two MX appliances together, and in stead singly connect each MX to a single switch or switch stack (the latter preferred for redundancy). See, having the MX's directly connected actually can cause issues under certain failures with Spanning Tree convergence. Further, it's also desirable to have the VRRP heartbeats traverse a path more representative of the path that user data actually takes. By creating a shortcut that heartbeats can take, but not user traffic, you are actually exposing yourself to a failure scenario where user traffic is blackholed, but VRRP is operation just fine and not allowing a failover to occur.
Failovers are a good thing! You want failover to occur when there's a failure. A direct link cheats the system and can actually prevent a failover when you actually want one.
I'll close this by pointing out that Meraki has actually changed their recommended topology for Warm Spare by removing the direct connection between MX's. It took me some time to convince them to do it, but they did come around 🙂
@RyanB Not at all! Join the conversation and bring forward ideas! If my assessment of that article is incorrect please by all means call me out on it 🙂
@LuigiJuve wrote:
jdsilva, so what do you mean my 15 installations are not working please explain..
Sorry, I think I didn't make that statement very well. I was trying to say that yes, your deployments are probably working just fine under normal network conditions. But, based on your comments around the heatbeat VLAN I do not believe they are working the way you think they are working. If you believe that you have a dedicated VLAN that's being used purely for heartbeats then you are operating under an incorrect assumption, and you don't fully understand how your network is operating. Again, I'm not trying to be harsh, I'm simply trying to point out an incorrect assumption and help you understand the way Warm Spare actually works 🙂
@PhilipDAth wrote:
My understanding is if VRRP fails on any VLAN the entire unit fails over.
I don't think this is true, and the way I read the doc you linked to doesn't support that.
Because the MX uses a single VIrtual Router with multiple Virtual IP's it can't actually detect a failure of a single VLAN. The heartbeat sent on each VLAN is identical to the heartbeat sent on every other VLAN. The only thing that the MX VRRP process is able to detect is whether it received a heartbeat at all within the dead time. As long as any heartbeat reaches the standby it will not assume the Master role. It must have a complete and total loss of all heartbeats on all VLANs to make the transition.
Hi,
I am new to Meraki and working on connecting MX250 to two ISP providers. Each ISP provides one leg connection, so I designed the topology as in this diagram, having a WAN transit switch stack using MS350-24. I read Meraki documentation, all the design topology shows ISP links directly terminating into MX devices, would like to check if this design will work for MX HA setup.
Thanks for your responses. The intent is to have dual internet link load balancing using MX, however each ISP provide one leg to connect. Understand that stacking those switches do not make sense as we will configure different vlan for each ISP links. Have updated the topology as below. We will also be using internal firewall for lan traffic and all internet vlan will bypass internal firewall to MX.
