cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

How to cable MX & MS for HA

SOLVED
Conversationalist

How to cable MX & MS for HA

What's the recommended way to cable and configure 2x MX250 operating in HA NAT, and connecting to 2xMS350-24 which are stacked?

 

My thoughts are that this requires 4 GbE connections.... Would aggregation be required for the 4 ports?

Primary MX

GbE 3 to MS350-1, port 1

GbE 4 to MS350-2, port 2

 

Spare MX

GbE 3 to MS350-2, port 1

GbE 4 to MS350-1, port 2

1 ACCEPTED SOLUTION

Accepted Solutions
Kind of a big deal

Re: How to cable MX & MS for HA

The way you have suggested is the way I do it.

 

However Meraki's recommendation is a cable from each MX to a switch, and then a cable directly between the MX's.

https://documentation.meraki.com/MX-Z/Other_Topics/Troubleshooting_MX_Warm_Spare_in_NAT_Mode_(NAT_HA...

I don't agree with this approach, because you often get all the traffic being switched through the spare to get to the active unit thanks to spanning tree often choosing to block the link to the active MX.

 

mv-was-here.png

54 REPLIES 54
Kind of a big deal

Re: How to cable MX & MS for HA

The way you have suggested is the way I do it.

 

However Meraki's recommendation is a cable from each MX to a switch, and then a cable directly between the MX's.

https://documentation.meraki.com/MX-Z/Other_Topics/Troubleshooting_MX_Warm_Spare_in_NAT_Mode_(NAT_HA...

I don't agree with this approach, because you often get all the traffic being switched through the spare to get to the active unit thanks to spanning tree often choosing to block the link to the active MX.

 

mv-was-here.png

Conversationalist

Re: How to cable MX & MS for HA

Thanks @PhilipDAth. But would you configure all 4 ports to be in 1 aggregate on the MS stack?

I've actually got a direct connection between the 2 MX's as well but have set this to be access with only a non-routable VLAN which has been pruned from the switch uplinks. The issues that you describe are exactly what we've experienced recently so am looking to sort things out.

Kind of a big deal

Re: How to cable MX & MS for HA

The MX does not support port aggregation - so I would not.

Kind of a big deal

Re: How to cable MX & MS for HA

I'd do what you suggested with twinax cables for high throughput. 

 

Adam R MS | CISSP, CISM, VCP, MCITP, CCNP, ITILv3, CMNO
If this was helpful click the Kudo button below
If my reply solved your issue, please mark it as a solution.
Here to help

Re: How to cable MX & MS for HA

Found the Gnome 

Meraki Employee

Re: How to cable MX & MS for HA

A couple of thoughts to add:

MX doesn't run STP itself, but it will forward BPDUs, so if you create any loops, they'd need to be resolved in the switching.  Probably best not to create them in the first place.

Any heartbeat link directly between the MXs should be in a dedicated VLAN.

 

In addition to the published MX documentation.meraki.com this is a useful unofficial resource (though created by a Meraki SE):  https://www.willette.works/mx-warm-spare/ 

Kind of a big deal

Re: How to cable MX & MS for HA

@GreenMan

 

There are issues with Willette's topology, and I would not suggest using it. The first is that VRRP heartbeats are sent on all VLANs, so creating a dedicated "heartbeat VLAN" is not actually possible. Second, what you can do is use this dedicated VLAN for DHCP database synchronization... But because there are no actual knobs in the dashboard to configure this the MX will use the link that comes up first. That's great if your dedicated VLAN comes up first, but if not then it could be any of the other links to the switches making this non-deterministic. 

 

IMHO this architecture is incorrect and falsely gives people the impression they are controlling something they cannot actually control. 

 

@PhilipDAth's suggestion is the best topology to use here. 

Meraki Employee

Re: How to cable MX & MS for HA

Hi @jdsilva - I should probably have called it a heartbeat link (although it would need a dedicated VLAN.) The idea of that path is for it to be as simple as possible (least likely to fail), avoiding dual-active MX scenarios.   There are indeed a number of ways of engineering such setups - I guess testing your preferred approach, in your customers actual network, taking into account likely failure scenarios (perhaps using a free trial) is always the best recommendation, rather than being fixed on any one topology as ‘best’.

Kind of a big deal

Re: How to cable MX & MS for HA

@GreenMan Yup, I'm with you on the simple path part. I'm just saying that the dedicated VLAN over a dedicated link for "heartbeats" is flawed thinking as VRRP doesn't work that way, and you can't deterministically predict where the DB sync traffic is going. 

 

What is needed to complete this setup is a way to flag the heartbeat VLAN as the heartbeat VLAN. Right now there is no such control on the MX.

Kind of a big deal

Re: How to cable MX & MS for HA

...And, Meraki has officially changed their documentation on this. The heartbeat cable is no longer a recommended configuration.

 

https://documentation.meraki.com/MX-Z/Deployment_Guides/NAT_Mode_Warm_Spare_(NAT_HA)#Recommended_Top...

 

Yay!

Getting noticed

Re: How to cable MX & MS for HA

My thoughts:

While VRRP packets will flow thru all VLANs, having a dedicated physical link on its own dedicated VLAN that VRRP packets flow thru allow for the shortest path on a VLAN that is exclusively VRRP packets. True, VRRP will go out all VLANs -- but in case of any sort of congestion or link failure in the switch stack, you have a dedicated link and VLAN that will still allow VRRP packets to make it to the warm spare. So, IMO, it is still advisable to use a dedicated link with a dedicated VLAN to ensure timely arrival of VRRP packets to the warm spare without having to worry about the rest of the network.

And Meraki has not changed their documentation, at least not fully, on this.

https://documentation.meraki.com/MX-Z/Other_Topics/Troubleshooting_MX_Warm_Spare_in_NAT_Mode_(NAT_HA)

Kind of a big deal

Re: How to cable MX & MS for HA

That page was not updated due to an oversight I believe. They are aware now that it is outstanding and hopefully it'll get changed soon.

 

In my experience the issues caused by creating the loop at L2 on devices that do not participate in STP are far more detrimental than having VRRP frames pass through one switch between MXes. I would agree that you don't want your VRRP to take the scenic route through your switch fabric to get to the other MX. But if you have problems getting VRRP through a single switch before the dead timer expires then you really have much bigger problems that you need to be looking at.

 

 

Kind of a big deal

Re: How to cable MX & MS for HA

@JasonCampbell you say:

"While VRRP packets will flow thru all VLANs, having a dedicated physical link on its own dedicated VLAN that VRRP packets flow thru allow for the shortest path on a VLAN that is exclusively VRRP packets. True, VRRP will go out all VLANs -- but in case of any sort of congestion or link failure in the switch stack, you have a dedicated link and VLAN that will still allow VRRP packets to make it to the warm spare. So, IMO, it is still advisable to use a dedicated link with a dedicated VLAN to ensure timely arrival of VRRP packets to the warm spare without having to worry about the rest of the network."

 

I don't think you understand the purpose of VRRP.  VRRP is a protocol to provide protection for the default gateway of a VLAN.  It allows clients configured with that default gateway to remain working during a failure.

 

Having VRRP running on a dedicated VLAN between two MX units is - pointless.

Getting noticed

Re: How to cable MX & MS for HA


@PhilipDAth wrote:

@JasonCampbell you say:

"While VRRP packets will flow thru all VLANs, having a dedicated physical link on its own dedicated VLAN that VRRP packets flow thru allow for the shortest path on a VLAN that is exclusively VRRP packets. True, VRRP will go out all VLANs -- but in case of any sort of congestion or link failure in the switch stack, you have a dedicated link and VLAN that will still allow VRRP packets to make it to the warm spare. So, IMO, it is still advisable to use a dedicated link with a dedicated VLAN to ensure timely arrival of VRRP packets to the warm spare without having to worry about the rest of the network."

 

I don't think you understand the purpose of VRRP.  VRRP is a protocol to provide protection for the default gateway of a VLAN.  It allows clients configured with that default gateway to remain working during a failure.

 

Having VRRP running on a dedicated VLAN between two MX units is - pointless.


Hi Philip,

 
I understand VRRP as a first hop redundancy protocol that, not by standard, but on Meraki equipment, VRRP heartbeats are sent out all Vlans and as long as the warm spare finds one of these, it considers primary still live. Having a VLAN with absolutely no traffic theoretically provides benefits to this VRRP heartbeat packet. Please feel free to provide evidence if I'm wrong. 
Kind of a big deal

Re: How to cable MX & MS for HA

I think this actually is detrimental to your network. You are creating a "shortcut" path that is not representative of the path your clients will use. This dedicated VLAN heartbeat cable leaves you wide open to the scenario where your clients lose connectivity to the active MX, but the active MX does not relinquish control to the secondary thereby taking your entire network down. You want VRRP to move over the path your clients actually use. 

Kind of a big deal

Re: How to cable MX & MS for HA

@JasonCampbell the only test that is used to determine if a unit is alive is weather it can talk to the cloud.  VRRP is used strictly for a FHRP.

 

An MX only participates in VRRP if it can talk to the cloud.  If it loose that connection it stops speaking VRRP completely.

Conversationalist

Re: How to cable MX & MS for HA

Appreciate the great discussion here. Whilst I'd originally planned to replicate https://willette.works/mx-warm-spare/ I'll now change our approach back to the updated method without the direct-connect cable. Cheers all.

Meraki Employee

Re: How to cable MX & MS for HA

While VRRP does indeed provide a resilient next-hop (either for clients or, for another example, an upstream router) it's also used for the two MXs to monitor each other.   If you have all your inside VLANs running over the same physical infrastructure (switches / fibre etc.) then a failure within that layer could result in both MXs becoming active.  Having as direct a path as possible between the two, separate from that shared infrastructure, to prevent active-active, is the basic aim of such a link.

Kind of a big deal

Re: How to cable MX & MS for HA

Not to necro this old thread but does any of this discussion change with the recent addition of the cellular MX models?

Nolan Herring | nolanwifi.com
TwitterLinkedIn
Kind of a big deal

Re: How to cable MX & MS for HA

No.

Meraki Employee

Re: How to cable MX & MS for HA

Whilst it's not related to the cabling, please note that, at this point in time, the new MX67 and MX68 models do not yet support VRRP (i.e. warm standby).  This will be added in a future firmware release.   Whilst writing, the same goes for wired 802.1x

Kind of a big deal

Re: How to cable MX & MS for HA

>Whilst it's not related to the cabling, please note that, at this point in time, the new MX67 and MX68 models do not yet support VRRP (i.e. warm standby).  This will be added in a future firmware release.   Whilst writing, the same goes for wired 802.1x

 

Wow, I mean wow!  I haven't heard that one.

Kind of a big deal

Re: How to cable MX & MS for HA

Yeh that's a nasty little gotcha. I hadn't seen that mentioned anywhere else yet. Good to know.

Getting noticed

Re: How to cable MX & MS for HA

Seriously? Lovely how that's not mentioned anywhere in the product documentation.
Kind of a big deal

Re: How to cable MX & MS for HA

There is this snippet here on this document:

 

https://documentation.meraki.com/MX/Deployment_Guides/NAT_Mode_Warm_Spare_(NAT_HA)

  

 

Cellular Failover Behavior
Meraki does not currently support any cellular failover with a high availability (HA) pair; as we do not perform connection monitoring on cellular uplinks (as of MX 10.X+), which is necessary for HA uplink failover. At this time, if a cellular uplink is used in an HA pair, the following will occur in order:

 

Primary MX WAN 1+2 fails > fails over to Secondary MX
Secondary MX WAN 1+2 fails > fails over to Primary MX Cellular
Primary MX cellular fails > fails over to Secondary MX Cellular


While it is possible to use cellular failover as described above, it is not officially supported by Meraki.

 

Nolan Herring | nolanwifi.com
TwitterLinkedIn
Getting noticed

Re: How to cable MX & MS for HA

That has nothing to do with the fact that VRRP isn't supported AT ALL on MX67/MX68 currently. That should have been made more prevalent.
Meraki Employee

Re: How to cable MX & MS for HA

VRRP is only used as part of the warm standby mechanism, on MX

Getting noticed

Re: How to cable MX & MS for HA

I don't understand what that means. MX67/68 is able to do warm standby without VRRP? How does that work?
Kind of a big deal

Re: How to cable MX & MS for HA

I have two MX67C right now and I was testing them. I don't have anything plugged into them yet (no LAN etc.) So I had to do the direct-connect cable between them on port 5. Before I connected them, they both showed as 'Current Master'.

 

Once I plugged the cable into the spare, it changed to 'Passive; Ready' status.

 

Running 14.34

 

vrrp.jpg

 

 

 

 

Nolan Herring | nolanwifi.com
TwitterLinkedIn
Highlighted
Meraki Employee

Re: How to cable MX & MS for HA

Hi Everyone,

 

@JasonCampbell@jdsilva@PhilipDAth @MoBrad

 

I manage our documentation. Hopefully I can clarify a few things here. First:

 


@GreenMan wrote:

Whilst it's not related to the cabling, please note that, at this point in time, the new MX67 and MX68 models do not yet support VRRP (i.e. warm standby).  


This is not correct. MX Cellular models DO support VRRP/warm standby, just not using cellular. You can absolutely set up MX Cellular models in an HA pair configuration, we just recommend doing so without LTE.

 

This is what your recommended failover options look like:

 

MXC-Recommended-Failover_Designs.png

On the left, you can use MX Cellular models just like any other MX model in a standard HA pair. The left-side design is the recommended HA design for ALL MX models. Notice that the topology matches the documentation. The design on willette.works is not an official Meraki design and is not recommended.

 

So you might be thinking: "The design on the left is using cellular MX models but isn't using LTE failover at all. What's the point in using cellular models?" - If your priority is HA failover, you may not want to use an MX Cellular model. If your priority is LTE failover, we recommend the design on the right side. If you happen to have an MX Cellular model on hand and WANT to do HA failover, the design on the left side is officially supported and recommended.

 

The design on the right side is how we officially recommend using LTE failover. On a single MX Cellular device, not in an HA pair.

 

Alright so, the next question is: "Alright, well what would happen if I DID set up LTE in an HA pair?"

This is answered in the HA documentation, as @NolanHerring pointed out.

 

Cellular Failover Behavior
Meraki does not currently support any cellular failover with a high availability (HA) pair; as we do not perform connection monitoring on cellular uplinks (as of MX 10.X+), which is necessary for HA uplink failover. At this time, if a cellular uplink is used in an HA pair, the following will occur in order:

 

Primary MX WAN 1+2 fails > fails over to Secondary MX
Secondary MX WAN 1+2 fails > fails over to Primary MX Cellular
Primary MX cellular fails > fails over to Secondary MX Cellular

 

What does that actually look like? Here's a diagram to clarify:

 

MXC-HA-LTE-Failover_Behavior-Not_Recommended.png

 

This diagram shows what happens when you use LTE in an HA pair. Note the RED TEXT at the top saying that this isn't officially supported or recommended. This is just to clarify current device behavior.

 

Cheers.

Cameron Moody | Documentation Manager, Cisco Meraki
Kind of a big deal

Re: How to cable MX & MS for HA


@CameronMoody Great post! Thank you very much for the clarification.

 

@CameronMoody wrote:

 

The design on willette.works is not an official Meraki design and is not recommended.

 

 


I'm also very glad to see that. I'm not a fan of that configuration at all. 

Kind of a big deal

Re: How to cable MX & MS for HA

Thank you very much for the response Cameron !

Nolan Herring | nolanwifi.com
TwitterLinkedIn
Getting noticed

Re: How to cable MX & MS for HA

@CameronMoody

Thanks for the clarification. That makes much more sense.

Meraki Employee

Re: How to cable MX & MS for HA

Many thanks to @CameronMoody for the clarification;  FYI he's also arranged a tweak to some of our internal information, that lead to my ovely simplistic (OK, OK;  inaccurate) comment regarding VRRP.

Conversationalist

Re: How to cable MX & MS for HA

@CameronMoody 

 

Is there a plan to eventually support a cellular HA pair? We currently have some Cisco 2921 routers that have an LTE failover setup. Was hoping this would come to our meraki devices as well. I would be willing to test some beta firmware if/when it becomes available.

 

Thanks for the informative thread!

Kind of a big deal

Re: How to cable MX & MS for HA

You can use cellular in an HA pair - just the cellular links are the last to be used.

 

Note you can't do "HA NAT" as the cellular interfaces can not share an IP address (the same as your 2900's), but it will still fail over. Any in-progress TCP sessions will be lost and will need to be restarted.

Kind of a big deal

Re: How to cable MX & MS for HA

What @PhilipDAth said.

 

Scroll down and you'll see real world fail over with cellular being used.

 

https://nolanwifi.com/2018/10/25/you-down-with-l-t-e-yeah-you-know-me-raki/

 

 

Nolan Herring | nolanwifi.com
TwitterLinkedIn
Meraki Employee

Re: How to cable MX & MS for HA

@NolanHerring That's an awesome blog page. Super kudos.

 

We did fix the LED lights in the install guide, good catch.

Cameron Moody | Documentation Manager, Cisco Meraki
Meraki Employee

Re: How to cable MX & MS for HA

@Kirk Regarding VRRP over LTE, I don't have that information, sorry!

 

Nolan has some clever suggestions for workarounds for now though.

Cameron Moody | Documentation Manager, Cisco Meraki
Kind of a big deal

Re: How to cable MX & MS for HA

I found the gnome!!!

Meraki Alumni (Retired)

Re: How to cable MX & MS for HA

YEAH! Nice work, @jdsilva! I think my hint was a bit too helpful... now I'll return MV gnome to my bag and fasten it tightly!

Caroline S | Community Manager, Cisco Meraki | @merakicaroline
New to the community? Get started here
Kind of a big deal

Re: How to cable MX & MS for HA

@CarolineS I actually went the wrong direction again at first. But that road didn't go very far, so I reread the post and got on the right track. After the last one @Adoos found I figured out what you were doing 😉

Head in the Cloud

Re: How to cable MX & MS for HA

Doh! Just moments too late for the gnome..

Meraki Alumni (Retired)

Re: How to cable MX & MS for HA

Nice try, @BrandonS! I bet MV will get restless tomorrow at the office too. Just a hunch.

Caroline S | Community Manager, Cisco Meraki | @merakicaroline
New to the community? Get started here
Meraki Employee

Re: How to cable MX & MS for HA

Nothing to see here.. haha.

Kind of a big deal

Re: How to cable MX & MS for HA

@RyanB I dispute it's greatness. See my comments above 🙂

Meraki Employee

Re: How to cable MX & MS for HA

I'll see myself out! Great discussion.

Kind of a big deal

Re: How to cable MX & MS for HA

@RyanB Not at all!  Join the conversation and bring forward ideas!  If my assessment of that article is incorrect please by all means call me out on it 🙂

 

 

Here to help

Re: How to cable MX & MS for HA

Sorry to dig up this old thread, but hoping someone can clarify something for me.

 

I'm looking to replicate the 'Fully Redundant (Multiple Switches)' setup from the HA documentation

using dual WAN links, two MX100s and two MS120s. However, the documentation doesn't really detail how WAN1 and WAN2 connect to both MXs.

 

From what I understand the willette.works design is not supported because it recommends a direct connection between the MXs for the VRRP heartbeats. However, earlier in the post, he talks about splitting the two WAN links using a breakout switch so that both MXs have a connection to both WAN links. Is this a good way of doing it? 

 

 

Newbie here - go easy please 🙂

Welcome to the Meraki Community!
To start contributing, simply sign in with your Cisco account. If you don't yet have a Cisco account, you can sign up.