How to cable MX & MS for HA

SOLVED
PhilipDAth
Kind of a big deal

Re: How to cable MX & MS for HA

Typically you get your ISP to provide at least two ports in the same VLAN, and you plug WAN1 from both MX into that device.  Ditto for the second ISP.

 

You can also use an external switch for this (plug in ISP and WAN1 on both MX).  Repeat for WAN2.

 

You can also use an internal switch.  Create a VLAN and put three ports into it in access mode.  Plug in ISP1 and the WAN1 ports of box MX.  Repeat for WAN2.

typeraj
Here to help

Re: How to cable MX & MS for HA

Unfortunately, neither ISP offers a second handoff but that would have been my preference. I'm hesitant to use an internal switch since ports are at a premium here. I have a spare MS220-8, which I think I can use as the external switch to do what you said. Thanks @PhilipDAth.

Ted_Dasher
Conversationalist

Re: How to cable MX & MS for HA

Just keep in mind that while you can create two VLANs and assign 3 ports to each making this work, by doing so you are leaving the MS-220-8 as a single point of failure which somewhat defeats the purpose of using a second MX, dual ISPs, etc.  Might be better to use the MS-220 for one ISP and use the other ports as edge ports then grab 3 ports on another switch for the other ISP. 

ether
Here to help

Re: How to cable MX & MS for HA

Hi all,

 

I am in the process of adding a second MX as a hot spare and need guidance in regards to the setup of the second MX. I am trying to follow the Meraki documentation for HA Switch Stack setup.

 

Our setup consists of:

-2x MX250 (Primary is active, the warm spare has not been installed yet)

-7x MS225 (stack 1)

-8x MS225 (stack 2)

 

Currently, the primary MX is connected to WAN1, WAN2 and directly to one of our MS switches in Stack 1 via the SFP port. Is this the recommended setup for redundancy? Or should the Primary MX be connected to every switch in stack one (so 7 total connections from the MX -> MS stack 1)? What about stack 2?

 

When I add the second MX to the equation, do I just hook it up to the same single switch in stack 1 that the primary MX is connected to? Or is it recommended to hook it up to all 7 switches in stack 1?

 

Thanks

Ted_Dasher
Conversationalist

Re: How to cable MX & MS for HA

I would connect each MX to two switches in each stack so you can have the WAN connection survive any single switch failure in the stack.  If you're using 10G connections then any more than 2 connection per stack isn't really gaining you anything beyond protection from the unlikely case of both up-linked switches failing at the same time and spending huge amounts of extra money.  Put them on separate AC and UPS  circuits and this becomes an almost impossible occurrence.

 

If you're up-linking with 1G connections then you're never going to be able to take full advantage of the throughput of this MX model.

nickydd9
Getting noticed

Re: How to cable MX & MS for HA

Sorry to re-hash an old thread, but on the topic of WAN breakout switches, I have always used a Cisco 8-port switch to break out an ISP that can only provide a single handoff. This switch is a simple configuration of a VLAN and we don't manage the switch any further and just deploy it. If it breaks we ship a new one with the same config. 

 

If I was to begin using Meraki 8-port switches for WAN breakout is it essentially the same concept? My concern is that having an MS120-8 upstream of my MX's will not work because they will not be able to communicate with dashboard. In addition to cabling the switch into my modem and both MX-A WAN 1 and MX-B WAN 1, would I also patch it in to the MX's or downstream switching on a trunk so the switch can be managed and communicate with dashboard?

 

I understand I can just use spare switchports on the downstream switching to breakout my ISP connection instead of using a separate WAN breakout switch, but again my question is around a WAN breakout switch upstream of the MX's. 

LuigiJuve
Here to help

Re: How to cable MX & MS for HA

I tested many scenarios but the best to get this to work is to have a designated vlan set on the MX for the heartbeat for example vlan 1111 ip 1.1.1.1/30.

 

then connect a cable between each MX for example MX1-e3 to MX2-e3 and set it as a trunk with the native vlan 1111 and allow all vlans (do not prune or pick any other vlans specified in your MX)

 

then your connections from the MX to the Skitch facing switches. Also if you are not using a Meraki switches like Cisco Catalyst or Arista then I set the Spanning of tree cost for the primary port to be 100 and the second link to the MX to 200

 

MX1-E4 —> SW1-E4 

MX1-E5 —> SW2-E4

MX2-E4 —> SW1-E5

MX2-E5 —> SW2-E5 

 

jdsilva
Kind of a big deal

Re: How to cable MX & MS for HA


@LuigiJuve wrote:

I tested many scenarios but the best to get this to work is to have a designated vlan set on the MX for the heartbeat for example vlan 1111 ip 1.1.1.1/30.

 


Can you please show me exactly how you designate a VLAN as heartbeat?

 

 

LuigiJuve
Here to help

Re: How to cable MX & MS for HA

H

LuigiJuve
Here to help

Re: How to cable MX & MS for HA

like this

 

HA-1.pngHA-2.pngHA-3.png

LuigiJuve
Here to help

Re: How to cable MX & MS for HA

does the LTE failover only work when the MX WAN1 and WAN2 physical links are down only?

 

I am seeing weird instances when the WAN links can still connect and ping their gateway to their connection to the cable modem or circuit ISP router that is still online but the internet peering next hop is down. From there it does not failover to the LTE.

 

what steps does the MX try before it determines this? Does it do some sort of SLA?

jdsilva
Kind of a big deal

Re: How to cable MX & MS for HA

Hi @LuigiJuve ,

 

No where in there did you designate that VLAN as heartbeat. You simply created a VLAN, assigned it a subnet. Can you please clarify what makes this VLAN a heartbeat VLAN different from all the other VLANs?

 

How do you control which VLAN is used for hearbeats? 

NolanHerring
Kind of a big deal

Re: How to cable MX & MS for HA

You can't designate. Heartbeats for VRRP are sent out across all VLANs by default, and there is no way to configure them to do otherwise. At this point that designated VLAN is probably redundant logically, maybe not physically if you separated it out (not sure if that is worth it either).

Nolan Herring | nolanwifi.com
TwitterLinkedIn
nickydd9
Getting noticed

Re: How to cable MX & MS for HA

I think @jdsilva understands this entirely, he's just trying to get @LuigiJuve to arrive at that answer on his own.... 😃 

jdsilva
Kind of a big deal

Re: How to cable MX & MS for HA


@nickydd9 wrote:

I think @jdsilva understands this entirely, he's just trying to get @LuigiJuve to arrive at that answer on his own.... 😃 


Shhh! 😉

LuigiJuve
Here to help

Re: How to cable MX & MS for HA

Like I said I use a direct link between the two MX units.

I specified a direct VLAN for only that link and setting it was the native vlan trusting all vlans to be used as the heartbeat. maybe it only works logically so be it but it works for over 15 setups flawlessly once I did it this way and the LAN base links if one of them goes down and helps prevents the Master to Master HA issues..

 

Try it, if you don't like so be it figure it out another way.

jdsilva
Kind of a big deal

Re: How to cable MX & MS for HA

Hi @LuigiJuve,

 

I'm not saying it won't work. What I'm saying is that it's not possible to configure a "heartbeat VLAN" as described in that document. It's a completely false pretense to think that this supposedly magical VLAN is any different from any other VLAN you have configured. As @NolanHerring pointed out, VRRP hellos (the "heartbeats") are sent out on every VLAN. Further, Meraki's implmentation of VRRP has elected to use a single Virtual Router with multiple Virtual IP addresses as opposed to a Virtual Router for each VLAN, meaning, that as long as a VRRP heartbeat makes is to the standby unit on ANY vlan the current active master will remain the master for all VLANs. It's not possible for only one VLAN to fail over. This is truly and Active/Standby only implementation.

 

So please forgive me if this sounds harsh as it's not meant to, but your 15 installations are not working because you followed that guide, and they also are not working the way you think they are working.

 

This is actually my objection to this configuration, and Aaron Willette's design. It's based on a falsity, and that falsity has been propagated to many corners of the Internet. Quite often people chime in here in the Community and reference that design, and nearly every time the person making the reference has no idea that it's simply not possible to actually designate a VLAN as a "Heartbeat VLAN".

 

My preferred way to do Meraki Warm Spare, which has come from my own experience deploying Warm Spare for numerous customers, and through the very insightful advice of @PhilipDAth, is to NEVER EVER directly connect the two MX appliances together, and in stead singly connect each MX to a single switch or switch stack (the latter preferred for redundancy). See, having the MX's directly connected actually can cause issues under certain failures with Spanning Tree convergence. Further, it's also desirable to have the VRRP heartbeats traverse a path more representative of the path that user data actually takes. By creating a shortcut that heartbeats can take, but not user traffic, you are actually exposing yourself to a failure scenario where user traffic is blackholed, but VRRP is operation just fine and not allowing a failover to occur.

 

Failovers are a good thing! You want failover to occur when there's a failure. A direct link cheats the system and can actually prevent a failover when you actually want one.

 

I'll close this by pointing out that Meraki has actually changed their recommended topology for Warm Spare by removing the direct connection between MX's. It took me some time to convince them to do it, but they did come around 🙂

 

https://documentation.meraki.com/MX/Deployment_Guides/MX_Warm_Spare_-_High_Availability_Pair#Recomme...

 

 

LuigiJuve
Here to help

Re: How to cable MX & MS for HA

nikkydd9.. what exactly are you trying to say here...

 

 

LuigiJuve
Here to help

Re: How to cable MX & MS for HA

jdsilva, so what do you mean my 15 installations are not working please explain..

 

In my setups I am not using Meraki switches but either Arista or Cisco Nexus not using Stacking but using MLAG or VTP.

 

Sure I get it, you do not need a set a VLAN to determine the heartbeat because VRRP is used on all VLANs sure but what if you connect the MX pairs to the south facing switches with only certain VLANs only and not the one VLAN you set for the heartbeat like in my example..

 

I know Meraki removed the peer link between MX HA pair but no one can explain to me why this is bad and why the failover will not occur when it happens..

 

You seem to know all the answers so explain it.

PhilipDAth
Kind of a big deal

Re: How to cable MX & MS for HA

>My preferred way to do Meraki Warm Spare, which has come from my own experience deploying Warm Spare for numerous customers, and through the very insightful advice of @PhilipDAth, is to NEVER EVER directly connect the two MX appliances together, and in stead singly connect each MX to a single switch or switch stack (the latter preferred for redundancy).

 

Not saying it won't work another way, but this approach produces a rock-solid solution.  When you dual connect them (or run a connection between them) you are far more likely to have outages because of the extra redundancy than you would have had otherwise with the simpler design.  Typical issues are in-appropriate spanning-tree port blocking, and short term spanning-tree loops.

PhilipDAth
Kind of a big deal

Re: How to cable MX & MS for HA

Here is a good explanation:

https://documentation.meraki.com/MX/Networks_and_Routing/NAT_HA_Failover_Behavior#VRRP_Mechanics_for... 

 

My understanding is if VRRP fails on any VLAN the entire unit fails over.  Having a dedicated link between the units will have no impact if a common switch (on another port in another VLAN) fails.

Typically, you want the failover to happen.  No point the master continuing on with a VRRP failure on any specific VLAN.

LuigiJuve
Here to help

Re: How to cable MX & MS for HA

thank you..

 

I was using this guide when setting these up. https://www.willette.works/mx-warm-spare/

 

not all my setups have this dedicated VLAN approach, but most do. 

I know understand it is kind of pointless and will remove this for future deployments.

 

As for the direct connections between MX units I will test more but it seems to work well and always worked well even with the LAN based failover recommended by Meraki..

jdsilva
Kind of a big deal

Re: How to cable MX & MS for HA

@LuigiJuve wrote:

jdsilva, so what do you mean my 15 installations are not working please explain..

Sorry, I think I didn't make that statement very well. I was trying to say that yes, your deployments are probably working just fine under normal network conditions. But, based on your comments around the heatbeat VLAN I do not believe they are working the way you think they are working. If you believe that you have a dedicated VLAN that's being used purely for heartbeats then you are operating under an incorrect assumption, and you don't fully understand how your network is operating. Again, I'm not trying to be harsh, I'm simply trying to point out an incorrect assumption and help you understand the way Warm Spare actually works 🙂

 

 

 


@PhilipDAth wrote:

 

My understanding is if VRRP fails on any VLAN the entire unit fails over. 


I don't think this is true, and the way I read the doc you linked to doesn't support that. 

 

image.png

Because the MX uses a single VIrtual Router with multiple Virtual IP's it can't actually detect a failure of a single VLAN. The heartbeat sent on each VLAN is identical to the heartbeat sent on every other VLAN. The only thing that the MX VRRP process is able to detect is whether it received a heartbeat at all within the dead time. As long as any heartbeat reaches the standby it will not assume the Master role. It must have a complete and total loss of all heartbeats on all VLANs to make the transition. 

Prithiviraj
Conversationalist

Re: How to cable MX & MS for HA

Hi, 

I am new to Meraki and working on connecting MX250 to two ISP providers. Each ISP provides one leg connection, so I designed the topology as in this diagram, having a WAN transit switch stack using MS350-24. I read Meraki documentation, all the design topology shows ISP links directly terminating into MX devices, would like to check if this design will work for MX HA setup.

 

Meraki Design.jpg

Ted_Dasher
Conversationalist

Re: How to cable MX & MS for HA

Should work but a few comments.

 

1. What do you gain by stacking the transit switches?

2.  If the transit switches are Meraki then you probably need to run separate LAN connections back to them from the firewalls or another downstream switch for management.  You could assign them public IPs but that's really not a good idea.

cmr
Kind of a big deal
Kind of a big deal

Re: How to cable MX & MS for HA

@Prithiviraj we use almost the setup you have and it works well.  The differences we have are that the WAN transit switches are dumb layer 2 unmanaged switches and not stacked.

 

Sometimes we have each MX connected like you have to two different stacked L3 core switches and sometimes we have them only singly connected.

 

I have come to the conclusion that dual connections are better only when the MX to L3 core link is a single VLAN.  This way the possible STP issues don't seem to cause a problem.  However this is open to discussion!

Prithiviraj
Conversationalist

Re: How to cable MX & MS for HA

Hi @Ted_E , @cmr 

 

Thanks for your responses. The intent is to have dual internet link load balancing using MX, however each ISP provide one leg to connect. Understand that stacking those switches do not make sense as we will configure different vlan for each ISP links. Have updated the topology as below. We will also be using internal firewall for lan traffic and all internet vlan will bypass internal firewall to MX.

 

Meraki Design-2.jpg

CptnCrnch
Kind of a big deal

Re: How to cable MX & MS for HA

I'm not shure if I got you correctl @Prithiviraj  but you won't be able to have load balancing with MX. Those are always Active / Passive.

cmr
Kind of a big deal
Kind of a big deal

Re: How to cable MX & MS for HA

@CptnCrnch  my assumption is that @Prithiviraj  is loadbalancing the ISPs on the Active/Passive MX pair.  Though I may have got that wrong...

 

Edited to remove incorrect query about dual links which are correct!

Prithiviraj
Conversationalist

Re: How to cable MX & MS for HA

Hi @CptnCrnch 

It is for 2 x Internet link load Balancing within a active MX device.

Get notified when there are additional replies to this discussion.
Welcome to the Meraki Community!
To start contributing, simply sign in with your Cisco account. If you don't yet have a Cisco account, you can sign up.