How to turn MS120 into WAN breakout for 2 ISPs and 2 MX250s in HA?

[CaseyFowler]

I'm far from a network engineer but have been tasked with transitioning our SPOF config into a HA.

Our MSP recommended we get a unmanaged L2 switch to sit between each ISP's router at our edge. They recommended we order a Cisco Cat 1000, but they are on backorder for 6 months

I've seen many state that an MS120 can be configured to support this(which we have many spares of), but I can't find specifically what the config looks like. I asked the question elsewhere and got this recommendation:

Set a vlan ID for each ISP(999/998) and just tag 3 ports per ISP. This will create the path the broadcasts can talk over and you don't need to buy anything new.

• ISP1>Switch(VLAN999)>MX(01&02) internet 1 ports.
• ISPb>Switch(VLAN998)>MX(01&02) internet 2 ports

I'm working my way through what each port should look like and I'm realizing I'm a bit out of my depth. Wold the MS120 not doing VRRP cause an issue here?

Here's an image of the layout, and what I'm assuming port 1 will look like for each breakout.

If this is correct, would the same port config be used for ports 2&3? What about port 4? I'm assuming if 1-3 are correct, then the only difference with port 4 is adding 999, 998 to the VLAN.

[alemabrahao]

Check the recommended topologies:  https://documentation.meraki.com/MX/Deployment_Guides/MX_Warm_Spare_-_High_Availability_Pair#Recomme...

[rhbirkelund]

I've done this a couple of times, and while it's not really the most optimal way of doing it, it's certainly a possibility.

There are some caveats with using Meraki Switches as WAN breakout switches.

One aspect that you need to have in mind is that Meraki needs to have an IP address on it's Management. And that's whether you're using it on the LAN or in a WAN breakout setup. So you need a connecting from the WAN switch and back to the LAN side of MX, for it to obtain an address for it's Management.

In terms of when you configure the switches, just remember, that it is imperative that the LAN side VLANs are not mixed with the WAN side i.e. do not trunk the WAN side VLANs into your LAN side.

Another issue is that fact that during maintenance windows where you perform firmware upgrades, you'll experience that your entire Meraki organization will go offline at least twice; once during MX upgrade, and second during MS upgrades. The WAN breakout switches will upgrade at the same time as the rest for your switching infrastructure, unless you define upgrade groups for staged upgrades.

And in the end, you can never guarantee that something doesn't go wrong during Firmware upgrades, that end up causing a Loop between you LAN side and WAN side.

The third caveat is that, your Network Clients page will also contain publicly routed addresses, and you'll end up seeing some odd statistics in the Applications details.

All this being said, as long as you are aware of what you are doing, this is certainly a possible setup. Like I said, I've done it a couple times, but mainly out of necessity, rather than opportunity.

But these are just my two cents.. 🙂

[CaseyFowler]

The third caveat is that, your Network Clients page will also contain publicly routed addresses, and you'll end up seeing some odd statistics in the Applications details. Meraki Switches as WAN breakout switches.

This is a great point that I did not consider. Thanks for all the info! Would placing the switches into a different dashboard network help with this? 

I spoke with a tech with Meraki and they said it's a fairly common config. He did recommend to not to connect the 120's together via port 4, but to instead link them to a LAN port of the MX so that they can communicate with the cloud.

[rhbirkelund]

You can definitely break them out into a network of their own, but I'm not sure if that will make you clear of the public addresses in the Clients page etc. Bit I'd love to here the result.

But still the other point stand, although you can better schedule the WAN switch firmware upgrade, in the sense that it won't happen at the same time as your downstream LAN.

[Ryan_Miles · Accepted solution]

@CaseyFowler I put together a deck on this topic awhile back. Might be of help to you.

https://docs.google.com/presentation/d/1xsb8imtUFjN13so86kIZ04IR9f6WEKdbpUrYVON64Zg/edit?usp=sharing 

[rhbirkelund]

@Ryan_Milesout of curiousity, what are your thoughts on models as WAN Breakout switch? I usually see customers opting for the small 8-ports MS120 switch, but seeing as it will be handling "raw" internet traffic? A customer I have, which has just been through this maneuver used an MS410.

[Ryan_Miles]

Same. I often see MS1xx & 2xx series used for breakout. But, depending on physical handoffs, performance, or redundant PS and fans I could see what someone might go higher in the product family.

[leewalhovd]

The routing for the ISP and handoff can play a role as well. There are some ISPs that will do a transit IP and a block of public IPs behind that require at a minimum a static route in front of a warm spare MX setup. 

[CaseyFowler]

This is awesome and a huge help.

The only other bit that I'm not 100% on, has to do with the uplinks on the MX. Support basically said I just need to make sure to change from the default of VLAN tagging off and change it to use VLAN tagging with the corresponding VLAN from the MS120 it's connecting to.

Basically, does this make sense based on my original image?

[Ryan_Miles]

The MX doesn't need to have any special VLAN configs. The switches hand off a L2 VLAN. No tagging comes into play.

[bulkjunooni]

Hi Ryan, this is awesome.

Curious if this will cover any redundancy as well??? in case one meraki fails or one ISP goes down?

[cmr]

We use small 5 port Cisco unmanaged Gigabit switches for the ISP splitting.  I prefer not to have managed (and therefore more hackable) switches outside of the perimeter.  They are cheap as chips, perform perfectly well and have been very reliable.  They are also pretty much always in stock!

[CaseyFowler]

---

@cmr wrote:

I prefer not to have managed (and therefore more hackable) switches outside of the perimeter.  

---

Great point that I wasn't really considering.

Do you have a particular model you've had good luck with? MSP had us looking at a catalyst 1000, which struck me as overkill but who am I to say...

I'm seeing a few like the CBS110 for $100. Would this be good? 

[cmr]

We use the SG110D-05 and I think the CBS110-5T-D is the direct replacement.  We've had very good service out of them and as they are so cheap, we always have a spare or two hanging around.

[Ryan_Miles]

In the Meraki breakout switch design the management tunnel between the switch and dashboard is going through the MX appliance. I'd argue that's better than a simple switch sitting outside a firewall that might have telnet, SSH, or a web page enabled with less than adequate security config in place.

[g_rob]

This is a bit of a rabbit trail from the OP, hopefully not too far off topic.  

@Ryan_Miles I've been following your slide decks that you provided above (thanks for sharing those by the way) for a new MX HA cluster setup I'm building out with two ISP connections.  I appreciate you simplifying a somewhat complex setup.

 I'm using two 8 port ms120's for the WAN breakout switches going to two MX85's and behind those are two 24 port MS120's.  I bought all of this thinking I already knew how to get it going, not realizing how much detailed configuration was involved.  It's been a good learning experience. 

I'm writing because I've got a loop of some kind that I have not resolved yet.  I figured out the wan breakout config, that and the firewalls seem to be working fine.  Where I'm getting stuck is on the switch side, setting up both 120's behind the MX's.  I'm using the recommended layout from MX Warm Spare guide.  So a link from each MX to each switch for four total and then one link between the two switches. 

In your WAN and LAN failure scenarios slide deck you say all the interswitch and switch to MX connections should be 'trunks, allow all VLAN's, and a native VLAN'.  Here are a few things that are confusing me about this.

1. What should the spanning tree settings (and by that I mean RSTP, STP guard, and UDLD) be for the trunk ports and are the ones to the MX's different at all from the inter switch link?  
2. Also, you said on the MX end of these uplink ports there are no settings to change, that it's all handled by the L2 of the switch, but I'm reading over on this post (https://community.meraki.com/t5/Switching/STP-guard-setup-best-practices/m-p/31165) about all the ST stuff and turning off Drop Untagged Traffic on the MX to avoid loops and it seems like I need to have some of those settings on there.  I can't tell if I'm just not grasping the details here or if I'm convoluting two unrelated things.

Thanks again Ryan for the slide decks and I'd be happy to hear from anyone on how these links should be configured to avoid loops.

Thanks All (in advance)

[Ryan_Miles]

What are the LAN port settings of the MXs? You don't want to have them set as drop untagged traffic. That will results in loops due to STP running on the untagged VLAN.

Practical example of a spoke I have. Diagram and failover animations here.

MX has VLANs: 10, 20, 30, 31, 32, 50. VLAN 30 is the mgmt VLAN for the switches and also what I use as the native VLAN.

So, MX LAN ports are trunks, native 30, allow 10, 20, 30, 31, 32, 50.

Switches ports are the same exact config - trunks, native 30, allow 10, 20, 30, 31, 32, 50.

As for MS side STP options. This doc covers the use case for each option.

My personally, I wouldn't use any guard mode for the switchports connected to the MX LAN ports.