Splitting dual WAN links

[typeraj]

Hi everyone, 

I'm trying to use an MS120-8 to split 2 WAN links between 2 MX100s that are in HA. I followed this guide to set up the HA pair using downstream switches in a full mesh config for VRRP.

However, I ran into real issues when I introduced the MS120 between the ISP handoffs and MXs. This is how I had it configured:

The first issue was trying to configure the breakout switch. It wouldn't connect to the Meraki cloud without also being directly connected downstream to one of the L2 switches. Without that connection, the LED would remain orange, and the switch shows as offline.

The second issue is I started seeing 'Uplink IP Conflict' errors on the primary MX. I'm not sure why given that, I have statically assigned the WAN IPs on both MXs. 

Finally, I also saw a spike in packet loss, where it used to be 0% before introducing switch, it's now consistently 15-20%. 

Unfortunately, I don't have a lab environment to test any of this, so to avoid disruption during our business hours, I've partially rolled back to the following setup which appears to be stable for now: 

WAN1 - MX1

WAN2 - MX2

MX1 and MX2 in HA via downstream switches per the diagram. 

From everything I've read on here, even a simple unmanaged switch should work, so I'm not sure what I'm doing wrong with this MS120. Any help would be much appreciated. 

Thanks.

Raj

[PhilipDAth]

> It wouldn't connect to the Meraki cloud without also being directly connected downstream to one of the L2 switches.

I would give the MS120 a static IP address from one of the two ISPs (on whatever VLAN you setup for that ISP) so that it can talk directly to the Internet.  You'll need to configure this via the local status page.

By making your external switch dependent on going through the inside switches and then back via itself - you have created a circular dependency.  So break the connection between the internal and external switches.

You should have a VLAN for each ISP (you can re-use VLAN1 if you like).  Each VLAN should have three ports in it (I'd make them access ports with the info supplied so far).  The three ports are for the ISP connection and a WAN port from each MX.  Repeat for the other ISP and the remaining WAN ports.

[jdsilva]

Personally, and the Meraki gods will likely strike me down, but this exact use case is one I'm not a fan of using Meraki switches for. The reasoning behind my preference is the lack of ability in properly controlling the management interface of the MS switch. 

As @PhilipDAth said, you should statically assign the mgmt IP of the switch. However, my own irrational paranoia prevents me from being comfortable with giving the MS switch a public IP directly on the Internet so I would prefer to run a dedicated link into is on the "LAN" that's behind the MXs for Mgmt. But, should the switch lose its connection to the cloud over the statically configured Mgmt IP it will DHCP for an address on any and every interface and VLAN it possibly can. This can have unexpected consequences like the switch grabbing an available IP, preventing your MX from getting one. You have to understand your specific situation and decide for yourself if things like this are a problem.

---

@typeraj wrote:

From everything I've read on here, even a simple unmanaged switch should work, so I'm not sure what I'm doing wrong with this MS120. Any help would be much appreciated. 

---

I disagree with this statement as your diagram would indicate that you require VLANs to segregate the two different WAN services, and unmanaged switches are note VLAN capable.

You are creating two different VLANs, one for each Internet service, yes?
 

[typeraj]

@PhilipDAth To configure the switch initially I plugged it in downstream, let it download the config, then re-cabled without that downstream connection but with a public IP. After that, it remained orange, until I added that link back to create the circular dependency. Maybe that initial step is where I went wrong?

Also, if I were to give it a mgmt IP from one of the two ISPs, how would the switch behave in the event of that ISP link failing?

@jdsilva Yes, I agree that 2 VLANs are required and that's exactly how I've got it configured: VLAN100 (ISP1, MX1, MX2) and VLAN200 (ISP2, MX1, MX2). 

run a dedicated link into is on the "LAN" that's behind the MXs for Mgmt.

Wouldn't this create the circular dependency that @PhilipDAth was referring too? 

[cmr · Accepted solution]

I know this is slightly off topic, but by installing a single PSU configured device to pass through both connections, aren't you damaging the viability of the HA setup?

We have the same as you have at each of our sites, except we have two separate 5-port layer 2 unmanaged switches, one on each ISP connection.

That way you maintain the availability whilst also allowing the use of both connections at once.  We used Cisco SG110 models and each one was less than half the cost of an ms-120.

[jdsilva]

---

@typeraj wrote:

@jdsilva Yes, I agree that 2 VLANs are required and that's exactly how I've got it configured: VLAN100 (ISP1, MX1, MX2) and VLAN200 (ISP2, MX1, MX2). 

run a dedicated link into is on the "LAN" that's behind the MXs for Mgmt.

Wouldn't this create the circular dependency that @PhilipDAth was referring too? 

---

Yup, it sure would. You either give your switch a public IP, or end up in this circular dependency.  All reasons I prefer to not use an MS as an Internet breakout switch 😞

[BogoMips]

The circular dependency in managing the switch isn't a big deal if the switch is only being used to separate and distribute the Internet circuits via VLANs. Once the switch is configured, it should be reachable on the LAN side of the MXs as long as one of the WAN paths is up. If the switch is configured to use a WAN address from one circuit, the switch will be unreachable if that circuit goes down. In any case, when both circuits are down, the MXs will be unreachable, and being able to reach the switch at that point is moot.

I like @cmr's use of two unmanaged switches for the use case being discussed. There is path and device redundancy, and replacing the switch requires no configuration. We have some MS120-8s filling the same role, and there are no practical features that make them worth the price disadvantage.

@typeraj, I'm wondering if the packet loss you saw was related to the 'Uplink IP Conflict' error. Perhaps your primary ISP handed out a DHCP lease to another device, which conflicted with the address you statically assigned? It could also be something simple like a bad cable.

[GaryShainberg]

So I was where you are a couple of years ago when the community was much less and help from people like @PhilipDAth  was few and far between 🙂

In the end, te only way I could make this work was with a combination of what has been suggested.

I had to use a non-meraki 8 port managed switch and configured 3 ports in one vLAN, 3 ports in another vLAN and one port in the management vLAN, this allowed me to maintain separation from each ISP, but also from the management LAN - TBHWY, if you dont need to manage the switch then you do not need to connect it to anything for management as its pretty dumb once set-up.

The bigger issue is that from an architectural point of view, you have now injected a single point of failure that could affect both ISP's.

Retrespecifly I have installed two Netgear GS105e's (Sorry Meraki) one for each ISP and this way, if either switch or ISP fails then at least service will continue - just for belts and braces (or suspenders for my American friends) I have also installed two cheap USB 4G/LTE dongles (with PAYG SIMS) one on each MX as a tertiary backup - just in case.

[typeraj]

Thanks, everyone for your input. The consensus seems to be 2x unmanaged switch solution to avoid the awkward MS120 mgmt and remove the single point of failure. 

@BogoMips You might be onto something here - I think my MS120-8 had a statically assigned IP from my ISP, which I then also gave to MX1 later on. Not sure how long the lease is for, but this could explain it.