MX100 HA issue

[Narnia534]

We have:

2 MX100's setup in HA mode

2 ISP's

2 MS425 Switches stacked

1 MS120 switch between the ISPs and the MX100s

The MS120 gives us signal from our ISPs to both MX100 (different IP addresses of course)

As far as I can tell, the MX100 are working in HA mode. And both ISPs are working.

On MX1

Ports 1 & 2 are connected to ISP1 and ISP2.

Port 9 is connected to an onsite web server (configured as a DMZ)

Ports 8 and 11 are each connected to a different one of the stacked switches.

So far so good

On MX2

Ports 1, 2, & 9 are the same as above.

However, as soon as I try to connect port 11 to one of the stacked switches, our Internet goes down instantly.

Since there is no connection at that point to the cloud, I can't capture packets from that port, can't see what is happening.  As soon as I disconnect it from the stacked switch, Internet is back up.

Can anyone give me a steer as to what to look for, or what I'm doing wrong?

Many thanks

[ww]

At the switch currently  1 Port is stp blocking?

How are the ports that connect mx and ms425  configured ?

[Narnia534]

[Nick]

You are not tripping over the anti spoofing on the MX?

Static routes etc all in place?

https://documentation.meraki.com/MX/Firewall_and_Traffic_Shaping/IP_Source_Address_Spoofing_Protecti...

[rhbirkelund]

Is it correct to assume, that the MS120 acting as a "wan switch", is connected back to the LAN side, to allow management?

Could you perhaps try to draw up the topology?

[Narnia534]

The MS120 is not connected back to the LAN. It is connecting to the cloud via the ISPs, and I can manage it from the Meraki Dashboard

I'm not very good at topology, so here is an attempt:

The MS120 has 3 VLANs connected to it: 110, 120, and 95

The Web server has 2 NICS in it, one for VLAN95 and one for VLAN1. This VLAN1 is connected to another MS125 switch which is plugged into both core switches

Each of the desktops is on VLAN1

As you can see, MX2 is not plugged into the LAN at all, because when I do, it stops our Internet. That's what I'm trying to correct.  MX1 is plugged into both core switches, with STP blocking one.

[ww]

Can you disconnect  Port 9 of mx2. 

And then try  connect mx2 to your stack again

Or

Connect the server at vlan 95 to the stack  and add vlan 95 to the trunk to the mx. (And Disconnect both 95 cables of the ms120)

[Narnia534]

I can try that tonight when most people have gone home.

I'm not sure I understand your 2nd option.  Vlan 95 is supposed to be a DMZ.  If it is going through the stack, wouldn't that eliminate the safety of the DMZ?

[ww]

If you connect it using layer2  vlans to the mx its no different then now.  The  layer3 interfaces  stay en on the mx.  Vlan 1 and 95 can only  communicate  when the traffic passes the mx (or not with firewall rules  in place).

Proposed  scenario 1 is the quickest  to test. If that works, its probably  best to config scenario  2 as solution

[cmr]

Personally I would ditch the MS120 on the ISP side and get a pair of unmanaged L2 switches, Cisco do some small ones that have been reliable for us. As for the DMZ, I'd use another separate switch for that, or if only the one host, directly cable both port 9s to the webserver.  You could use the MS120 for that role

Sorry, forgot to explain my reasoning, which is that is the MS120 goes down you lose everything.  The purpose of two ISPs is normally to increase availability and by putting them both in one switch you have largely negated that benefit.  Secondly having outside and DMZ on the same switch is not 'clean' and could easily lead to the wrong port being used for the wrong connection.

[Nick]

Not to go off topic - but how would you propose those layer2 ISP switches are setup @cmr 

[cmr]

@Nick no config, use unmanaged switches, we use 5 port models with MX WAN ports occupying two and ISP drop cable in the third.  Two spare ports to use for diagnostics. 

As for the MS-120 I'd have it all on one VLAN and it can reach the Meraki cloud by going though port 9 on the MXs and then out to the internet.

When dealing with outside/DMZ I always like to keep it simple as when it matters, it won't be you there, it will be someone like a security guard on the end of a phone who may not understand the complexities of the setup. 

[Nick]

@cmr 

Ah I see what you mean now - I wasn't reading the previous properly

I concur with that, keep the upstream nice and simple in you can as CMR says its easy to figure out what is what and you don't benefit much from have an MS in play there

[Narnia534]

Good call, @ww , when I disconnected Vlan 95 from MX2, then I could successfully plug in to VLAN1 on MX2.

So why is that an issue? Shouldn't STP on the MS120 have stopped it?

Or is it possible because the WebServer only had 1 IP address?

[Narnia534]

This is still not solved.  Been working with Meraki Tech Support, but so far nothing

What works:

1. MX Warm Spare - High Availability Pair
2. Two WAN uplinks

What doesn't work:

1. DMZ (VLAN95) connected to both MX devices

The problem, as I understand it from Meraki, is that the MX Firewalls do nothing with STP.  And the MS Switches use RSTP, but do not deal with STP on a per VLAN basis.  As @ww saw, as soon as I unplug the DMZ VLAN from the secondary MX, everything else works.  Apparently VLAN95 connected to both MX devices completes a loop and STP shuts down VLAN1 from the Primary MX.

One solution, they tell me, is to replace the WAN Breakout MS switch with a Cisco Catalyst switch which has PVST or Per VLAN Spanning Tree.  This would isolate VLAN1 from VLAN95 STP wise

Can anyone think of anything solutions?

Thanks

[ww · Accepted solution]

The second  solution i gave you should  also work

[Narnia534]

Thanks, I am definitely looking into this further.

[Narnia534]

Thanks @ww . That did take care of it.  Meraki support kept trying to find a way to make my solution work, but in the end he agreed with you.

Thank you very much

[Nick]

So your revised setup looks the same - with the VLAN 95 connected into the MS Core instead of the MX units themselves?

Twin connections from each MX into the MS Cores as well?

[Narnia534]

Below is what it looks like now (hopefully a better drawing):

• That port on the core is now configured as Access, Vlan 95. 
• Nothing else on the Core is configured for Vlan 95 (no routes, no ACLs. It just goes to the MX),
• But the MX is configured for V95. 
• STP disables the redundant ports on the Cores

[PhilipDAth]

>Ports 8 and 11 are each connected to a different one of the stacked switches.

So far so good

Connect each MX with a single connection to the switch stack.  Then you remove the L2 loop.  Then it will work and be reliable.

[Narnia534]

Unfortunately that was not the problem.  I started out that way with just one connection to the stack from each MX.

I still had to disconnect MX2 to have Internet.

[rhbirkelund]

I have previously done something like this. Some of the elements could match yours. It might not be the prettiest, but under the circumstances that were given to me, it works. Been running for the last two years, and so far haven't heard anything from it.

Ignore the blue line. It was for a telephone service provided directly to the LAN.

In your case HX1-SW1 would be your MS425 stack.