Two MX84 on same network

[IanMartin]

Hi everyone,

I'm not sure if I've created a problem on our network with our MX devices.

We have a single LAN network 192.168.100.0./22 (No VLANs)

We have a single DMZ network 172.16.30.0/24 (No VLANS)

These networks are physically separated different switches and cabling

We have two independent internet links

We originally had two independent ASA 5510s on each link

and we planned to replace them with MX84

We run fixed IP for everything and control which link is used with the default gateway on each PC/Server/Device 

So the MX's are setup the same with two networks

Local VLAN 2 192.168.100.0/22 (LAN)

Local VLAN 3 172.16.30.0/24 (DMZ)

Port 3 is set to VLAN2

Port 5 is set to VLAN3

Today i installed the 2nd MX for the first time and within a hour or so i started to get calls about network issues from multiple different systems (non linked systems) we even started getting issues with phone calls (VOIP)

After taking the 2nd MX out and putting the ASA back all the issues stopped.

Is there something fundamentaly wrong with the setup we are trying to create, essentially we are just recreating the setup we had with two ASA 5510's which has been fine for years.

Any help or pointers would be greatly appreciated.

Regards

Ian

[ww]

what is your 2nd mx doing?

[jdsilva]

If I understand you correctly you'll want to set up the MXes in a Warm Spare configuration, and connect both Internet connections to both MXes.

You can't have both MXes active, but you can have both Internet services active on the active MX. For the LAN side Meraki uses VRRP to give clients access out.

You do not want to have two MXes in two different networks. 

https://documentation.meraki.com/MX-Z/Other_Topics/Warm_Spare#NAT_Warm_Spare

[Adam]

Agreed, the only way to have two MX's in the same network is if one is a spare.  Otherwise, they'd need to be in separate networks.  If the ASA is operating where the MX isn't then there is likely an interface, route or other configuration that wasn't migrated fully?

[PhilipDAth]

So the two existing systems were 100% independent with con connection between them (apart from the Internet).  And your new configuration is the same?

If they are completely independent then there should be no issue.  It sounds like there is actually some kind of link between these two networks.

[IanMartin]

Hi Everyone and many thanks for the replies.

I'm not sure i explained our situation very well, also i'm no network engineer!

Original config:

One LAN network (192.168.100.0/22)

One DMZ Network (172.16.30.0/24)

We have two active internet links both being used at the same time

We host a web server and ftp server on each link

We have an ASA 5510 on each link and the ASAs operate independently BUT both have interfaces IPs on the LAN and DMZ.

A PC or server on our LAN could use either internet link by changing the gateway 192.168.100.2 (ASA 1) or we could use the other 192.168.100.10 (ASA 2)

This has work fine for years

The Plan was simply to replace the ASAs with MX84s

I'm using the same interface IPs from the ASAs on the MXs 

ASA 1 Outside interface IP, LAN Interface and DMZ interface IPs have been set on MX 1

ASA 2 Interface IPs have been set on MX 2

But when we power off the ASAs and power on the MXs (We do power cycle the ISP routers to clear the ARP cache) we start getting major network issues, internal systems that should not be going through the firewalls opening files from file servers, internal voice calls (voip) breaking up, internal management systems crashing after loosing connection to the SQL server!?

@ww I hope that explains what the 2nd MX should be doing.

@jdsilva No I understand the idea of a warm spare but that is not what we are trying to do we need both to be active and in use at the same time.

@Adam Your reply is worrying as it suggests that we can not replicate the original setup with MXs in place of the ASAs do you still think this is the case with the explanation above? 

Also each ASA had just 3 interfaces and so do the MXs as listed above.

@PhilipDAth Yes spot on the two ASAs are completely independent of each other they just sit on the same LAN and DMZ network and that is what we are trying to replicate with the MXs

Many Thanks

Ian

[Adam]

More questions sorry:

1.  Do you have two active internet links total?  Or are two internet links connected to each ASA separate?

2.  Are the MX84's both in the same dashboard network or different networks?  If they are in the same network only one will be active.  If they are in separate networks they can both be active at the same time. 

[IanMartin]

Hi Adam, 

No problem.

1. We have two independent internet links total (2)

One link connects to ASA 1 and the other link connects to ASA 2

and thats how we cabled the MXs on 1 link 1 MX (It has to be this way as the internet links are in different buildings)

2. We can see both MXs in the Meraki Dashboard but I think they are different networks as we have to select an MX to configure or look at statistics of each one. here is a screen shot.

EDIT

But both MXs have this same network config see screen shot.

Many Thanks for your continued help.

Ian

[jdsilva]

They both have the same IP's?  

[IanMartin]

Hi jdsliva,

Sorry no the MXs interfaces have unique IPs

MX 1

Internet port = 80.194...

LAN port = 192.168.100.2

DMZ port = 172.16.30.1

MX 2

Internet port = 62.255...

LAN port = 192.168.100.10

DMZ port = 172.16.30.113

But as you can see both the LAN and DMZ IPs are within the same network range because we have just one subnet for LAN and one for DMZ. The LAN and DMZ are physically separated networks different switches and cables. We have no VLANs at all.

Thanks for your help.

Ian

[jdsilva]

OK, I thought that's what you said above. just clarifying.

Oddly enough, the behaviour you're describing does sound like it could be caused by an IP conflict... But that's only one possibility.

I think I'm thinking like @PhilipDAth here... Maybe there's a stray link somewhere? 

A packet capture during the problem might show what's up too... But of course that means putting everything back in place an inconveniencing your users.

[IanMartin]

I've just double checked and the MX1 and MX2 LAN and DMZ ports all have the correct IP's no duplicates.

I guess well have to look at a packet capture with both MXs up and running.

Thanks

Ian

[jdsilva]

One other thought... Are there any redundant links here? The MXes do not run Spanning Tree so there's a risk of a loop if things aren't set up correctly. That may also explain your behaviour.

[IanMartin]

What do you mean by redundant link?

I've never really got my head around spanning tree!

I always assumed it was to do with accidentally creating loops on the network.

Some of our switches have trunked fibre ports to give larger bandwidth between buildings and i know from painful experience if the trunk gets broken so the two fibre are no longer linked and they essentially create a loop on then all hell breaks loose on the network!

[Adam]

For each of the two networks, do they have unique LAN/DMZ addresses?  Are you using any of the Site to Site VPN features of the MX?  If a computer from one of the LANs wants to talk to a computer on the LAN of the other network what path does it take?  Does it go out one MX and back in the other or is there a local path?

[IanMartin]

Hi Adam,

We don't have two networks there is only one LAN 192.168.100.0/22

We don't have site to site VPNs

ALL PCs and Servers sit on the same LAN

Thanks

Ian

[T-800 · Accepted solution]

Yes there is something fundamentally wrong with your setup. You essentially have a "flat" LAN and achieve fail-over by changing the gateway address on the NIC of the PC/device. You don't have 2 LAN's and you don't have separate networks. (This is how I understand your post; correct me if I am wrong)

**If you had to plug the cable into a different network jack to achieve fail-over, that would be separate networks as you have described it.  

What you have is one network with two different L3 gateway addresses. Layer 2 is on a single broadcast domain. ARP is being done by two separate sources and that is your issue (which could lead to IP conflicts and other weird things)  

Minimum best practice would be:

--WAN

MX84#1

Internet 1 = ISP1

Internet 2 = ISP2

--LAN (use 3 separate ports to the switches or a trunk por to switches)

Data VLAN

Phone VLAN 

DMZ VLAN

** Switches would be setup with voice VLAN's if computer plugs into phone.

*** MX84#2 should be used a warm spare so that ARP table is synced between devices. 

This would solve your issues and let you achieve fail-over without changing the computer IP address. It would also let you make load balancing rules on the 2 WAN connections. Assuming your switches support VLAN's/Voice VLAN's this could easily be changed overnight, depending on number of switches/clients/floors/how well things are organized.  

-T-800

[IanMartin]

Hi T-800,

Your first sentence is spot on that is exactly what we have!

(well what we inherited from previous employees no longer here anyway)

Would i be correct in saying that the ASA5510's were not acting as L3 gateways ?

Hence us not having any problems with them in our setup?

Also

Can i just add that we are using public IPs from BOTH ISPs for web and ftp services at the same time.

So we have some servers pointing at ISP link1 and some pointing at ISP link2 Could that be achieved with your suggested setup?

Many Thanks for your help and advice.

Ian

[T-800]

I believe the ASA's were true L3 gateways, but they may not have been as picky about ARP, or basically may have just played nicely with each other. There are certainly other options as to why, but I wouldn't worry about trying to figure out why it was actually working. Move on and just fix it.

*It would be a good idea to make sure the switches are indeed operating in L2 mode with just a management interface and are not doing any routing. 

For incoming services from both ISP's you would look under the "port forwarding" section of the firewall configuration page. Each of the options let you choose the up-link (Internet 1 or Internet 2), so you should be able to use both connections for different services inbound. You'll need to figure out if you which method you were/are doing for inbound connections. 

For true DMZ, you'll need to define in the firewall that the "DMZ VLAN" can't talk to the internal "Data VLAN" as the "default" rule. Then make individual rules to allow communication as necessary. Meraki's essentially have all VLANS as same security zone (ASA lets you set security levels so that DMZ can't talk to zone with a higher security level) 

These articles will be helpful:

Port forwarding and NAT rules:

https://documentation.meraki.com/MX-Z/NAT_and_Port_Forwarding/Port_Forwarding_and_NAT_Rules_on_the_M...

Creating a DMZ on the MX

https://documentation.meraki.com/MX-Z/Firewall_and_Traffic_Shaping/Creating_a_DMZ_with_the_MX_Securi...

T-800

[jdsilva]

Oh hell, I just thought of one more thing.

By default MXes will created a DHCP scope for every subnet configured on them, and assign addresses to clients. If you have two MXes in two different Dashboard networks, they will both try and assign clients addresses with themselves as the gateway, and they could assign duplicate addresses to your clients. 

How are you handling DHCP on your LAN?

[IanMartin]

@jdsilva 

That's one thing we don't have to worry about as we run fixed IP for everything.

(something else we inherited!)

I disabled DHCP on both interfaces on both MXs before we tried going live.

@T-800

I'll take a look at those articles thanks.

EDIT

Following a call with Meraki Technical sales this morning it is clear we have a non standard/recommend network setup, Meraki do not support two live active MXs on the same network!

We will have to make changes to our network setup to get them to work properly and hopefully improve the network in general.

In light of this @T-800 I've marked your reply as the solution.

Many thanks to all that have contributed to this thread it has been very enlightening and now looks like I have quite a bit of learning and work to do.

Ian

[Adam]

This may be an opportunity to consolidate those connections to the MX as redundant load balance links (WAN1 and WAN2).  But with that aside.  Do you have to select the network drop-down box on the left to go to each MX or do they show up like this in the dashboard?

[IanMartin]

We have to select the network drop down box to select either MX