Looking for thoughts on load balancing and vpn on mx100

[Kent]

Hi Community!

I need to implement a redundant/load balancing system between to buildings about 5 miles apart. We are connected via 2 microwave links (on different towers, two 1Gb feeds from each tower), VPN via Internet (1Gb and 10 Mb), and maybe a LTE device to get some email out if all the above goes pear shaped.

I was planning on initial 2 MX100's then add two more as warm spares.

I expect each mx100 would look like:

MX -- WAN1 -- Internet (Access VLAN 100, Auto VPN)

MX -- usb -- Modem LTE Internet failover

MX -- GbE1 -- Microwave 1, Radio 1 (Trunk)

MX -- GbE2 -- Microwave 1, Radio 2 (Trunk)

MX -- GbE3 -- Microwave 2, Radio 1 (Trunk)

MX -- GbE4 -- Microwave 2, Radio 2 (Trunk)

MX -- GbE5 -- Core network (Bonded)

MX -- GbE6 -- Core network (Bonded)

MX -- GbE7 -- Core network (Bonded)

MX -- GbE8 -- Core network (Bonded)

Will this work? Better ideas? 

Thank you.

Initial drawing below.

[Tony_Ang]

Yep, very good.

Should be fine.

But note you will require 2 WAN/Public IP from each of the Microwave ISP for the MX.

1 - MX100 WAN Port 1 - WAN IP 1 from ISP 1
1 - MX100 WAN Port 2 - WAN IP 1 from ISP 2
2 - MX100 WAN Port 1 - WAN IP 2 from ISP 1
2 - MX100 WAN Port 2- WAN IP 2 from ISP 2

[Kent]

The microwave radios connect from building to building. Internal only. Not an Internet link. I am treating this as if they were fiber links between buildings.

[PhilipDAth]

Is your only concern providing redundant connectivity between the buildings (as opposed to redundant Internet connectivity)?

Are you wanting a layer 2 stretched network between the buildings (so they share the same subnet), or can each site have its own subnet?  For example, the "red" Police network has one subnet at site 'A' and a different subnet at site 'B' - or does it need to be the same?

[Kent]

The Internet VLAN (100) is connected to 2 Cisco routers using BGP to the two Internet links. It has as a 24 bit subnet and can be routed from either building's Internet router (currently a Cisco 2800 router and an ASA). 

The police VLAN (101) 24 bit subnet , and City networks (102-130) 20 bit subnets, are current working across a trunk port like:

Cisco 4900m -- Microwave 1 Gbs -- Cisco 3850 (primary) (Spanning Tree enforced)

Cisco 4900m -- 802.11 30 Mbs -- Cisco 3850 (backup) (Spanning Tree enforced)

This link has failed (after 11 years 😉 and I have the opportunity to spec out a new solution. 

There are rules to where the City network and Police network can access. I might remove the police VLAN from transportation completely (some new networking rules are coming down the pipeline).

I am hoping for something like:

Meraki MX100 -- Microwave 1 Gbs -- Meraki MX100 (load balanced)

Meraki MX100 -- Microwave 1 Gbs -- Meraki MX100 (load balanced)

And have a 2 Gbs links between buildings with failover to 1 Gbs

[PhilipDAth]

And when you say "load balancing" - do you mean across the two microwave links - and the remaining Internet and LTE circuits are used purely for failover?

[PhilipDAth]

Also note that the MX series does not support LACP/Bonding.

[Kent]

Both sides of the microwave link have their own Internet links - 1 Gbs and 10 Mbs. For most users, the Internet access is via the 1 Gbs link, if that is down, then all users will go out the 10 Mbs link. If the 10 Mbs link is down, I expect to have a LTE Lin up with traffic shaping rules that allow only specific high priority traffic.

[Kent]

Yes. Normally, the 10 Mbs internet connection in Transportation is not used. Only for failover.

[PhilipDAth]

Hi @Kent.

If you had engaged me - I think I would be recommending the user of Cisco Enterprise kit for your job.  I have the feeling that aspects of public safety may be involved, and for me that raises the bar on the design being super robust and to handle a greater range of natural and unnatural disasters than a normal "corporate" solution might require.

However, lets continue along the Meraki vein.

The first thing to note is you are using the Microwave links for layer 2 extension and that Meraki AutoVPN is a layer 3 technology (meaning AutoVPN would require a different subnet at each site).  So we have an issue there.

So accepting that limitation I would be using more than one technology.

First, I would use LACP and bond your two Microwave links together.  If you were using a Cisco Enterprise switch I would recommend LACP "fastrate" which would allow sub-second failover between the Microwave links.  Meraki doesn't have this feature (I have asked for it so many times ...).

I don't know the answer for sure but I would expect the LACP hello time to be 30s, and that you would have to loose three of these before declaring a microwave link down, so the failover time would probably be 90s.  These are "standard" times - I'm not sure what Meraki MS uses - but they are probably the standard values.

You could use an existing core switch for this, or you could put in a little pair MS220-8 switches.  Personally I would probably run each microwave link into a switch stack, with each link plugged into a different switch.

So that is the best I can think of to manage the layer 2 side.  Now onto layer 3.

You would obviously run an Internet circuit into the MX at each site.  Yes you can plug in an LTE device as well for failover.  You could also run a transit VLAN between the two sites which is only used to connect the MX's, and you could run AutoVPN over both sets of links (there is an extra caveat here, but lets put that aside for the moment).  You could also use simple tracked routes with AutoVPN failover.  NOTE this will only provide redundant access to layer 3 domains at each site - basically to VLANs you are not extending over the microwave links.  This scenario is discussed in this article:
https://documentation.meraki.com/MX-Z/Deployment_Guides/MPLS_Failover_to_Meraki_Auto_VPN 

@Kent I would encourage you to engage with a local Cisco partner on your solution.  I get the impression that your solution may involve aspects of public safety, so it is really important to get the right design.  Hopefully my comments have given you more ideas to think about and how they might be incorporated into your solution.

[Kent]

PhilipDAth:

The existing Microwave link is connected via Cisco 4900m to 3850stack with backup 802.11 on different switches. That said, I was requested to design using only Meraki equipment as the Cisco devices are no longer under maintenance and will be retired - they want to move completely over to Meraki by attrition.

During a microwave and Internet failure, the LTE link will only be used by EOC mail and web services - via traffic shaping. 

We may completely remove the police VLAN from transportation - this is still in discussion. 

I brought this up about 6 months ago to the government cisco rep. I had trouble getting him to use only Meraki equipment 😉