Connecting MX's with a Layer 2 connection.

[MDHackett]

A local vendor has provided us with a private, Layer 2 connection, handing off copper to our 100Mbps sites (MX64's), and fiber to our 1Gbps sites (MX250's).  The connections we're getting are completely un-routed, of course, and effectively just a big switch that connects the sites.

Without that connection having internet access, we're struggling with how we should configure the MX's to pass traffic.  Since it's not an MPLS, there's no router to talk over.

My gut tells me we'll need to have the provider re-jigger the connection to a straight internet and AutoVPN the connection, but we're still looking for options to make this worth without having to do that, and let a cable modem connection be the backup and management.

Has anyone done this, and what should the configuration look like on the MX's?

(The sites we're connecting are VERY flat, and have only 1 subnet, except 1 (MX250)  site that has 2.)

[KarstenI]

First question is if you want to have all traffic encrypted that is passing over this connection. Then you likely can use the documentation for the MPLS-AutoVPN.

If traffic over this connection can be sent in clear, without encryption, then you can configure an additional MX LAN-interface on each MX, all in the same IP subnet and configure static routes with route-tracking pointing to the other sides MXes.

You do not say how many sites you have, if there are a couple of hundreds, the second option will certainly not scale. 

[MDHackett]

I've looked at the MPLS-AutoVPN documentation, and I'm not sure it applies, as the L2 connection doesn't have any address/routing going on inside it.  The handoff is basically a connection to a switch, with no internet connection, so I can't see how we would route to it the way you would an MLPS router.

The L2 link only connects our ports, and have no other traffic inside them.

There are 9 sites.  6 of them are MX64's as small branch locations with between 5 and 20 users at a site, they're all flat networks (10.1.AAA.BBB)  The other 3 are MX250's, that include our Data Center (with our firewall connection out), and our main site with about 50-60 users (who has a 2nd subnet for VoIP phones, 10.1.1AA.BBB and 10.1.2AA.BBB), and our backup site with a small set of hosts and about 25 users.  

Let me make sure I'm hearing your suggestion:  Setup WAN1 for internet/maintenance (cable modem/DHCP) connection, WAN2 to the Layer 2 connection and assign all the WAN2's in the same subnet (10.2.11.xxx).  Then static routes like 10.1.AAA.0/24 (next hop) 10.2.11.xxx?  I don't think it'll let me drop in that static route, because it doesn't see the WAN2's 10.2.11.xxx address.  (Yeah, it wouldn't let me.. "..invalid next hop IP. The IP address 10.2.11.xxx is not on a configured subnet." )

[Bruce]

@MDHackett, all the Meraki devices need access to the internet. This can either be directly, as with a MX connected to an internet connection, or indirectly, like a switch behind a MX or a MX on a MPLS connection that has access via a head-end site. Since you state "Without that connection having internet access"  about your Layer 2 network you'll need to provide the internet access yourself, most likely via a head-end. You could do this by creating a VLAN on the MX250, assign an IP address, and then assign IP addresses to each of the MX64 in the same IP subnet (you could use DHCP is the Layer 2 network supports this). The MX64s use the VLAN interface as their default gateway, then you connect an internet service to the WAN port of the MX250 

As @KarstenI stated, whether you use AutoVPN or not comes down to whether you want encryption or not. If you do want encryption then AutoVPN tunnels are only formed between WAN ports on MX devices. So in the scenario you have you'd likely end up needing another device to provide internet connectivity, and then have the head-end (MX250 - or a smaller MX, and use the MX250 for the internet firewall) in VPN concentrator mode, behind the device providing internet connectivity.

[KarstenI]

Hmm ... In contrast to @Bruce I interpreted the original post in a way that all sites have already MX devices with internet and the L2 connection is an additional way to connect the sites. That perhaps needs some clarification.

[MDHackett]

@Bruce The Layer 2 network is meant to be upstream/outside.  Each MX stands between the LAN and the Layer 2 connection.  Encryption isn't an issue on the Layer 2, as we're the only traffic passing on it.  All the traffic that's not bound for one of the 10.1.AAA.BBB networks (we only have 10 subnets for the 9 locations), should go to our 10.1.15.254 destination to be parsed out the Firewall.

The Layer 2 connecting fiber doesn't have DHCP in it. But you're suggesting I use a LAN port at the Data Center to connect to that Layer 2, assigning the WAN ports on all the other MX's on that subnet, then any non-local traffic would naturally flow up to the Data Center, which could route 0.0.0.0/32 to our internal Firewall at 10.1.15.250.  The Data Center would have a configuration/maintenance connection on the WAN port.  This wouldn't include the use of the Auto-VPN.  I'm just trying to mentally sort out how we would get traffic passing between the MX64's under there.  10.1.2.0/24 to 10.1.3.0/24 to 10.1.3.0/24, etc..

Is that what you're thinking?

[Bruce · Accepted solution]

@MDHackett yes, that's what I was originally thinking, downside is that all traffic from the MX64 will get NATed to the IP address on the WAN interface of the MX64, which may not (or may) work for you. That NAT is how the traffic gets back to where it came, but obviously hides all the addresses behind the NAT.

You can achieve this without encryption using the No NAT feature (you have to request) support to turn it on, on a per network basis, but then you have to make sure all your routing on the head-end is correct to send data back across the Layer 2 WAN. The other way of achieving this it to use the AutoVPN.

Since you already have a firewall with internet access you can set your MX250 up as a VPN concentrator. Establish an IP subnet across the Layer 2 WAN, assign the MX64 WAN port an IP address from the WAN subnet, put a gateway for the WAN subnet at the head-end, and put the MX250 in concentrator mode at the head-end (so you can route to it from the WAN subnet). Then establish AutoVPN between all the MX64 (as spokes) to the MX250 (as a hub). The traffic will then be tunneled across the Layer 2 WAN and won't be NATed. You'll need a route to all the networks sitting behind the MX64s pointing to the MX250 (the MX250 will learn all the remote subnets through AutoVPN), and the MX250 will need a default route to the rest of your network. (You could also do the head-end routing with OSPF or BGP if you've a neighbor to peer with).

Essentially you are doing this, https://documentation.meraki.com/MX/Deployment_Guides/VPN_Concentrator_Deployment_Guide, but with the Layer 2 network in place of the Internet. You just need to ensure that there is a path from the MX64 WAN IP address to the internet so they register with the Meraki cloud. This would, be via the existing firewall.

[cmr]

In addition to @Bruce's comment above, we actually run a setup like this where each site has 2 WAN connections from the MX pair, with either one or both as VPLS (L2 MPLS) circuits.

At the main data centre the WANs are terminated on routed ports of a L3 switch stack and the MX250 pair are connected to them as single ended VPN concentrators, terminating all of the SD-WAN site-to-site connections.

It's worked well for about 2 years now on 15.x firmware and although we got no-NAT enabled, we never used it as the user traffic all goes through the tunnels anyway, with only the MX to cloud traffic passing outside.

[MDHackett]

@Bruce We're working to resolve this through attaching WAN1 from all the remote sites (big and small) to the L2 fiber connection and bringing those into the Data Center LAN port in it's own subnet, drawing an internet connection through WAN1 on the Data Center, allowing them to enable an AutoVPN connection to one another to allow them to control their own routing.

I've connected WAN2 at the remote sites to a cable modem as fail-over for the AutoVPN and management if it should otherwise lose connection to the L2 fiber.

We're in the midst of rolling this into place at all our locations for testing before we cut over on a weekend to live, but so far it looks very promising.