MX with MPLS and NO Internet

[AdamAus]

Hi, I've battled with this issue in the past, and am about to do so again, so wanted to check if I'm doing it the "right" way.

I have a multi-site client with an MX at each site, and a single MPLS connection at each site, that will route all traffic back to HQ.

At HQ, the MX has an internet connection on WAN1 and the MPLS connection on LAN port with static routes.

At the branch, the MPLS connection connects to a cisco router, which then splits the /24 into a /25 and a /29. The /29 is connected to the MX WAN as a Internet connection, and the /25 is connected to the MX LAN with static routes.

All that means that we need additional cisco router, or layer 3 switch with VLAN, but at least it works.

I want to reduce that to simple connect the MX directly to the MPLS, preferably on the WAN, and allow it to get Internet access, and route private IP address space. I think the issues with this are:

1) The MX will NAT the private IP addresses, instead of just routing them and letting the HQ MX do NAT at the Internet gateway

2) The MX requires a working Internet connection on all WAN ports (ping to 8.8.8.8 at a minimum)

What do others do? Is this functionality better in firmware 13/14 ?

[PhilipDAth]

1. The MX will NAT any destination address not in its routing table, with the exception of the default route.  So you need to include a route for all private IP address in your network on the other side of the WAN port of the branch MX.  Traffic to the Internet will be NATed by the branch MX.  The only other except would be if you were running the MX in transparent mode ... but you are going to loose a lot of features.

2. The MX must have working Internet connection, so this can be provided via the HQ connection.

There are two common ways of doing this.  First use AutoVPN and run a VPN over the MPLS network.

https://documentation.meraki.com/MX-Z/Site-to-site_VPN/Configuring_Site-to-site_VPN_over_MPLS

The other is to simply use routes, or tracked routes if you want to be able to fail over to a backup VPN.

https://documentation.meraki.com/MX-Z/Deployment_Guides/MPLS_Failover_to_Meraki_Auto_VPN

[tomkiv897]

deleted

[ww]

why not use a cheap internet provider or 3/4g dongle in addition.

[hilehoffer]

Usually to provide SD WAN you would have an MX in one armd mode at HQ.  In your environment I would try connecting WAN 2 to MPLS and then it should be fine for all the MX appliances to NAT.  The autovpn will form an overlay on the MPLS network. You should not need additional routers.  In this setup your MPLS provider will not need any lknowledge of your internal networks, as they will be nat'd via the MXs.

[ali_abbass85]

I have the same case, where I would like to run MX devices(hub and spoke), on both I will connect the internet to WAN1 and MPLS(without internet access) directly to WAN2, then form an SD-WAN setup over both links.

This what I will try! I hope it works!

[ali_abbass85]

Hi @hilehoffer 

This did not work for me, i am using a pair of MX68 devices where both are connected on WAN1 for Internet and on WAN2 for MPLS.

I could see the tunnel is up on the WAN1, but will never be up on the WAN2, is there a special configuration I need to add?

[PhilipDAth]

If the tunnel is not up on WAN2 (so assuming you are using AutoVPN over MPLS) make sure the that the public IP address that the spoke gets NATed to is the same as the public IP address that the HUB is nat'd to when the traffic goes over the MPLS, otherwise it wont work.

[ali_abbass85]

Hi @PhilipDAth Actually I am trying to achieve what we have Auto VPN over MPLS and SDWAN:

https://documentation.meraki.com/MX/Site-to-site_VPN/Configuring_Site-to-site_VPN_over_MPLS

https://documentation.meraki.com/Architectures_and_Best_Practices/Cisco_Meraki_Best_Practice_Design/...

With the following assumptions:

-Both MX Devices will run in NAT Mode (Not VPN concentrator)

-Both MX Devices will have one leg to MPLS (which has a router splitting traffic to the internet)

-Both MX Devices will be connected to WAN1 to the internet

I cannot get the tunnel on WAN2 up, I am not sure how to configure it.

[PhilipDAth]

The WAN2 interface of everything MX, including the DC/head office, needs to get NATed to the same public IP address.  If you look in the dashboard, does it report the same public IP address for WAN2 for each MX?

[ali_abbass85]

Thanks @PhilipDAth 

So we got everything set now, and both MX devices see there interfaces as active.

However I am facing an issue with VPN tunnels, when both lines (MPLS and Internet) are up and running, everything works great (eventhough I did not test SDWAN and path selection yet).

However when the Internet line is down at any of the devices, the tunnel would be down and it seems that the Internet1 of the working Internet line is trying to form an IPSEC tunnel to the other MX device over the MPLS link! and it not getting up (that is normal because the MPLS connectivity does not allow traffic from the outside except for Meraki Cloud).

I attached a diagram.

[Maccaa31]

Hi @ali_abbass85 

I am looking at doing a similar configuration but ran into similar issues. Did you have to setup any flow preferences and enable NO-NAT to get this working? I am looking at trying again this week to get this working. 

Thanks

[ali_abbass85]

Hello @Maccaa31 

yes this actually worked for me and has been working perfectly for years now with no issues at all.

Make sure your MPLS has a breakout to the Meraki cloud IPs (same as the diagram I posted above), the IPs are in the following link:

https://documentation.meraki.com/General_Administration/Other_Topics/Upstream_Firewall_Rules_for_Clo...

[Maccaa31]

Thanks for the tip, that might be why I got a "Failure" message on the MPLS connection when I last tried to get it working. I will let you know how I go. Much appreciated! One other question, did you have the MPLS port set to Internet or LAN?

[ali_abbass85]

No problem @Maccaa31 I am sure this would work for you, as worked for me

Answering your question, the MPLS should go to the internet port(1 or 2) on both MX devices (look at the diagram).The MX devices will treat it as an internet link, as long as you are allowing the Meraki cloud IPs from the MPLS. Technically you would be cheating the MX devices to think this is an internet link.

Do not forget to allow ICMPs to Google 8.8.8.8

[stroighne]

I used internet line into WAN 1, with MPLS into LAN port 1.

Created the Site to Site VPN and included the MPLS subnet into the VPN.

Created a static route pointing all traffic destined for any MPLS connection 172.16.0.0/12 (for example) via the MX MPLS subnet gateway IP address. 

MX will always favour the static route over the VPN route. So no NAT'ing to worry about on MPLS traffic, say voice or other business critical apps that dislike NAT. Local breakout too, which speeds up things like office 365 over going through a central HQ/Datacentre. 

We have the advanced security licence to help with security. In a nutshell, that's pretty much all I did. 

Everyone's a winner.

HTH. 

[ali_abbass85]

I depends on the type of applications/ traffic flow you have. In my case I do not mind traffic being Natted on MPLS network (no voice traffic), and I was able to achieve what I want from the setup

However I would always prefer to have WAN links to be connected to WAN1 and WAN2 then achieve an SD-WAN setup or load balancing

For your case, you are stuck with static routes which hinders the power of SD WAN policies. Meraki recently added to the beta firmware release 15.2 a feature for NO-NAT where the MX behaves like a L3 device, but does not do NATing on the uplink. This makes a perfect option for your case, if you would like to take the risk of having running a beta version

[stroighne]

Agree re: no NAT beta version. However we're not an organisation prone to taking 'risks' such as using beta software. Should it ever make it to a stable version, then we're likely to be all over it. 

[TheRealPucci]

@stroighneI am doing the same setup you are. Were you able to successfully get the MPLS network up and running?

[cmr]

@TheRealPucci are you trying to do SDWAN over MPLS without using a concentrator, or are you okay with a concentrator?  We have a 9 site SDWAN using VPLS and MPLS as the two WANs (neither with internet) on v15 with a central concentrator, but fully meshed.  we have Cisco voice and video and Zoom video but with NAT enabled.  It seems to work quite well for our ~800 users.  I guess we should try no NAT but perhaps we don't need it?

[TheRealPucci]

No SD-WAN currently. i just need to decommission old hardware and i am putting in MX100's. I have a broadband connection and i also have a verizon MPLS circuit for all of my sites and our hosting vendor.

[TheRealPucci]

@cmrwe have a verizon MPLS network that has the last leg going into our hosting vendor. I am trying to use BGP from the MPLS as this is what they use currently.

[stroighne]

Hi, yes of course. 

Ultimately I created static routes to the entire RFC1918 address space. Which had the result of pushing traffic destined for our MPLS network over the MPLS connection. Anything else then went via our local internet break out connection. 

I decided against connecting the MPLS to a WAN port as we only currently have 1 pair of MX450 VPN MX concentraters, and didn't want to cause any additional latency and/or a bottle neck. 

Works as it says on the tin. Simples. 

[TheRealPucci]

@stroighnesorry for my ignorance, you are telling me you created static routeson your MX that pointed to the MPLS network and that worked?

[AdamAus]

Some of the replies here seem to be correct, and I finally had the chance to do this on a live network, and can confirm it does work, with some pre-requisites.

1) One site must have Internet access, for us, this is the "Head Office" or "Main Site". It could be your DC or whatever. Connect this to WAN1 (or WAN2 as needed).

2) The MPLS connection will be on LAN side of your head office/main site. Add static routes as needed, so your head office knows to use the MPLS connection.

3) Your remote office will connect the MPLS to WAN1 (or WAN2) and if you have a backup Internet, connect it to WAN2.

4) Use site-to-site autoVPN

5) Set your traffic preferences to use the MPLS (or split however desired).

In my case, my MPLS doesn't have an Internet connection, so it took some discussion with Meraki Support to accept that this happens in the real world.

I also didn't want to buy another MX just for VPN, I already had a perfectly working MX acting as my Internet gateway/router.

Also, I wanted the Internet on WAN ports, and also the MPLS on remote sites on WAN ports.

Finally, I wanted automatic failover from MPLS to Internet without any manual intervention.

My understanding is that the remote site has "two internet connections", one is the backup Internet on WAN2, the other is via the MPLS to head office, and then out the Internet connection there. The reason this magically works is because meraki "see" the autovpn on from the remote office WAN1 and the head office as having the SAME IP address, so it shares the internal IP's with both of them, and they then setup the VPN directly, and thus it works.

If you want more details, let me know, and I can add in some more IP address examples, and try to expand on the description. I'm hoping to roll this out for another customer shortly. My plan is that all remote sites would have a second Internet on WAN2 (4G mobile), with MPLS on WAN1.

[KarlH]

@AdamAus , would you please share your topology with IP addresses?

[Gingerking]

Were your connections to the MPLS routers at branch sites utilizing individual /30 networks? It kind of sounds like your MPLS connections were all part of the same network. 

[TheRealPucci]

@Gingerkingyes, all of my MPLS's are on the same network. Its through a Verizon MPLS so they all have an IP and a BGP for the next hop.

[zball8888]

Hey this is a long shot if you reply but i am somewhat in the same position. I have a branch sit the is using a mpls with no internet backing. The idea is to put a mx in place at the branch and setup an autovpn over a internet connection and have the mpls as backup. but i am being told my a senior tech that the mx will not do vpn over the mpls circuit without internet. the branch site already has a isr router and my thought is to leave the router (to do the bgp/mpls) and let the mx vpn through the mpls.

I dont quite understand this works. how does the mx get to the internet  trhough the mpls>head office

do you have to do pbr are static routes to allow the mx to reach the cloud through your hq?

"My understanding is that the remote site has "two internet connections", one is the backup Internet on WAN2, the other is via the MPLS to head office, and then out the Internet connection there. The reason this magically works is because meraki "see" the autovpn on from the remote office WAN1 and the head office as having the SAME IP address, so it shares the internal IP's with both of them, and they then setup the VPN directly, and thus it works."

[Oleh]

Adam:

I would love to get more info on how you did this cuz I am having the same issue.  My MPLS connection does not have Internet so the MX always sees it as Failed if I have it in my WAN2 port so it never gets used.  I know layer 3 is working on MPLS on the WAN 2 cuz when I do  a traceroute and force it out WAN 2 i get expected results.  Did you find a way to bypass the "Internet" checks that the MX always does on the WAN ports to show them as either failed or Up ?  I know its more that just pinging 8.8.8.8  ANy details of how you got it to work on WAN 2 would be greatly appreciated

[cmr]

@Oleh we run multiple MX SD-WANs over MPLS networks without internet.  At the main hub site the MPLS is terminated in L3 switches rather than MXs and the MXs are in VPN concentrator mode.  The L3 switches have a route to the internet and this is advertised over the MPLS to the remote sites where the other MXs are installed.

[ali_abbass85]

Hi @Oleh 

Well I got this resolved and running smoothly for more than a year now, by having an internet breakout somewhere on the MPLS, you only need to allow access to a set of IPs and subnets and advertise them to the MPLS network, where they can be reached by remote MX devices

I am assuming you are not running MXs in Concentrator mode at the HQ.

Look at my diagram above.

[Oleh]

Thx Guys for the reply.  I made some progress and did as suggested.  I added a route to the Internet for the MPLS segments and lo-and-behold the Uplink went green !!  SO now I am testing the ability to use both links.  So my MPLS is WAN2 but it seems to be ignoring my Flow Preferences unless I really force it with a static route.  IS this normal ?  I would rather regulate my traffic via FLow Preferences in SDWAN rather than static routes

[cmr]

What are the relative throughput rates, latency and packet loss of the connections.  You can steer traffic based on the latter two quite easily.

[Oleh]

Hi cmr,

The MPLS circuit is much smaller, at only 10 MPBS but it has 6 classes of servcie end-to-end and lower latency to my media site so I just want to make sure that voice/video and media go over the MPLS where I can control things a little better 

[cmr]

@Oleh you can use the built in bes for voice and video rule in the SD-WAN & traffic shaping section, or build some of your own as below:

 

(apologies for the terrible redaction)

[nvrdone]

We have our MPLS plugged into WAN1 at the branch, and landed the MPLS connection at our datacenter onto our Nexus 7k.  We created a VLAN that allows traffic out to the internet only, and ACL'd our local stuff out of it.  We then created a VRF and landed the MX into the nexus on the same VLAN.

The down side to this is that the headend MX will always show WAN1 as the internet traffic for it bleeds out the VLAN we created and up to our 3rd party firewall.  The branch sites will work correctly as if the MPLS is down then the meraki traffic can't bleed into the vlan because it can't get to the vlan.

We do have internet at all locations as well plugged into WAN2.

[Newbie99]

Hi nvrdone,

I am.planning to do what you have done with your setup. Could you share with me your configuration? Thank you so much