NAT Exceptions on MX68CW

[RichardChen1]

FW: MX 15.27

I am testing the "Direct Internet Access with MPLS failover" use case described under this Meraki doco.

https://documentation.meraki.com/MX/Networks_and_Routing/NAT_Exceptions-No_NAT_on_MX_Security_Applia...

Questions from my test:

1. Both WAN 1 and 2 must have internet access for this to work? In other words: WAN1 and 2 must be online first? What if my MPLS link does not have internet access?
2. If I enabled NONAT on WAN2, I can't see "Inbound Rules" enabled under Firewall options. But as per the guide I should do a deny any any. What's the expect option?
3. To send traffic to WAN2, I need to apply PBR under Uplink selection -> Flow preferences right? Not to use static route? My idea is to force certain MPLS subnets/routes to go via WAN2/MPLS.

Other question: I can't terminate MPLS/EBGP on MX because EBGP is on BETA and only supported on concentrator mode right?

[PhilipDAth]

I don't know the answers for sure.

>Both WAN 1 and 2 must have internet access for this to work?

My expectation is yes.  If it can not get to the Internet it will consider the path to be failed.

>What if my MPLS link does not have internet access?

Then that path would be considered permanently failed and unused.

You are probably better off considering a design more like this (just ignore the AutoVPN failover part):

https://documentation.meraki.com/MX/Deployment_Guides/MPLS_Failover_to_Meraki_Auto_VPN#:~:text=The%2...

Here the MPLS connects via a VLAN interface on the MX instead of a WAN interface.  The MPLS network does not require Internet access in this case.

[iConnect]

About "MPLS failover to Meraki auto VPN" solution, at first sight looks great but when you go deep on it you realize that the solution is very limited. The problem is that option "Active as long as a device on the other side responds to ping" is limited to a host / interface directly reachable by the MX (same segment). 

Above image has been taken from the Meraki document. Here, the MX on the left is able to check MPLS availability by pinging the interface 192.168.128.249 (cannot do it against 192.168.129.249). The problem here is that interface 192.168.128.249 can remain up even if the link is down, in which case will never choose the second route over the VPN.

Quote form the doc:

The next hop IP cannot be out the internet interface or over another static route.

Am I missing something?

[Bruce]

You should be able to get it to work. I think below is the way to do it (although I haven't tried it), its not exactly as per the document, but hopefully should achieve the outcome you require.

1. Don't include 192.168.128.0/24 or 192.168.129.0/24 in the AutoVPN (if you do it will certainly break)
2. Configure a static route, 192.168.0.0/16, that points to the local next MPLS hop. So in the case of the 'left' side this would be 192.168.0.0/16 is via 192.168.128.249. This should be set to always available. Obviously this is great if you can summarize all the subnets on your MPLS WAN, or you'll have to add more specific subnets. It also assumes the subnets on the MPLS networks are only used for transit, no hosts.
3. Configure the static route, 10.0.0.0/24, on the 'left' side with the next hop of 192.168.128.249. This should be available if a host responds, and the host should be 192.168.129.1 (i.e. the MPLS connection on the MX on 'right' side).

The theory is this: the 'always' static route ensures that the ping to test if the host is alive is always sent via the MPLS network. If that ping test fails then the 10.0.0.0/24 route will be removed, and traffic sent via the AutoVPN. However, when the AutoVPN is being used the 'left' side pings to the 192.168.129.1 will still be sent to the MPLS (even if it is a black-hole) and never via the AutoVPN since the 'left' side doesn't know that 192.168.129.0/24 is accessible via the AutoVPN.

I think that should get you working.

Forget that - the testing against a remote host doesn't work that way; the host has to be in the subnet specified in the static route (which won't work). Looks like you have to test against the next hop, which means you're only testing for a failure of the local link, not the path all the way through to the other side.

The alternative would be to turn this into a SD-WAN solution, but in that scenario you'd either need internet access from your MPLS cloud, or to configure one end as a VPN concentrator - which means another firewall (potentially MX) required.

[iConnect]

Thanks @Bruce .About  "The alternative would be to turn this into a SD-WAN solution, but in that scenario you'd either need internet access from your MPLS cloud, or to configure one end as a VPN concentrator - which means another firewall (potentially MX) required.", that is something we do pretty often and works. There is another reason (not part of this threat) why this option is the one we need.

Any way, we finally made it work and it goes like this. For the redundant route, instead of selecting the option "While next hop respond to ping" you have to select "while host respond to ping". This host can be on the other side as long as the IP of the host (192.168.0.1) it is included in the subnet (192.168.0.0/24 )that you want to reach.

[Bruce]

Good to hear that you use the SD-WAN method for sites where it’s applicable. With this site have you fully tested it and did it work?

I’m surprised that using a host in the destination subnet works as when the AutoVPN route takes over that host would become accessible again, and so you’d expect the route would become active and start black-holing traffic, then it would drop again, and you’d get a route flap.

If it is working, all I can assume is that the source used for the routing ‘ping’ can’t be routed into the VPN... that would be what is ‘solving the problem’ (although isn’t intuitive).

[iConnect]

We just tested it on a lab that we created for this purpose (2 MX's and 2 routers to simulate the MPLS) but the plan is to move to production this week.

What is important to have clear is :

1.- The MPLS has to be on the LAN of the MX.

2.- To have an SDWAN  tunnel up (not ordinary VPN's) using a WAN port. 

3.- To understand the purpose of each route of your table, the priority of each route and to be clear what happens when one of your routes overlap another one (the redundant one). If you are not clear about the purpose of each of your routes and how they behave in every scenario, then it could not work as it should.

[WillN]

1.Both WAN 1 and 2 must have internet access for this to work? In other words: WAN1 and 2 must be online first? What if my MPLS link does not have internet access? Each WAN uplink conducts a connection monitor (CMON) test to confirm the availability of the port. One of those tests is to resolve a DNS and to see if the Dashboard can be reached. Should this fail the port itself will fail. Any traffic will be pushed on the secondary link only and not utilise the primary link at all. Your options here are an MPLS cloud gateway for breakout, or some kind of hub site with a cascaded solution to allow for central internet breakout. Or you could put routers/switches in front and using pbr to push CMON traffic up to an internet-facing route. There are settings being tested for private networks, but no guarantees that VPNS would survive even with that.. for now. 2.If I enabled NONAT on WAN2, I can't see "Inbound Rules" enabled under Firewall options. But as per the guide I should do a deny any any. What's the expect option? Meraki MX will still behave as a Stateful Firewall even in NONAT configuration. So unsolicited inbound traffic will be blocked. You can contact Meraki to make the MX stateless, in which you gain control over the inbound FW rules (default to allow all so better be quick to apply deny all when it gets changed over) - I think this requires the MX to be running beta code though and the Meraki TAC advise against it. 3.To send traffic to WAN2, I need to apply PBR under Uplink selection -> Flow preferences right? Not to use static route? My idea is to force certain MPLS subnets/routes to go via WAN2/MPLS. Yes, Flow preferences or VPN SD-WAN policies to control traffic flow to uplinks. Allow the MX to do its job and manage your connectivity at this level rather than large static route policies. <- That's just my personal opinion though, matter of taste 😛 Other question: I can't terminate MPLS/EBGP on MX because EBGP is on BETA and only supported on concentrator mode right? Correct, concentrator only.. for now.

[RichardChen1]

Thanks guys. Meraki Support has enabled this feature for me.

I will start testing soon.