VMX Static Route - Next Hop IP?

[Andrew_Laeddis]

When deploying a VMX, the documentation states:

"Deploy a virtual appliance into a different subnet than the resources that route through the virtual appliance are deployed in. Deploying the virtual appliance to the same subnet, then applying a route table to the subnet that routes traffic through the virtual appliance, can result in routing loops, where traffic never leaves the subnet."

I have deployed the VMX in routed mode, in it's own subnet, and it was given the x.x.x.68 IP address in the x.x.x.64/28 subnet, and given a default gateway of x.x.x.65. I'm not sure what the .65 address actually is, or how that works.

I'm attempting to create static routes to my server resources in another subnet, and I set a static route to that subnet with the next hop of the default gateway IP address that the VMX was assigned. This isn't working however, and the route is not showing as active.

Am I missing something, or misunderstanding how the VMX should route to other subnets in Azure?

[tomas209ca]

I believe you will want to deploy the VMX in concentration mode. Routed mode is still not fully supported or still has issues.

[Ryan_Miles]

Clarification, routed mode is now the default mode for newly deployed vMXs (LINK). It is supported.

[PhilipDAth]

OMG.  That boggles my mind.  I'm struggling to conceive why what is a good idea.

[MyHomeNWLab]

The Limited NAT mode setting is special.
I am explaining the components step by step.

vMX has only one NIC, regardless of mode.
Communication via vMX to another segment is routed according to Underlay's default route. (Uplink's default route)
Therefore, Static Route settings are not required.

Also, from other MXs, a default route must be set up to the vMX. (Full Tunnel)

This is due to the special LAN Config settings in Limited NAT mode.

The LAN Config settings are specified as follows

[LAN Config (Limited NAT mode)]
* MX IP: Specify the IP address of the vMX Uplink. (x.x.x.68)
  MX IP is used by Source NAT.

* Subnet: Specifies the subnet to which the vMX belongs. (x.x.x.64/28)
  The specified subnet will be unreachable. I think it is equivalent to Null0.

  It is a required setting, so there is no choice but to set it.

  Therefore, specify the subnet to which the vMX belongs and make it a dedicated segment for vMX only.

* VPN mode: Enblaed or Disabled
  Either is acceptable.
  This setting is virtually meaningless because the vMX cannot communicate to the subnet to which it belongs.

Please note that DC-DC Failover redundancy is not available in Limited NAT mode.

[Andrew_Laeddis]

After some further investigation, and calls to Meraki support, this looks like it might be the correct way to configure this, however, I have got it configured the way I want it (not force tunnel), and it seems to be working. I'll explain:

-I don't want to setup full-tunnel and send all my traffic to Azure. This is just supposed to be an extension of the SD WAN. (It wasn't clear to me initially that the requirements were to set this up in full tunnel. I've only seen this in an FAQ section - seems it would have been important to mention this in the setup docs)

-I want to use routed mode so that the VMX is the firewall between Azure resources, and the internet.

-I have managed to set this up as split tunnel, by creating a static route to the subnets in Azure that I want my branch sites to reach. I've pointed this static route to the IP on the VMX's default (192.168.128.0) LAN subnet...I've noticed that the static route doesn't go active, so it's almost useless, *except* it allows me to advertise that route over the SD WAN. So now branch sites have that route, know to route to the VMX, and then the VMXs default routing behaviour knows how to get to the subnet (I'm not sure what wasn't working previously, as it was the same principle, though I wasn't using the default LAN subnet)

My concerns deploying the solution are:

-Since this is not explicitly the expected behaviour, then some patch in future may break the whole solution

-My understanding of how my workaround is working is not correct, and the solution breaks

Seems crazy that they've changed the default mode to routed mode, with minimal documentation, and with the expectation that everyone is going to full-tunnel all their branch traffic to the cloud?

[MyHomeNWLab]

I did not find sufficient information in the documentation, so I previously contacted support to confirm the specifications.
At the time I was also confused.

> -I have managed to set this up as split tunnel, by creating a static route to the subnets in Azure that I want my branch sites to reach.

I had never tried this method. It feels like a tricky workaround.

As a prerequisite, the instance must be restarted to reflect the vMX settings.
Static Route is installed statically from Meraki Cloud to other MXs, so it is reflected without restarting.
I am wondering if it will work after rebooting.

For your information, I will share a case study that I know of.

If "Subnet (e.g. 192.168.128.0/24)"" setting is configured when VPN mode is Enable, the route will be publicized to other MXs without rebooting.
This is because Auto VPN Route is installed statically from Meraki Cloud to other MXs.
Therefore, the communication may appear to be working without problems.
In other words, the Split Tunnel appears to be working.
But, after rebooting, the route to "Subnet (e.g. 192.168.128.0/24)" becomes equivalent to Null0 and communication is no longer possible.

Regardless of the mode of vMX, I don't think it is expected to communicate from the Azure subnet to the Internet via vMX.

It is not possible with One-Armed Concentrator, at least not because it is not source NATed.

Since communication from the Public Cloud side to the Branch side is not possible in Limited NAT mode, it is assumed that communication to the Internet side is also not possible.

It is similar to a typical NAT Router that does not allow WAN to LAN communication. (Unless you are setting up Static NAT, etc.)

In the case of your requirements, though, it is WAN to WAN (Uplink to Uplink).

Perhaps a dedicated Firewall product would be suitable.

Previously, changing vMX to Limtied NAT mode required contacting support. (Previously, The vMX "Addressing & VLANs" configuration menu did not appear.))
However, since the vMX mode setting screen now appears, I think it was matched with the default mode of MX. Whether it is appropriate for the customer or not.
That is my guess.

[Andrew_Laeddis]

I had restarted the branch MX to ensure that the tunnel was re-established and still working with the setup, which it was (unless you mean restart the VMX?)

My understanding of what was happening was the static route on the VMX does not go active, and is ignored in VMX routing lookups, and all VMX traffic (except traffic to  branch sites) is sent out the default route. I have now actually set the static route to be inactive if the next hop IP is unavailable, which it always will be, as the IP I've used for next hop is not assigned to anything. So the static route is always inactive, but always advertises the Azure subnets to branch sites.

Ultimately, the solution is not supported, and this is not the desired behaviour, so I can't deploy it.

It's just very strange that the default deployment for a VMX is routed mode; a mode which means that as part of my SD WAN solution, instead of just extending SD WAN to Azure, I now have to full-tunnel all of my branch traffic to Azure (for no good reason, and for it to be subject to Azure's egress charges???)

Regardless of the mode of vMX, I don't think it is expected to communicate from the Azure subnet to the Internet via vMX.

This might be the case, but who knows, as the documentation (that I've seen) doesn't explicitly say that it can't/shouldn't be doing this. I'm not sure why it couldn't act as a firewall between the internet and Azure - it is a firewall.

[MyHomeNWLab]

> I had restarted the branch MX to ensure that the tunnel was re-established and still working with the setup, which it was (unless you mean restart the VMX?)

It was meant to restart vMX (Virutal MX).

> My understanding of what was happening was the static route on the VMX does not go active, and is ignored in VMX routing lookups, and all VMX traffic (except traffic to branch sites) is sent out the default route.

Yes, if the route is inactive with Static Route Tracking, I think that would be the behavior.

Just FYI, topologies that accommodate MPLS on LAN ports take advantage of this behavior.

Integrating an MPLS Connection on the MX LAN - Cisco Meraki
https://documentation.meraki.com/MX/Networks_and_Routing/Integrating_an_MPLS_Connection_on_the_MX_LA...

> So the static route is always inactive, but always advertises the Azure subnets to branch sites.

This is expected behavior.
Even if Static Route Tracking is Inactive, routes are ALWAYS advertised to the Auto VPN Peer.

Within the SD-WAN Fabric, Routed Mode cannot have duplicate routes.
This eliminates the need to track status changes and reroute routes.
I have previously contacted support about this behavior.

If you are forced to use Full Tunnel & Limited NAT mode, there is a Local Internet Breakout feature as a mitigation measure.
It is only a mitigation measure...

VPN Full-Tunnel Exclusion (Application and IP/URL Based Local Internet Breakout) - Cisco Meraki
https://documentation.meraki.com/MX/Site-to-site_VPN/VPN_Full-Tunnel_Exclusion_(Application_and_IP%2...

[Fady]

Hey Andrew, 

There are 2 solutions here

• Static Solution: 
  • Have the vMX in VPN Concentrator and you can either deploy it in the same VNET as your server or separately.
  • If you deploy vMX in separate VNET then make sure to have a route table.
  • The vMX default GW will be Azure wired IP and that is enforced by Azure. That default GW will be mapped to the route table that you will attach the VNET subnet to.
  • Make sure to add the static routes inside Azure route table in both directions (towards vMX for remote SDWAN branches and towards other Azure subnets/VNETs)
  • Add local subnet in your Meraki vMX for all your azure subnets to inject those routes into the site to site VPN.
• Dynamic Solution:
  • Build Azure Route server to exchange BGP routes with vMX.
  • In this solution, you wouldn't need to add local subnets in your vMX
  • You still need to have route table attached to your vMX as Meraki resources are locked and Azure BGP routes won't be injected into the vMX NIC card.

Let me know if you need more info and I will be happy to help you out.

[Andrew_Laeddis]

Thanks Fady,

I've had to revert to the VPN concentrator mode, and I'm trying the static solution, though the Azure resources cannot ping the branch sites, and vice-versa:

-The Azure Server subnet is correctly advertised to the branch site. 

-Pings from the branch site can be seen on packet capture to cross the VPN, and leave the VMX, but do not come back to the VMX

-The VMX can ping the servers, and the servers can ping the VMX

-There is a route table attached to the server subnet which includes the VMX subnet and the branch subnet

-There is an entry in the server subnet's NSG allowing traffic from the branch site

-Essentially the same settings as when the routing worked when the VMX was in routed mode

You've mentioned that there needs to be a route table attached to the VMX subnet to point to the server subnet if they are in separate VNETS - they are in separate VNETS, but there is VNET peering, and the VMX can ping the servers without a route table.

I've added a route table, but this doesn't make a difference - Can you confirm what the next hop should be in the route table?

Thanks, 

[Fady]

Hi Andrew

It seems you followed the correct steps, the next hop of the route in Azure should be 

- The next hop type is "Virtual Appliance"

- Next hop address is "the ip of your Meraki vMX"

I had some use cases that needed to forward all traffic from the branch to Azure and NAT out from there. I used Cisco virtual routers on top of vMX to perform NAT, you can follow the video and consider the LAN side of the Cisco router as your server farm.
https://www.youtube.com/watch?v=MljINqgmDkM

Let me know if you still stuck.

[IntrovertEngg]

Hello,

Did you make it work ? If yes, would you be able to share some pointers? Currently I am stuck in same  situation.