Meraki MX route preference

[Chema-Spain]

Hi, I have a topology with two on-prem sites comprising MXs as spokes of an Azure vMX hub. As hub is not redundant, customer has asked us to deploy non-meraki tunnels against Az VPN GW in the region as AutoVPN tunnels backup. First we had no dynamic routing in Azure but vMX publishing Az vnet address spaces as AutoVPN local networks. Provided I had /24s as real azure subnets, I configured two /25s per /24 as local network. Non meraki tunnels for onprem MXs had real /24s in Azure as remote prefixes. 

Fine... however it did not work. Appart from having udr routes pointing to vMX that had to be deleted, when stopping the vMX BU did not work due to the fact Meraki keeps publishing local prefixes even when vm is down 😞

OK, we deployed a RS instance in Azure, deleted VM UDRs pointing to vMX and ensured VM effective routes now learnt both onprem prefixes from vMX and summarized onprem prefixes from AZ VPN GW. Great!

We checked all was working fine in normal situation and also when stopping the vMX VM from Azure. However, when looking at the onprem MX Route Tables I found different behaviour from both MX. Both running same firmware 18.211.5.2. One of them is showing the green ball for Non meraki tunnel and grey ball for AutoVPN BGP learnt prefix, the other MX is showing grey ball for AutoVPN BGP learnt prefix and a hyphen for non-meraki tunnel. 

Looking at route preference, the order should be:

1. Directly Connected
2. Client VPN
3. Static Routes
4. Auto VPN Routes
5. Non-Meraki VPN Peers
6. BGP learned Routes
7. NAT*

It is supposed BGP learned it means from a direct ebgp peer, right? Provided you are an spoke receiving BGP prefixes over AutoVPN, it is considered as AutoVPN route, right?

However, capturing at onprem MX side, I only see traffic towards Azure when selecting IPSec VPN as the interface. Return traffic is taking AutoVPN path. This way I have asimmetrical routing. And it works... what I'm not sure is it would also work when next week we would change AutoVPN SDWAN fw from allow default to deny default, provided syn packets from onprem are not seen by vMX and syn + ack responses from Azure are going to reach the sdwan fw as non existing session and no allow default would be there to catch it. Am I right?

Thanks!

[PhilipDAth]

I don't know the answer.

I have only done this using dual vMXs.  I would personally use that configuration.

https://documentation.meraki.com/MX/Deployment_Guides/vMX_and_Azure_Route_Server

[Chema-Spain]

Hi,

Thanks for your reply.

Same customer already have dual vMX is another Azure Region. No problems there. We've tried proposing that solution but problem is they only have two onprem sites in the region so they don't consider it due cost reasons.

At the end of the day, I guess I can do nothing to manipulate azure prefixes to ensure AutoVPN tunnels as preferred. No way to summarize or prepending at VPN GW, no way to publish more specific routes from vMX as they are learnt dynamically from the RS. 

I will open a ticket to ask Meraki what they mean with BGP routes anyway because from my understanding bgp originated prefixes in Azure hub should be considered as AutoVPN routes at the spoke side. Hence spoke MX should IMO choose AutoVPN path and not IPSec tunnel.

As customer wants us to change SDWAN corporate FW from default allow to default deny I'm afraid a rule like source "Az Region prefixes" to "onprem sites in Region prefixes" would be a must.

Regards,

[Chema-Spain]

Meraki TAC confirmed BGP-originated AutoVPN prefix should be preferred. Just sent them the packet captures that prove it is not the case. Hope they could find and explanation for this behaviour.

[Chema-Spain]

Hi, TAC has answered me. They don't see the issue as a bug but an expected result. Surprisingly, they state the bgp-learnt prefixes at the hub from the AZ RS are considered BGP routes at the spoke sites even when they learn them over AutoVPN tunnels. Great! It makes no sense, they are AutoVPN learnt routes at spoke sites, it is supposed they are not running BGP. I'm aware they do behind the scenes, once you enable BGP in one hub, an ibgp session is eestablished with all other hubs and its spokes. 

They also said we have two ways to solve the issue. One is forgetting dynamic routing at Azure side and configure local routes. No way, I could split every azure /24 real prefix into 2x /25s. Fine. The problem is these prefixes are not removed from the network even when the vMX is stopped or fails. Another great! 

The other option is a recent deployment in 19.X. Now you can run bgp inside Non-Meraki IPSec tunnels. Fine, AZ VPN GW also supports it. Good advance from Meraki. I will test it. However, according to the meraki document, only AS-path prepend on egress and weight attribute are sopported. Why not Lpref???? The former does not solve the on-prem MX route decision process issue, it works for the other traffic direction. Weight... maybe. Meraki does not say what value is the default between 0-49. They only say the higher the value, the higher the preference. In case default value is not 0, I could try setting 0 por IPSec tunnels (from the dashboard I can't configure any weight por autovpn prefixes (local or bgp originated). This way it could work... or not.

Assuming they consider bgp-originated AutoVPN routes as bgp routes, I hope they would consider bgp-originated IPSec  routes ass BGP and not IPSec to maintain its rare route decision algorithm consistency. This way it could compare weights from both paths. Or maybe, provided even weight does not work and it is the same for both paths, MX decides to chose autovpn instead of ipsec for Az same mask prefixes. In AZ it is not possible to manipulate prefix at egress nor in RS nor AZ VPN GW and as-path would be the same for both paths as RS and VPN GW must be running ibgp themselves. 

In case all this does not work, in order to ensure an automatic BU for AutoVPN tunnel with an IPSec tunnel I guess it could only be done by Meraki behind the scenes.

If not, I'm afraid two vMX or manual backup with someone accesing the dashboard in case vMX fails.

I can understand mine could be corner case. However, route decision is not properly designed. And local prefixes permanently present in the network regardless of its state is even worse. 

Regards.

[Chema-Spain · Accepted solution]

Issue solved! Provided I realized it was imposible to make IPSec tunnel the automatic autovpn BU with BGP inside, I went back to static routing.

BGP inside IPSec tunnel does not solve our issue due to the fact Azure does not allow outgoing bgp attribute manipulation and Meraki does not allow incoming bgp attribute manipulation. As AutoVPN bgp peering at the spokes is ibgp and IPSec Tunnel requires ebgp, MX will always select IPSec. 

I redeployed the IPSec tunnels just configuring a /16 as remote prefix at both sides. Doing so, we reproduced the issue we had when we tested it: It worked fine in normal situation and it failed when stopping the vMX in Azure. Capturing at one branch MX, I saw packets exiting the IPSec Tunnel with no return. However, that flows did not reach Azure. Route table pane also got hung as last time. Impossible to check it even waiting. Running a traceroute from a branch PC to a vm in the AZ region, it was sent to LBO instead of the IPSsec tunnel. I decided to upgrade the MX firmware from 18.211.5.2 to candidate stable 19.1.8. After the upgrade, automatic BU when stopping vMX worked fine. We tried successfully twice to stop/start vMX. We also did all these for the other branch and also worked fine.