Azure vMX packet loss and dup MAC for src/dst in packets. Plus VPN drops

[Dennis_S]

I’ve seen similar messages like this minus the dup MAC in the packets.

We have packet loss on the uplink of our vMX which in turn creates VPN tunnel drops. The loss is regularly ranging from 0.3%-5% with spikes up to 100% but mostly around 70%.We are also seeing high session counts. We started with a medium size vMX then added a second to segment our sites to combat the session count issue. This did not correct the issue, we then added a large model, loss and vpn drops still happening.

Opened a TAC case and after a few techs we found packets with duplicate macs in the src/dst fields. The recommendation is to create an NSG to filter the MAC issue, we have not found a way to do this.

Below is our topology. (Please disregard the vMX01-M)

I’ve taken some of the actions I found in other threads to have the VPN registration checked and the MTU size for AnyConnect. Waiting the response form TAC on that.

One trigger we have found is our vulnerability scanner that hits all subnets for our remote sites. The upside to this is that it is scheduled so we were able to isolate that part of the problem or at least know a cause of a large portion of the traffic. But we still have the issue on a smaller scale throughout the day.

My next actions are to re-work the Azure route tables to make them more specific, i.e. remove the RFC1918 routes and put in routes specific to our sites.

Any other thoughts? All help is greatly appreciated.

-Dennis

[VivekT]

Azure NSGs (Network Security Groups) primarily work at Layer 3 and Layer 4, so they can't filter based on MAC addresses directly. If the duplicate MAC issue persists

Use vSwitch Policies (if applicable): Depending on your Azure environment, you may be able to apply port-level security or forwarding rules on the virtual network interface level to address such anomalies.

• • Check for Source: Identify where the duplicate MACs originate. This could be from misconfigured or looped virtual NICs in your Azure environment. Tools like Wireshark can help pinpoint the source.
• Update Azure VM Configuration: Ensure the vMX instances are configured with unique MAC addresses. You might need to regenerate or explicitly set MAC addresses in your Azure VM network configurations.

One trigger we have found is our vulnerability scanner that hits all subnets for our remote sites ::: I would say limit the speed between scanner IP address and remote subnet .

[PhilipDAth]

You have to put the VMXs into their own dedicated VNET.  If you put them in the same VNET as servers you get packet loss.

https://documentation.meraki.com/MX/MX_Installation_Guides/vMX_Setup_Guide_for_Microsoft_Azure

"Do not associate the vMX SD-WAN Subnet to the same route table that is associated with the resources subnet, this is known to cause packet loss."

"Deploy a virtual appliance into a different subnet than the resources that route through the virtual appliance are deployed in. Deploying the virtual appliance to the same subnet, then applying a route table to the subnet that routes traffic through the virtual appliance, can result in routing loops, where traffic never leaves the subnet.:

[Dennis_S]

@PhilipDAth 

The route tables are different but yes the subnets are part of the same vnet as our Azure HUB environment. The resources being accessed are all in different vnets. 

Per the doc and your recommendation I need to have a new vnet built and place the vMX in there own subnet also. Just asking so I can be certain.

Thanks for the assistance.

-Dennis

[PhilipDAth · Accepted solution]

>Per the doc and your recommendation I need to have a new vnet built and place the vMX in there own subnet also.

That is correct.

[Dennis_S]

Now to convince my team of this.

Thanks.

[PhilipDAth]

I did include a link to the official documentation, and quoted the text from it that says you will get packet loss unless you deploy using a seperate VNET ... 🙂