MX to MX Site-to-Site VPN Dropping Packets

[CMTech1]

I have a strange issue I'm unable to locate and spent a couple hours with Meraki support only to ask we reboot the MX's, which we did with no resolution. Even changed S2S VPN from Hub to Spoke to Off and then back on with no resolve and finding it hard to track down actual root cause.

Have (8) sites all with Meraki MX's and all are Hub to create a mesh network. Has been working this way for months without issue, however yesterday between two of the sites we have a 20-40% packet loss. It is only between these two sites that have been working perfectly since January and one site has MX100 while other has MX84 not that it should matter since same under the hood.

What's interesting is both these sites can ping anything anywhere, WAN, LAN or any other S2S MX's without an issue. However, these two sites in question have issues pinging each other and we're having difficulty operating due to the issues since these two particular sites share many resources. The issue is present in both directions.

Anyone experience similar?  

[Nash]

What does a traceroute look like from Site A's public IP to Site B's public IP, and vice versa? Is there a hop in the route that goes slow?

[CMTech1]

WAN from either side is solid as well as to any other WAN site. This is isolated issue to just the S2S VPN. Firewall rules on VPN is open and enabled in both directions. Just all the sudden started dropping packets when we came in on Monday and nothing nor any changes were done over weekend.

[Stealth_Network · Accepted solution]

I have had a similar issue where the traffic actually was changed on the path between sites. One of the many ISP's had an issue on a US to CDN handoff. I found MTR was really helpful in troubleshooting showing where traffic was dropping. We ended up using another tunnel as the exit point to bypass the poor path until it got fixed.

Good luck

[CMTech1]

That could very well be an ISP issue as you mentioned and where I'm going next with this if Meraki support unable to help further. I did have a similar issue with ISP where CDN was blocking port 500/4500 though entire VPN was down unlike this situation where it sorta works I guess 🙂

[PhilipDAth]

>between two of the sites we have a 20-40% packet loss

What is reporting this packet loss, or how are you seeing this packet loss?

[CMTech1]

Solarwinds and various other manual tests via the firewall native tools. 

[CMTech1]

Seems to have been CDN issues as noted. ISP found root cause and have escalated for repair to international vendor edge/border Route. 

Thanks!

[Stealth_Network]

Awesome!

Glad I could help, this was a super hard problem to troubleshoot (for me), so I am glad the ISP took responsibility (they usually don’t) 

😄

[BrechtSchamp]

What do you see in the Security & SD-WAN > VPN Status page of the two problematic appliances? Everything green regarding VPN registry, NAT and encryption?

[CMTech1]

Correct, everything is green, s2s vpn connected correctly and can pass traffic between the two sites, just keep freezing up when multiple packets dropped. Firewall VPN rules are Open/Open and Enabled as they have always been. Allowed VPN VLAN's are still the same ones permitted to pass and technically are passing, just dropping major packets.