MX appliance shows latency spikes

[TheAlchemist]

Hello,

Have two ISP connections to an MX250 as WAN 1 and WAN 2 in a load balanced mode. Time to time latency spikes are noticed on both the ISPs at the same time. Checked with ISPs and both report stable, low latency internet connection. So, why would such latency spikes show up intermittently. Could a sudden burst of high uplink traffic lead to such latency spikes ? Uplink is mostly around 1.5 Gbps on a 4 Gbps internet connection.

Thanks,

[ww]

Are you running autovpn or just internet traffic.

The uplink page latency and loss are based on icmp packets. So most destinations drop icmp traffic or queue/buffer this traffic in case its busy.

You could add another destination to compare if you see the same loss/latency  at the same time. 

https://documentation.meraki.com/MX/Firewall_and_Traffic_Shaping/SD-WAN_and_Traffic_Shaping#Uplink_S...

In case you run aotuvpn then use this statistics https://documentation.meraki.com/MX/Monitoring_and_Reporting/SD-WAN_Monitoring

[TheAlchemist]

So, on Uplink Latency page of MX 250 if destination is a google DNS server i.e 8.8.8.8 and we see latency spike on both carriers, is it due ICMP packets unable to hit DNS server?

[TheAlchemist]

Active-Active Auto VPN is On and load balancing enabled. VPN traffic is for control of some devices but we see hits on all our video streams traffic both downstream and upstream via internet.

[Bumla]

---

@TheAlchemist wrote:

Active-Active Auto VPN is On and load balancing enabled. VPN traffic is for control of some devices but we see hits on all our video streams traffic both downstream and upstream via internet. camzap

---

It's not always web browsing traffic, but visitors streaming media to the site are mostly UDP-based and with each delay spike of around five ms or more, a hit is achieved.

[PhilipDAth]

Is this just web browsing traffic that is affected, and are you using content filtering?  If so, if may be affected by your choice of "URL category list size".  "Full List" defers the user's web browsing session till the MX completes looking up the content category.

[TheAlchemist]

@PhilipDAth it is not web browsing traffic but media streaming traffic based of UDP and with every latency spike of even around 5 ms or more, a hit is taken.

[cmr]

@TheAlchemist the MX250 has the following recommended capacity limits:

VPN 1Gb/s

Advanced features 2Gb/s

Enterprise (basic) features 4Gb/s

If you are running 1.5Gb/s of traffic including VPN and advanced features then the box might simply be overloaded.

[TheAlchemist]

@cmr We are not using Advanced features so have 4 Gb/s of throughput available for Uplink traffic. Also, isn't the VPN traffic also added up in the Uplink traffic ? I thought all traffic (including VPN) shows up as uplink traffic which for is is around 1.5 Gb/s. 

[cmr]

@TheAlchemist The VPN traffic uses the most resources, so an MX250 is designed for up to 1Gb/s of it.

As an approximate example, if you had 750Mb/s of VPN traffic you could have an additional 1Gb/s of basic traffic.  If you had 900Mb/s of VPN traffic, you could have an additional 400Mb/s of basic traffic. 

Im guessing you have the Enterprise license (like we do), as you mention you aren't using advanced features.

[TheAlchemist]

@cmr yes, we have enterprise license with 4 Gb/s throughput, so ok at that end. Also, most of our VPN traffic has been less than 10 mbps. So, for some reason, latency spikes are hard to understand.

[finzwake]

What firmware are you currently running?  Since upgrading to 16.X, we've seen weekly spikes of latency (1500-3000ms) on our MX250 stack.  Working with support we also saw CPU spiking in the same way.  A reboot or swapping primary/secondary was almost always required to level out the MX

[TheAlchemist]

@finzwake we are running on version 16 as well. But, have not done a reboot in a while. CPU utilization has been been below 50% even during high upstream traffic.

[ewcs]

I'm seeing this on multiple networks, each showing both ISPs with similar spikes.

I've narrowed this down to the IDS feature:

Setting "Intrusion detection and prevention" to disabled returns latencies to normal values.

I've opened a support case on two different networks, but the support response did not really assist.

[TheAlchemist]

@ewcs our setup does not have Advanced Security License so cannot see this Threat Protection>Intrusion detection and prevention but still get these latency spikes time to time on both ISPs at the same time. Definitely a Meraki issue rather than ISPs.

[ron415]

Did tac find the solution? I have the same problem

[TheAlchemist]

I think our MX appliance when running its algorithm for load balancing between WAN 1 and WAN 2 to decide which device uses which uplink connection is leading to a latency spike.

[TheAlchemist]

Actually, even after disabling load balancing and Active-Active Auto VPN , still MX-250s continue to show latency spikes on both ISPs at the same time for unknown reasons. No particular event is captured at the mentioned time which might point to this latency spike.