Mx64 Latency increase

[SlothSlayer]

About a month ago one of our many Mx64 Meraki appliances had a jump in latency from 24 ms to 30-40 ms on average.  I contacted the ISP that covers that site and area and they advised that they have no latency up to their gear.  I ran a trace and it appears everything is 1ms or less up until their gear which it doesn't hit and then the Mx64 which gets a 34+ms final hop.  Anyone know why this may have happened, if their is a bug on 14.40 version, what I should be checking to fix or improve this, or what I should be checking on? I have rebooted our gear, ISP rebooted his gear. The Template we are using is pretty basic, we don't traffic shape, and everything for the most part is allowed. Any advice or helpful troubleshooting thoughts are appreciated. 

Thanks,

[kYutobi]

14.40

Bug fixes

• Corrected an issue that could result in a device reboot when there were two or more port forwarding rules were configured for the same port.
• Resolved an issue that resulted in files failing to be submitted to ThreatGrid.
• Various fixes and security improvements

Known issues

• Please note that until certification has been obtained, the Z3C will not be supported on Verizon's network.
• World-wide device SKUs of the MX67C, MX68CW, and Z3C units cannot be deployed in North America and North America device SKUs of the MX67C, MX68CW, and Z3C units cannot be deployed outside of North America.
• When deployed in warm spare / high availability (HA), MX67C and MX68CW do not support using their cellular connectivity to pass client traffic. In this deployment, the cellular connectivity can only be used for device monitoring or network troubleshooting. This is an expected limitation for these platforms.
• When MX67(C,W) and MX68(W,CW) units are deployed in warm spare / high availability (HA), rebooting the spare appliance may cause a disruption of client connectivity for 10 or more seconds.
• After making some configuration changes on MX67(C,W) and MX68(W,CW) appliances, a period of packet loss may occur for 10 or more seconds.
• For a brief period of time upon boot, MX67(C,W) and MX68(W,CW) platforms can become bridged. This increases the likelihood of network loops forming in topologies with multiple inter-connected network devices for this brief period of time.
• MX67C, MX68CW, and Z3C units must be connected to the Meraki Dashboard initially to retrieve an update to allow for proper use of the integrated cellular connectivity. This is most likely to be an issue when bringing the units online for the very first time.
• On the MX67(C,W) and MX68(W,CW) platforms, when the MX is providing PoE to a connected device, this information will not be reflected on the Meraki Dashboard.
• Once a Z3 has been updated to this firmware version it can only run MX 14.31 or MX15.8 and higher. This is an expected result of updates to the device booting mechanisms and this limitation will not be resolved in future releases.
• After making some configuration changes on MX84 appliances, a brief period of packet loss may occur. This will affect all MX84 appliances on all MX firmware versions.
• Some stability-impacting issues present in MX 14.19 that affect a small population of MX250 and MX450 devices still exist.
• Due to issues still under investigation, MX67(C,W) and MX68(W,CW) appliances may become inoperable after a device reboot occurs

Other

• Multiple changes have been made to improve the scalability and reliability of downloading firmware during device upgrades.

[cmr]

Have you tried a different device, even a PC to test with as perhaps the tail circuit has changed or has an issue?

[SlothSlayer]

Yes, PC are still the same as well. Site hasn't noticed difference.  I may contact the ISP again, they said the circuit is seeing no different times or setting since the issue started.

[cmr]

Sorry, I meant a (suitably protected) PC or other device directly on the connection instead of the MX.  If the circuit is copper then perhaps they have changed the encoding or similar.  Does the ISP own the tail or is it a reseller, or is the tail yours?

[PhilipDAth]

The reported latency is to a defined IP address, typically 8.8.8.8.  You may just find your ISP no longer has such an optimal route to Google's DNS server.

What latency do you get when you ping 8.8.8.8?

[SlothSlayer]

When the Mx64 pings against 8.8.8.8 it averages 50ms times.

[cmr]

Please try 1.1.1.1 and 8.8.4.4 just to see if it is simply an issue with the default 8.8.8.8

[PhilipDAth]

https://documentation.meraki.com/MX/Firewall_and_Traffic_Shaping/SD-WAN_and_Traffic_Shaping 

[SlothSlayer]

We don't do anything shaping. We use meraki on our smaller clinics so it is set to default.

[gauravgupta]

Hi @SlothSlayer ,

I would do the following to isolate the issue and know my troubleshooting target:

1. Connect a Linux machine directly into one of the LAN ports of the MX64.

2. Install hping3 utility on Linux machine so I am able to mark packets in a unique way so that they can be filtered and identified.

3. Set up two simultaneous packet captures on the Dashboard - One at Internet interface and other at LAN interface of the MX with output as download .pacp file for wireshark.

4. Start both the pcaps, and quickly run command: "hping3 -K 8 1.1.1.1 -c 30 -o 28" on Linux machine's terminal.

5. Once done, open both pcaps and compare the delta of ICMP request and response calculated from Internet interface pcap with delta calculated from LAN interface pcap.

6. If value of delta is same, MX is not at fault. If there is a noticeable difference, MX is adding latency. And If there is no difference in delta but you are seeing high delta (RTT) on internet interface of the MX, ISP is at fault.

You can use icmp identifier filter (Expand ICMP header from lower half of Wireshark screen, find and copy identifier value, and use filter "icmp.ident == ident_vlaue") on both pcaps and compare DSF value (which should be 28) to confirm that you are looking at right sequence of packets. 

BR
Gaurav Gupta

[gauravgupta]

If you do not have a Linux machine handy, you could simply run command "ping 1.1.1.1 -n 30" on a cmd or "ping 1.1.1.1 -c 30" on a terminal and follow the same steps mentioned above, but make sure the laptop is plugged directly into LAN port of the MX. You could still use filter "icmp.ident == ident_value" to display interesting packets only.