About a month ago one of our many Mx64 Meraki appliances had a jump in latency from 24 ms to 30-40 ms on average. I contacted the ISP that covers that site and area and they advised that they have no latency up to their gear. I ran a trace and it appears everything is 1ms or less up until their gear which it doesn't hit and then the Mx64 which gets a 34+ms final hop. Anyone know why this may have happened, if their is a bug on 14.40 version, what I should be checking to fix or improve this, or what I should be checking on? I have rebooted our gear, ISP rebooted his gear. The Template we are using is pretty basic, we don't traffic shape, and everything for the most part is allowed. Any advice or helpful troubleshooting thoughts are appreciated.
Thanks,
Have you tried a different device, even a PC to test with as perhaps the tail circuit has changed or has an issue?
Yes, PC are still the same as well. Site hasn't noticed difference. I may contact the ISP again, they said the circuit is seeing no different times or setting since the issue started.
Sorry, I meant a (suitably protected) PC or other device directly on the connection instead of the MX. If the circuit is copper then perhaps they have changed the encoding or similar. Does the ISP own the tail or is it a reseller, or is the tail yours?
The reported latency is to a defined IP address, typically 8.8.8.8. You may just find your ISP no longer has such an optimal route to Google's DNS server.
What latency do you get when you ping 8.8.8.8?
When the Mx64 pings against 8.8.8.8 it averages 50ms times.
Please try 1.1.1.1 and 8.8.4.4 just to see if it is simply an issue with the default 8.8.8.8
We don't do anything shaping. We use meraki on our smaller clinics so it is set to default.
Hi @SlothSlayer ,
I would do the following to isolate the issue and know my troubleshooting target:
1. Connect a Linux machine directly into one of the LAN ports of the MX64.
2. Install hping3 utility on Linux machine so I am able to mark packets in a unique way so that they can be filtered and identified.
3. Set up two simultaneous packet captures on the Dashboard - One at Internet interface and other at LAN interface of the MX with output as download .pacp file for wireshark.
4. Start both the pcaps, and quickly run command: "hping3 -K 8 1.1.1.1 -c 30 -o 28" on Linux machine's terminal.
5. Once done, open both pcaps and compare the delta of ICMP request and response calculated from Internet interface pcap with delta calculated from LAN interface pcap.
6. If value of delta is same, MX is not at fault. If there is a noticeable difference, MX is adding latency. And If there is no difference in delta but you are seeing high delta (RTT) on internet interface of the MX, ISP is at fault.
You can use icmp identifier filter (Expand ICMP header from lower half of Wireshark screen, find and copy identifier value, and use filter "icmp.ident == ident_vlaue") on both pcaps and compare DSF value (which should be 28) to confirm that you are looking at right sequence of packets.
BR
Gaurav Gupta