MX 64 packet loss

[Dreamwvr]

WAN 2 on our MX64 had high packet loss.  The modem was swapped and the packet loss went down.  It is now intermittent and always at 0.28%.  Our customer does not want any loss, no matter how small.  With it being consistently at 0.28%, could this be some kind of bug?   I don't know if you could troubleshoot packet loss so small.

[MerakiDave · Accepted solution]

That's not uncommon and typically represents the occasional missed ping response back from google public DNS, the MX will check health stats and run continuous pings every second to 8.8.8.8.  I see the same in my own home lab MX64, see attached image.  They could prove this out by running a packet capture on an upstream device and filter by host IP and icmp protocol to confirm there might be an occasional echo-reply packet lost.  They can also go into the Security Appliance > Traffic Shaping page and add another destination IP address besides the default 8.8.8.8 and perhaps something closer upstream, maybe their ISP's DNS server, to get better results showing zero or close to zero loss.  Then, on the Uplinks tab they can select between multiple destinations for connectivity status.

[RainMan]

I entirely agree that a little bit of packet loss to a Google DNS server is normal.  this has become more frequent now that Google itself is an internet provider in many markets with their Fiber product.  This can become an issue though as with some providers like CenturyLink,for a few sites that is a constant 5-8% which has become worse with the loss of  net neutrality.

By default, the dashboard will report packet loss to 8.8.8.8 on the Security Appliance, - Uplink page.  You can add another address to ping instead such as one of your head end servers or even your provider's DNS server which will give a more accurate representation of the quality of the connection.  From Dashboard, Select Security Appliance - Traffic Shaping.  Under Uplink Statistics, you can see it will ping 8.8.8.8 by default, but you can add your own destination as well.  Options here would be your head end server if you are a large organization, your local provider's DNS server (whatsmydns.net), a hop on a trace route that is still on your provider network, or maybe even your public DHCP gateway address (not 192.168.x.x).

Be careful here if you are on a metered connection i.e. 4G as this will double your ping traffic on all of your internet connections on the Security Appliance.  For instance if your backup connection is through a cellular modem or the built in cellular connection on the MX, you will now be pinging 8.8.8.8 in addition to the address you have specified.  The standard ping to 8.8.8.8 is a 64 byte ping plus overhead (header, etc.) once per second and there should be a reply as well.  based on what we have seen this adds up to about 450 MB per month, per ping endpoint. This is in Bytes and there are eight bits per byte, so check with your provider on if you are paying for Megabytes/Gigabytes per month, or Megabits/Gigabits per month.

Would be neat if this could be changed instead of adding an additional ping, but for many this is quite helpful.

[BHC_RESORTS]

---

@Dreamwvr wrote:

WAN 2 on our MX64 had high packet loss.  The modem was swapped and the packet loss went down.  It is now intermittent and always at 0.28%.  Our customer does not want any loss, no matter how small.  With it being consistently at 0.28%, could this be some kind of bug?   I don't know if you could troubleshoot packet loss so small.

---

If they don't want packet loss, a coaxial internet connection (or *DSL) probably isn't going to help. Even business class is almost always sold as best-effort. Less than 1% is perfectly acceptable, and well within design. I'd explain to your client what packet loss actually means and why less than 1% is not going to effect whatever they are doing. Unless they are doing FOREX or something else needing 0ms responses, and which case, they shouldn't be using a MX64 anyways.

[PhilipDAth]

0.28% packet loss is completely acceptable.  Note that such a small amount may be measurement error - rather than actual packet loss.

For example, when the ARP entry for your default gateway expires, the next packet (such as your ping measurement packet) gets blocked until an ARP request is sent, and a response is received.  Typically ARP is not a high priority process on most networking devices.  If the process takes too long the delay may be seen as packet loss, when it is really just packet delay (aka measurement error).

[Chaitanya]

Hello,

I have MX400  WAN1 is connected to cogent 500mbps internet recently. Every 4 hours 30 min we see 21.7% packet loss constantly. ISP tested several times and says nothing wrong from their end. But meraki send alerts that uplink lost and all site to site vpn is disconnected.   Any recommendation?

Cogent==>TWC (RAD) converter ==>DMZ Switch==>MX400  (swtich port is full duplex and Speed is 1000)

[Mr_IT_Guy]

We actually had a similar situation a few months back. What we did was put each hop from our router to Google DNS until we found what was causing our packet loss issue. Once we saw which hop along the way was causing the loss, we contacted them and they were able to resolve the issue. Keep in mind that while 0.28% is reported, this number is an average of a time period. When we were able to drill down to the 2 hour view vs last day or week, I was able to see packet loss of 5-9% during the same time period. 

In my mind, the most important packet loss you should be concerned with is that from your modem to your modem's gateway. If you are experiencing loss at that point, you need to work with your ISP to resolve that link. Once it goes past that, you are at the mercy of the Internet.

[Chaitanya]

Hello, I have MX400  WAN1 is connected to cogent 500mbps internet recently. Every 4 hours 30 min we see 21.7% packet loss constantly. ISP tested several times and says nothing wrong from their end. But meraki send alerts that uplink lost and all site to site vpn is disconnected.   Any recommendation? Cogent==>TWC (RAD) converter ==>DMZ Switch==>MX400  (switch port is full duplex and Speed is 1000)

[phanlawi]

Hello.. I would like to know how to deep dive the packet loss in order to find out the exact issue causing the loss. As Chaitanya mentioned, ISP was saying that nothing wrong we as client keeps getting packet loss.