Packets don't lie ...! or do they ?

RaphaelL
Kind of a big deal
Kind of a big deal

Packets don't lie ...! or do they ?

Hi ,

 

Just encountered once again a very special behavior on the MX plateform ( running 19.1.5 ).

 

Testing scenario : I'm capturing IPSec ( VPN tunnel from my computer to a VPN gateway ) trafic on my PC and on my MX which performs L3 routing for my LAN. My goal is to capture all ESP packets and compare the ESP Sequence to quantify the number of ESP packets inbound that are out of order. To my biggest surprise the 2 traces are way different. 

 

RaphaelL_0-1732280315032.pngRaphaelL_1-1732280349582.png

 

As you can see the exact same packet with the same IP ID and the same ESP sequence appears out of order only on the pcap taken on a MX. 

 

Conclusion :  The packet capture engine on the MX is way to unpredictable. Packets are routed and forwarded in the correct order , but they will be scrambled in the captures making troubleshoot harder. 

 

 

Yes. I had a case opened in the past. Yes they said this is expected.

 

Reason #34591 that I dislike MXs.

11 Replies 11
Frank-NL
Getting noticed

Thanks for sharing, how is this expected, did they give more info?

RaphaelL
Kind of a big deal
Kind of a big deal

I can't quite find the exact answer I got months ago. It was something that the packet capture engine is not prioritized by the CPU. So It can miss packets or show them in the incorrect order.

AlexP
Meraki Employee
Meraki Employee

If you can recall a case number, would you mind DM'ing it to me? I'd like to see the verbiage/conclusions you were provided with.

rhbirkelund
Kind of a big deal
Kind of a big deal

Is that MX behind a Firepower by any chance?

LinkedIn ::: https://blog.rhbirkelund.dk/

Like what you see? - Give a Kudo ## Did it answer your question? - Mark it as a Solution 🙂

All code examples are provided as is. Responsibility for Code execution lies solely your own.
RaphaelL
Kind of a big deal
Kind of a big deal

No. It is my MX at home. Directly plugged into my ONT.

 

My PC is a client behind the MX. My PC is bulding a VPN tunnel to a VPN appliance somewhere else. The MX is displaying the packets in the wrong sequence but are received in the right sequence.

GIdenJoe
Kind of a big deal
Kind of a big deal

There are situations where an MX does have issues with pcaps.
However I have seen MS switches also do strange stuff.

I once had a VoIP complaint and went to investigate locally.
I did a traffic mirror on an MS switch (MS120) and realized every 10 voice packets it lost one.
However when I captures via dashboard the packets were all there.

RaphaelL
Kind of a big deal
Kind of a big deal

Yep pretty sure I also have experienced this behavior on a small MS.

RaphaelL
Kind of a big deal
Kind of a big deal

Re-tested this behavior this morning and with MS 17.1.3 I still do have this problem of random packets dropping every x packets on a span port.

Brash
Kind of a big deal
Kind of a big deal

It's an interesting one.


Inline packet captures across the board seem to be more possible than ever now that ASIC's and CPU's are getting much faster and buffers can be deeper. It'll always be de-prioritized to ensure packets are routed/switched if the CPU is too busy to make a copy, but it seems that most devices these days are fairly capable to capture a high number of packets across the device.

 

That said, I've had issues with inconsistencies with captures from the dashboard.

 

I'm currently investigating an issue with small UDP packets being dropped.

The packet capture on the MS misses a decent number of packets that I know with certainty went across the port I'm capturing on. And we're nowhere near pushing line rate across the switch.

I'm hoping a SPAN (Port Mirror) will be more reliable.

RaphaelL
Kind of a big deal
Kind of a big deal

I just tested a Span port on a MS120 , it is much more reliable , but I'm still dropping packets out of the blue. 

This is a bit annoying when you want to do a proper troubleshooting.

jimmyt234
Building a reputation

I have always found the packet capture a bit flakey, sometimes just not showing any packets at all and then you refresh/come back 5 minutes later and it works!

Get notified when there are additional replies to this discussion.