Packets don't lie ...! or do they ?

[RaphaelL]

Hi ,

Just encountered once again a very special behavior on the MX plateform ( running 19.1.5 ).

Testing scenario : I'm capturing IPSec ( VPN tunnel from my computer to a VPN gateway ) trafic on my PC and on my MX which performs L3 routing for my LAN. My goal is to capture all ESP packets and compare the ESP Sequence to quantify the number of ESP packets inbound that are out of order. To my biggest surprise the 2 traces are way different. 

As you can see the exact same packet with the same IP ID and the same ESP sequence appears out of order only on the pcap taken on a MX. 

Conclusion :  The packet capture engine on the MX is way to unpredictable. Packets are routed and forwarded in the correct order , but they will be scrambled in the captures making troubleshoot harder. 

Yes. I had a case opened in the past. Yes they said this is expected.

Reason #34591 that I dislike MXs.

[Frank-NL]

Thanks for sharing, how is this expected, did they give more info?

[RaphaelL]

I can't quite find the exact answer I got months ago. It was something that the packet capture engine is not prioritized by the CPU. So It can miss packets or show them in the incorrect order.

[rhbirkelund]

Is that MX behind a Firepower by any chance?

[RaphaelL]

No. It is my MX at home. Directly plugged into my ONT.

My PC is a client behind the MX. My PC is bulding a VPN tunnel to a VPN appliance somewhere else. The MX is displaying the packets in the wrong sequence but are received in the right sequence.

[GIdenJoe]

There are situations where an MX does have issues with pcaps.
However I have seen MS switches also do strange stuff.

I once had a VoIP complaint and went to investigate locally.
I did a traffic mirror on an MS switch (MS120) and realized every 10 voice packets it lost one.
However when I captures via dashboard the packets were all there.

[RaphaelL]

Yep pretty sure I also have experienced this behavior on a small MS.

[RaphaelL]

Re-tested this behavior this morning and with MS 17.1.3 I still do have this problem of random packets dropping every x packets on a span port.

[Brash]

It's an interesting one.

Inline packet captures across the board seem to be more possible than ever now that ASIC's and CPU's are getting much faster and buffers can be deeper. It'll always be de-prioritized to ensure packets are routed/switched if the CPU is too busy to make a copy, but it seems that most devices these days are fairly capable to capture a high number of packets across the device.

That said, I've had issues with inconsistencies with captures from the dashboard.

I'm currently investigating an issue with small UDP packets being dropped.

The packet capture on the MS misses a decent number of packets that I know with certainty went across the port I'm capturing on. And we're nowhere near pushing line rate across the switch.

I'm hoping a SPAN (Port Mirror) will be more reliable.

[RaphaelL]

I just tested a Span port on a MS120 , it is much more reliable , but I'm still dropping packets out of the blue. 

This is a bit annoying when you want to do a proper troubleshooting.