Packets don't lie ...! or do they ?

[RaphaelL]

Hi ,

Just encountered once again a very special behavior on the MX plateform ( running 19.1.5 ).

Testing scenario : I'm capturing IPSec ( VPN tunnel from my computer to a VPN gateway ) trafic on my PC and on my MX which performs L3 routing for my LAN. My goal is to capture all ESP packets and compare the ESP Sequence to quantify the number of ESP packets inbound that are out of order. To my biggest surprise the 2 traces are way different. 

As you can see the exact same packet with the same IP ID and the same ESP sequence appears out of order only on the pcap taken on a MX. 

Conclusion :  The packet capture engine on the MX is way to unpredictable. Packets are routed and forwarded in the correct order , but they will be scrambled in the captures making troubleshoot harder. 

Yes. I had a case opened in the past. Yes they said this is expected.

Reason #34591 that I dislike MXs.

[Frank-NL]

Thanks for sharing, how is this expected, did they give more info?

[RaphaelL]

I can't quite find the exact answer I got months ago. It was something that the packet capture engine is not prioritized by the CPU. So It can miss packets or show them in the incorrect order.

[rhbirkelund]

Is that MX behind a Firepower by any chance?

[RaphaelL]

No. It is my MX at home. Directly plugged into my ONT.

My PC is a client behind the MX. My PC is bulding a VPN tunnel to a VPN appliance somewhere else. The MX is displaying the packets in the wrong sequence but are received in the right sequence.

[GIdenJoe]

There are situations where an MX does have issues with pcaps.
However I have seen MS switches also do strange stuff.

I once had a VoIP complaint and went to investigate locally.
I did a traffic mirror on an MS switch (MS120) and realized every 10 voice packets it lost one.
However when I captures via dashboard the packets were all there.

[RaphaelL]

Yep pretty sure I also have experienced this behavior on a small MS.