MTU issues after upgrading to MX18+

[RaphaelL]

Hi, 

It seems that Meraki has changed the ESP overhead size from 64 bytes to 68 bytes if you are running MX 18 ++

This may affect IP Fragmentation as shown below. 

This is a capture done on MX15.44. 

Client sends a maximum of 1408 UDP payload + 20 IP header = 1428. Then you add the encryption + 64 bytes = 1492 MTU , this fits almost any normal WAN link ( DSL and fiber and other ) 

( top is 'Internet' capture , bottom is AutoVPN capture ) 

With MX18++ 1408 UDP payload + 20 IP header = 1428 if you add up the new 68 bytes = 1496 !

DSL links might not like that number ! We have encountered some ISP that instead of fragmenting those packets , they were simply dropping them.

I haven't seen any documentation / changelog regarding those 4 new ESP bytes ,  but the 68 bytes is now included in the MTU troubleshooting guide

1. If your packet is traversing over Auto VPN, you will need to account for the 68 byte overhead when determining MTU size.

: https://documentation.meraki.com/General_Administration/Tools_and_Troubleshooting/Troubleshooting_MT...

So , heads up for MTU issues ! 

Cheers , 

[alemabrahao]

These latest versions are increasingly unstable, several problems have been reported and it seems that Meraki is doing little to resolve. I'm in version 16.16.9 until today because whenever I update my network is unstable and I have to rollback.

[ww]

It was already 68 bytes. So maybe something else was changed. The mx should take in account pppoe,  at least it did before

[RaphaelL]

PPPoE is 8 bytes. 

1506 - 14 L2 header = 1492 

1492 - 1428 = 64.  Same test ran with MX18 will give you 68 bytes.  

4 Bytes increase in ESP seems like an upgrade from SHA1 to SHA256. I'm asking support confirmation on how and why he confirmed that the ESP overhead has changed. 

[ww]

I asked over a year ago, and they told me 68 bytes. I dont remember if it was 15 or 16 firmware back then. Did you also test on 16.x?

[RaphaelL]

Trying to pin that firmware on my test setup ! I also asked support if it is since MX18 or since our upgrade ( big jump from 15 -> 18 )

[RaphaelL]

From support : 

Yes this is expected for MX18+

I do not currently have the exact details for why the overhead has increased, but we have made improvements to the VPN design that have resulted in this overhead. There should be no expectation that this will ever return to the previous overhead size in firmware versions greater than MX18

Still chasing in which firmware version this was changed.

[Bucket]

I is 4 bytes lower on MX16.

[RaphaelL]

We changed the MTU of a MX ( running 18.107 ) HUB in a lab from 1500 to 1468.  The advertised MSS is now 1360 ( 1468 - ESP overhead - IP - TCP = 1360 )

Taking a packet capture shows ESP payload of 1400 bytes and total WAN packet lenght is 1474. 1474-L2 header = 1464. Which again gives a ESP overhead of 64 bytes instead of 68 bytes.  That's curious. 

There is something funky going on or I'm missing something obvious written in a RFC older than me.

[HarryCalayan]

We tried to reach out meraki support but they not responding if the case is related to MTU.

[RaphaelL]

What are you experiencing ?

[HarryCalayan]

Our biometric devices in different sites cannot sync to our main office server since we are using AutoVPN. When the support from meraki identified that biometric devices send above the MTU (i think around 1500bytes). They stop replying on the case and left the ticket open.

Thanks for your reply.

[RaphaelL]

Is it over UDP or TCP ?

The MX should have done IP fragmentation. 

[HarryCalayan]

its over TCP. 

They just inform us that the header DF is set to 1. Unfortunately we don't have idea where to change any settings since its only a biometrics.

Some of our sites works fine, I can ping other vpn subnet with 1500 bytes and those biometric devices sync with the server. But some of sites are dropping the packets from the biometric.

[RaphaelL]

Mmmm.... take a packet capture , you will see that the TCP 3WAY handshake contains the MSS ( Maximum segment size for TCP ). It will be  AutoVPN MTU - 40.  

1500-68-40 = 1392 ( if you WAN MTU is 1500 all across your AutoVPN domain ) it might be lower. 

That means that the client is aware that it cannot send over 1392 bytes to the destination over VPN.

[HarryCalayan]

Yes, the client device is aware that it cannot send over 1392 bytes, but based on the packet capture the device still keep trying to send around 1500 bytes multiple times. Then we will get a timeout error. Currently we don't have any idea what to do 😄 😄 😄

[RaphaelL]

if the client is not honoring the MSS recevied then it is not a meraki issue. I could contact the vendor with the appropriate pcaps.  

However , I have never seen a IP stack behaving like that,  so I have some doubts.

[HarryCalayan]

When i tried to ping the spoke MX with 1500bytes it gives the packet fragmentation message w/c is expected. But when i tried around 1399 up to around 1450 bytes, it just timed out.

Edit: 1398 works but not 1400 is timed out

Edit: I have 2 different spokes MX here with different behavior when i ping with 1500bytes

[HarryCalayan]

I don't think that MX is informing the client device that the packet needs to be fragmented.

[RaphaelL]

Because there is no fragmentation happening.  1392 + 68 + 40 = 1500.

[HarryCalayan]

Is that mean the MX just drop the packet? Thanks

[RaphaelL]

Why would  it ? The payload size received is smaller or equal to it's WAN / AutoVPN MTU

[HarryCalayan]

As per meraki support the packet is not received in the other side of vpn. So we don't have any idea.

[RaphaelL]

Take a packet capture on the spoke and the hub , you will see if the packet is received or not. 

Does your WAN support 1500 bytes MTU ? On both side

[HarryCalayan]

Yes, both WAN of hub and spoke support 1500 bytes