MX64 fragments per packet limit

[benny]

Hi Community, 

I've come across a issue with MX64 routers and large UDP packets sent over the WAN to LAN hosts. It seems Meraki limit the UDP payload size or fragmented packets to 4 fragmented packets, I think this roughly translates to 5000 bytes. 

The issue causes problems with our Polycom VOIP handsets and the BLF status of other handsets. This results in call drops and unusable phones within the branch. When I revert back to a old Cisco 871 router it handles these fragmented packets without any issues. 

Anyone experience this particular issue or have any ideas how to overcome this?

[PhilipDAth]

Reduce the MTU on your devices to remove the need for fragmentation by devices in the middle.

[benny]

The large UDP packets are coming from a upstream voice switch, changing the MTU on the interface or device will not change this as the packet needs to be fragmented regardless.

The UDP datagram can potentially be up to 65507 bytes which means over a 1500MTU ethernet link it will be fragmented into 45 packets. 

Traditional Cisco routers do not have this problem.

[PhilipDAth]

I guarantee you that if you lower the MTU to say 1400 bytes, that the device will not send packets bigger than this regardless of the size of the buffer being en-queued by the application - unless its IP implementation is seriously broken.

Consider the case where the MTU is 1500 bytes and their is a 32k buffer to send.  Does the device try to send a 32k packet - no - because it would be dropped immediately.  The device will always break the buffer down into MTU sized packets.

An intermediate device has to perform fragmentation where there is an MTU mis-match between the source and destination interfaces - such as a normal Ethernet interface on one side and an IPSec tunnel on the other.

If the MTU is the same on the inbound and outbound interfaces then the intermediate device will not fragment anything.  It has no reason to do so.

[benny]

Hi Philip,

I do not understand how lowering the MTU on the WAN interface is going to help. The application server on the carrier end is a Broadsoft SIP server.

Here is a excerpt from the Broadsoft documentation about BLF:

Depending on the combined length of contact information fields of the users to be monitored, the NOTIFY packets are typically larger than the MTU (maximum transmission unit) defined in the routers, firewalls, and other data transmission equipment. As a result the UDP packets have to be fragmented into a number of smaller packets to fit the MTU size as they are sent out to the network.

In Ethernet networks, MTU size is typically set to 1500 bytes. A UDP datagram can be up to 65507 bytes in size. A UDP datagram of 65507 bytes will have to be fragmented into 45 packets ( Max UDP size of 65507 bytes / UDP size per packet of 1480 bytes ~ 45) to be transmitted across a network. With this many number of fragments, customer firewalls or NAT devices may prohibit the entire packet from being delivered to the endpoint.

A common issue that has been observed is the firewall or NAT traversing issue. Some firewalls will simply drop any fragmented packets while the others will drop packets when the number of fragmented packets exceeds a configured or default limit. For example, the Cisco ASA 5500 firewall by default will drop a packet if the packet has been fragmented into more than 24 packets. When this happens, phones will not receive the BLF NOTIFY message. As a result phones can only display a certain number of BLF lines.


The recommended solution is to configure the firewalls and/or NAT routers at customer premises to handle fragmented UDP packets correctly. These firewall and NAT routers must be configured to support the maximum UDP payload size of 65507 bytes and to allow at least 45 fragmented packets per packet.

The Meraki MX64 seems to be only allowing 4 fragmented packets or roughly 5000bytes.

I will try your suggestion of lowering the MTU as we are now snookered with this issue, I'll let you know how it goes.

[PhilipDAth]

Don't change the MTU on the WAN interface.

I meant to change it on the server, but in this case the server is Broadsoft, and obviously something you do not have access to.

Does your provider support either SIP/TCP or SIP/TLS (we use SIP/TLS - we like the encryption)?  This will be the next best fix.

[TonyB808]

Hello,

Were you able to resolve this issue with Broadsoft?  I am experiencing the same problem.  Same MX64  with MS switches.  the MTU size on my switches are set to 9578 bytes by default.

Any suggestions would be most appreciated.

Mahalo,

Tony 

[benny]

Hi Tony,

This turned out to be a very interesting issue that we had with our provider.

In the end Meraki support were the most helpful and eventually we proved it was a issue with our carrier. It wasn't until we swapped out the MX with a traditional ISR that we knew it wasn't the MX causing issues.

The back haul on our providers link into our MPLS network had a incorrectly configured packet size. It wasn't the MTU but rather the switched packet size itself.

If you are experiencing BLF issues as well it is possible to convert the SIP registration and BLF updates from UDP to TCP with specific tags and configuration on the handsets. Note this will tax a small percentage of your link with overheads found in TCP packets.

Hope this helps

Ben

[TonyB808]

Thanks.. I still need to have a talk with our carrier but that was my next step.  We don't have BLF issues that I know of. The problem is with Phones with integrated video capability, ex. Cisco 8845, Polycom VVX 1500.  When the phones are video enabled, the phone seizes to dial out.  

Broadsoft came back with:

“The recommended solution is to configure firewalls and/or NAT routers at customer premises to handle fragmented UDP packets correctly. These firewall and NAT routers must be configured to support the maximum UDP payload size of 65507 bytes and to allow at least 45 fragmented packets per packet.”

It's similar to what you were saying, and a quick search popped up with this blog.

I don't really think (my assumption) that our MX64's the issue but I do see the MTU size on our MS switches to be set at 9578 bytes. Don't know if changing the MTU on the switch will help.

[benny]

Interesting.

We have predominately VVX411's and 1500s too.

Are you using MPLS to route voice traffic back to the broadsoft platform?

Are your voice endpoints sitting behind NAT?

The MX's dont allow 45 fragmented packets per packet, I think when we ran tests the maximum was 12 from memory. Start running wireshark and see how many packets you can get fragmented before the conversation is dropped.

[TonyB808]

We have no MPLS to Broadsoft.. 

Yes, our endpoints does sit behind NAT..

We are not  experiencing any dropped calls.. It's only happening to video phones.  When video enabled phones create a session,  the phone does not ring till a busy tone ends the session.   Broadsoft's TAC has the capability to change packets from UDP to TCP. They said that's a temporary fix.  The temp fix works and the video phones work.  

[BenLaakso]

Hi, we're having a similar problem with 9000B packets being sent from our far-end (Oracle cloud) to the Meraki MX67 or MX84. The packets are being chopped up and we believe we see them arrive, but they are not being re-assembled.

Does anyone know if there is a limit to how many packets can be re-assembled?