MX85 and MX64 site to site VPN slow file copy

[kidtriton]

I have a MX85 configured as a one arm concentrator at work.  The internet gateway is a Cisco ASA and both devices are connected to a Cisco 3750 2 switch stack.  There is a file server also connected to the 3750.  All three of these are on the same class C internal network.  The internet circuit is 100/100.

I have a MX64 at a remote site (at home for testing) with a single PC connected.  I have it configured as a vpn spoke and the tunnel is up and working. I have defualt route checked so all traffic, even internet, is tunneled through the work site.  This is on a 1000/50 circuit. 

If I do an internet speed test I get ~40Mbps both ways, which I consider good through the VPN.  If I download a file from the internet I get around 40Mbps download.  The problem is a Windows file transfer (SMB).  From Work to Home I can only get 355kbps (~3Mbps) but copying from Home to Work I get almost 20Mbps.  You would think if it was slower one way it would be in the opposite direction since my home upload is 50Mbps, but that's the direction I'm getting decent file transfer speeds.  I also installed a FTP server and client on both ends and FTP has the same speed issues as SMB, while HTTP downloads are fast from Work to Home. 

I did a packet capture on the tunnel using the build in capture on the Meraki dashboard and there are a lot of out of order packets.  But I don't know how much is too much or what is normal, and if that's what is causing the slow copy in one direction but not the other.  

Thoughts?  I've talked to Meraki support about it but they didn't have much to offer.

I want to roll these out to 10-12 home workers but I'm worried its going to be too slow.  

Edited to add:  I also tore down the MX85 config and tried it as a VPN hub in routed mode, with one interface having it own public IP and one interface on the internal network, to eliminate the hair-pinning and take the ASA out of the equation and it was the same exact speed resutls, so I set it back up as one armed. 

[cmr]

@kidtriton we have a similar setup as below:

Wired endpoint->MX65 -> home broadband router -> ISP -> internet -> ISP -> business fibre NTE -> Sophos firewalls -> 3850 stack -> MX250 with the file server also hanging off the 3850 stack. 

The three links to the 3850 stack are all separate VLANs, each with a /24.

Home circuit is 400/400 XGPON

Corporate circuit is 500/500 dedicated fibre

MX65 running 16.15 and MX250 16.12

Download to home 50Mb/s sustained and upload from home 65Mb/s sustained for a 1.2GB file.

3850 link to firewalls running at 1Gb/s, link to MX250 also 1Gb/s and link to file server 10Gb/s

Home wired connection is 1Gb/s

VPN latency as reported on dashboard is 4ms for a 45mile link.

How do those details compare to your testing?

[PhilipDAth]

I'm going to guess the issue is either an asymmetric response time on the circuit, or an MTU squeeze.

Let's test the MTU squeeze first. On either the server or the workstation try:

netsh interface ipv4 show subinterface 

Locate the name of the interface in use and then try:

netsh interface ipv4 set subinterface “<name goes here>” mtu=1300

If that solves it, run the command again on the server and add "store=persistent" on the end to save the setting.

If that doesn't help, try enabling timestamps on both the server and the client:

netsh int tcp set global timestamps=enable 

[kidtriton · Accepted solution]

Well I figured it out, it's a problem with my MX85.  I did all kinds of things over the weekend like put the MX85 on a different subnet, plugged into different internal switches, etc. to no avail.  

I took another MX64 and set it up on the Work side with exactly the same settings as the MX85, just one IP address away, and updated my internal static route to it.  My head end config is very simple, passthrough VPN concentrator, site to site hub with one /16 subnet and that's it.  

The MX64 works great, I'm getting 20-40mbps VPN file copying.  I can point the remote MXs back to the MX85 and change the static route back and get the slow copying again.  I did a packet capture on a file transfer to the MX64 and it looks "clean" whereas the packet capture to the MX85 is full of out of order packets and retransmits.  

I did notice my MX64s are all on firmware 15.44 and the MX85 is showing 16.8, I guess this is because it's a brand new device?  I wonder if its a hardware bug or software bug?  I'm about to get back on the phone with support and show them these findings.  

EDIT:  Just got off the phone with support.  We upgraded the MX85 from 16.8 that it shipped with to release candidate 16.15 and the problem is fixed.