Odd and annoying file transfer issue

[JoshPGC]

We recently went to a full Meraki setup (at my insistence) here at my current employer. Four offices across the state of Florida. Two larger offices and two smaller. The two larger offices use MX84 firewalls, and the two smaller have MX64's.

Main office: MX84 Firewall, two WAN links. Link 1 is a 50/50 fiber, Link 2 is a 30/10 Cable Modem. Firewall is set for fail over, not load balancing, though internet/web traffic is set to use Link 2.

Other large office: MX84 firewall, two WAN links. Link 2 is a 30/30 fiber, Link 2 is a 20/5 cable modem. Firewall is set for fail over, not load balancing.

First smaller office: MX64 firewall, One Wan link, 40/40 fiber, super cheap because of location.

Second smaller office: MX64, One Wan link, 25/25 fiber.

There's a fair amount of file transfers between the offices daily, we are a financial services/bookkeeping/tax/audit firm. Ever since we put all this in place, we've had a very strange problem.

File transfers between the main office and the two smaller offices, as in standard Windows file transfers, are running around 5-6 MB/sec. However, file transfers between the main office and the other large office, the one that is also running a MX84 are capping out at 2 MB/sec. It's like a flat line if you graph it. It hits that limit and stays there.

File transfers between the second larger office and the two smaller offices are all over the place.. upwards to 4, then fall flat to 300k, then back up.. etc.

Iperf testing between the main office and the other office running the MX84 give me 17/18 MB/sec transfers. But any Windows file transfers are dog slow in comparison. 

This behavior happens regardless of the windows machines.. client to client, client to server, server to server, physical machine to virtual, virtual to virtual, doesn't matter. Any and all smb/cifs file transfers are somehow, somewhere being capped out at 2 MB/sec.

This did NOT happen with the old firebox firewalls we had in place before...Any help? Going crazy trying to figure this out, and catching a fair amount of flak over it.

[Jack]

It sounds like you have a site to site Traffic shaping rules in place. If not I would just try to disable Advanced Malware Protection (AMP) & Intrusion detection and prevention for both sites ( not recommended since you guys are handling financial services/bookkeeping/tax/audit) But worth a try maybe after hours etc. if Meraki support has no better idea. 

[JoshPGC]

Forgot to mention that we have tried traffic shaping rules.. and turned them off. I believe we tried turning off those rules, but I find it odd that they are on for all four locations but this issue only raises in that one office, just my luck it has to be the two offices that have the most file traffic between them.

[PhilipDAth]

I have seen issues like this before (not just with Meraki). If iPerf is getting 17/18 MB/sec and windows file transfers are dead slow it is almost certainly a windows TCP stack setting that needs changing.

If you tell me what Windows OS you are using for the two boxes you are copying the file between I'll recommend some settings for you to try changing. I bet we can get this going considerably faster,

[JoshPGC]

The two boxes we get the most complaints about are a pair of server 2012 r2 file servers. 

[PhilipDAth]

Please try this command on the server (and ideally) on the workstation involved.  It doesn't need a reboot.  It affects all new TCP connections (but not those already negoiated):

netsh int tcp set global timestamps=enabled

You can undo the setting by setting it back to disabled.

By default windows bases its round trip time on the average of the send and receive time.  This works ok if the send time is the same as the receive time for a packet.  It does not work very well on asymmetrical circuits.  If you look in Wireshark you'll see a lot of duplicate packets and retransmits (if you are affected by the issue).

Enabling timestamps causes Windows to calculate the sending and receiving time separately, rather than just averaging them.  This stops the retransmission issue.

[JoshPGC]

Tried it, didn't help sadly. I should mention that the server in question is a Hyper V virtual Machine, running on a Server 2012 R2 host.

[PhilipDAth]

Buggar, Ok. I have seen a lot of issues with Hyper-V and NIC drivers. Can you confirm that the issue does not happen on the local LAN - that you can copy a file over the local LAN and get near 100 MB/s (that is Megabytes/second - assuming everything is Gigabit connected)?

[PhilipDAth]

When you ping over the VPN to the server what is the typical round trip time?

[PhilipDAth]

Does by chance your server have Broadcom NICs in it?

[JoshPGC]

ok.. 

1) Yes, well above it. Everything is Gigabit, getting 210 MB/sec when transferring a large file locally.

2) across the vpn average is 27ms

3) And yes, running the latest firmware and drivers, and verified the VMM queues are off

[PhilipDAth]

It is not possible to get 210 MB/s on Gigabit Ethernet.  The maximum you can get is around 100MB/s.  Perhaps you meant 210 Mb/s - but that would not be a great result.

Lets test the MTU theory.  Using PPPoE often causes an MTU squeeze.  You can do this on either the server or a workstation to test.  If it  solves the performance issue on a workstation then you'll need to repeat it on a server so everyone benefits.

Use this command to display all your interfaces:

netsh interface ipv4 show interfaces

Note the interface number that is configured with the IP address used for communication.  Lets pretend it is interface 10.

Then issue this command:

netsh interface ipv4 set subinterface "10" mtu=1400 store=persistent

To undo this repeat the command with an MTU of 1500.  This change will affect all new TCP connections.

[JoshPGC]

No change.. still hitting that 2 MB/sec and staying there. Well it did go to 2.23 MB/sec for a few seconds.. but dropped right back down.

[PhilipDAth]

The "other" large office has a 30/30 circuit and a 20/5 circuit.

The 30 Mb/s circuit should give you around 3MB/s and the 20Mb/s circuit should give you around 2MB/s - if they have all of the bandwidth available.

This smells like it is using your 20Mb/s circuit.

[JoshPGC]

Checked that, even had a guy unplug that Cable modem in Wan 2, same behavior.

[PhilipDAth]

Ok, that is a tough one then. You are basically getting 2/3 of the bandwidth you should be (assuming nothing else is using the bandwidth).

[JoshPGC]

Yeah.. I've been beating my head against this for a few weeks now. 

[PhilipDAth]

So iPerf was reporting 17 Mb/s - a similar result.

Have you done a speed test to make sure your Internet circuit is the speed it is meant to be?

[JoshPGC]

Yep. Multiple times. 

[PhilipDAth]

The only other thing I can think to try is to grab a WireShark of a file transfer session and see if it brings up anything interesting.

[JoshPGC]

Yeah will probably try that Monday. We are also going to call the ISP.. not that I think there is anything on their end, but just worth checking. 

[PhilipDAth]

Perhaps you should go to Appliance Status/Uplink, and confirm the traffic is going out the uplink that you are expecting.

[PhilipDAth]

Another option would be to force the VPN out the other uplink at the other large site and confirm that the speed goes down. Perhaps the uplinks are transposed to how you think they are.

[JoshPGC]

It does go down. By a lot.

[BHC_RESORTS]

---

@JoshPGC wrote:

Iperf testing between the main office and the other office running the MX84 give me 17/18 MB/sec transfers. But any Windows file transfers are dog slow in comparison. 

---

Site to site VPN, correct? Windows file sharing over VPN throughput is AWFUL. I have 150mbps fiber and I can pull a speedtest at full speed, both directions, using a VPN connection. But transferring a file through Windows explorer...garbage throughput. In a nutshell, the problem is the MTU. SMB is designed for LAN links, not WAN links - their MTUs are set wildly different. Do some googling on SMB over VPN and you'll find a lot of information on it. That's where I would start, with the MTU.