We recently went to a full Meraki setup (at my insistence) here at my current employer. Four offices across the state of Florida. Two larger offices and two smaller. The two larger offices use MX84 firewalls, and the two smaller have MX64's.
Main office: MX84 Firewall, two WAN links. Link 1 is a 50/50 fiber, Link 2 is a 30/10 Cable Modem. Firewall is set for fail over, not load balancing, though internet/web traffic is set to use Link 2.
Other large office: MX84 firewall, two WAN links. Link 2 is a 30/30 fiber, Link 2 is a 20/5 cable modem. Firewall is set for fail over, not load balancing.
First smaller office: MX64 firewall, One Wan link, 40/40 fiber, super cheap because of location.
Second smaller office: MX64, One Wan link, 25/25 fiber.
There's a fair amount of file transfers between the offices daily, we are a financial services/bookkeeping/tax/audit firm. Ever since we put all this in place, we've had a very strange problem.
File transfers between the main office and the two smaller offices, as in standard Windows file transfers, are running around 5-6 MB/sec. However, file transfers between the main office and the other large office, the one that is also running a MX84 are capping out at 2 MB/sec. It's like a flat line if you graph it. It hits that limit and stays there.
File transfers between the second larger office and the two smaller offices are all over the place.. upwards to 4, then fall flat to 300k, then back up.. etc.
Iperf testing between the main office and the other office running the MX84 give me 17/18 MB/sec transfers. But any Windows file transfers are dog slow in comparison.
This behavior happens regardless of the windows machines.. client to client, client to server, server to server, physical machine to virtual, virtual to virtual, doesn't matter. Any and all smb/cifs file transfers are somehow, somewhere being capped out at 2 MB/sec.
This did NOT happen with the old firebox firewalls we had in place before...Any help? Going crazy trying to figure this out, and catching a fair amount of flak over it.
It sounds like you have a site to site Traffic shaping rules in place. If not I would just try to disable Advanced Malware Protection (AMP) & Intrusion detection and prevention for both sites ( not recommended since you guys are handling financial services/bookkeeping/tax/audit) But worth a try maybe after hours etc. if Meraki support has no better idea.
Forgot to mention that we have tried traffic shaping rules.. and turned them off. I believe we tried turning off those rules, but I find it odd that they are on for all four locations but this issue only raises in that one office, just my luck it has to be the two offices that have the most file traffic between them.
The two boxes we get the most complaints about are a pair of server 2012 r2 file servers.
Please try this command on the server (and ideally) on the workstation involved. It doesn't need a reboot. It affects all new TCP connections (but not those already negoiated):
netsh int tcp set global timestamps=enabled
You can undo the setting by setting it back to disabled.
By default windows bases its round trip time on the average of the send and receive time. This works ok if the send time is the same as the receive time for a packet. It does not work very well on asymmetrical circuits. If you look in Wireshark you'll see a lot of duplicate packets and retransmits (if you are affected by the issue).
Enabling timestamps causes Windows to calculate the sending and receiving time separately, rather than just averaging them. This stops the retransmission issue.
Tried it, didn't help sadly. I should mention that the server in question is a Hyper V virtual Machine, running on a Server 2012 R2 host.
ok..
1) Yes, well above it. Everything is Gigabit, getting 210 MB/sec when transferring a large file locally.
2) across the vpn average is 27ms
3) And yes, running the latest firmware and drivers, and verified the VMM queues are off
It is not possible to get 210 MB/s on Gigabit Ethernet. The maximum you can get is around 100MB/s. Perhaps you meant 210 Mb/s - but that would not be a great result.
Lets test the MTU theory. Using PPPoE often causes an MTU squeeze. You can do this on either the server or a workstation to test. If it solves the performance issue on a workstation then you'll need to repeat it on a server so everyone benefits.
Use this command to display all your interfaces:
netsh interface ipv4 show interfaces
Note the interface number that is configured with the IP address used for communication. Lets pretend it is interface 10.
Then issue this command:
netsh interface ipv4 set subinterface "10" mtu=1400 store=persistent
To undo this repeat the command with an MTU of 1500. This change will affect all new TCP connections.
No change.. still hitting that 2 MB/sec and staying there. Well it did go to 2.23 MB/sec for a few seconds.. but dropped right back down.
Checked that, even had a guy unplug that Cable modem in Wan 2, same behavior.
Yeah.. I've been beating my head against this for a few weeks now.
So iPerf was reporting 17 Mb/s - a similar result.
Have you done a speed test to make sure your Internet circuit is the speed it is meant to be?
Yep. Multiple times.
Yeah will probably try that Monday. We are also going to call the ISP.. not that I think there is anything on their end, but just worth checking.
Perhaps you should go to Appliance Status/Uplink, and confirm the traffic is going out the uplink that you are expecting.
It does go down. By a lot.
@JoshPGC wrote:
Iperf testing between the main office and the other office running the MX84 give me 17/18 MB/sec transfers. But any Windows file transfers are dog slow in comparison.
Site to site VPN, correct? Windows file sharing over VPN throughput is AWFUL. I have 150mbps fiber and I can pull a speedtest at full speed, both directions, using a VPN connection. But transferring a file through Windows explorer...garbage throughput. In a nutshell, the problem is the MTU. SMB is designed for LAN links, not WAN links - their MTUs are set wildly different. Do some googling on SMB over VPN and you'll find a lot of information on it. That's where I would start, with the MTU.