One Spoke has slow speeds on VPN

[KenobiWan]

Let me just say I am by no means a networking expert. I’m just a troubleshooter. If I use the wrong approach or terminology, please feel free to point me in the right direction.

My company has 3 sites all using Meraki MX devices. Connected over VPN. Let’s call them A, B, and C. 

Sites A and B are in the same town. Transfer speeds seem fine. 

Site C is about 130 miles away. we cannot get our data transfer speeds above an average of 15-20Mpbs. Both sites A and C have symmetrical Gigabet internet. all internet speed tests are fine. 

All servers are at Site A and all sites use ERP software that communicates to SQL at Site A. 

I did a packet capture at Site C of a speed test sending 1 megabyte of data to and from the data collection folder. (Granted it was also showing whatever other traffic was happening between my computer and the server in that couple seconds as well) It shows that there is a lot of fragmentation on the sending but none on the receiving. Like thousands of instances of fragmentation. 

According to support the default MTU on all the Merakis is 1500. The Tunnel MTU IS 1400 and the packets were fragmenting at 1386. I pinged the server and told it not to fragment, hoping to find the biggest acceptable MTU. This returned a result of 1344 was the biggest the server would take without fragmenting. 

I lowered my Ethernet MTU to 1344 and did another packet capture. Running the same test of reading and writing 1 megabyte to the data collector. Now the packet fragments at a lower number. Still seemingly thousands of instances of fragmentation. 

I also did a tracert and pinged all the hops between me and server (12 hops) with -f and gradually found their fragmentation points as well. Some were lower than 1344. 

I should also note that I brought the Site C Meraki to my home about 20 miles from C and tested it on a completely different ISP, also with gigabit speed. No difference. 

can anyone suggest the next step? I just started with this company and I would basically be a god among men if we could solve this. (Not really but it would feel really good to solve this for them.)

Thanks in advance!

[Ryan_Miles]

What model MXs are these?

[KenobiWan]

Site A: MX95

Site B: MX68

Site C: MX95

[PhilipDAth]

It sounds like you are experiencing an MTU squeeze.

Here are some experiments you can try.

Try reducing the MTU one of the actual computers instead.  Start by listing the interfaces.  If you are using a cabled connection, it is often called "Ethernet".

netsh interface ipv4 show subinterface 

Once you have the interface name, reduce the MTU:

netsh interface ipv4 set subinterface “Ethernet” mtu=1300

If you reboot, the above setting will be undone.  To make it permanent add "store=persistent" to thre above line.

Timestamps aid in recovery.  This needs to be done on both ends.  Timestamps especially help on circuits with asymmetric response times.

netsh int tcp set global timestamps=enable 

[KenobiWan]

Hello and thank you for taking time to reply. I did try a wide variety of MTU settings on the ethernet port with the same results.

We ended up RMAing the site C just to see if it was a hardware issue (it wasn't) and we havent returned it yet so I currently have a spare at home I am tinkering with so as to not disrupt anything at the office. Ive rebooted it multiple times. 

Yes we only run stable firmware.

I have enabled timestamps on noth end with no effect. 

Thanks for trying!

[PhilipDAth]

Another experiment you could try - disable IPS on Site C and repeat the speed test (and then enable again afterwards).

Also I assume you are running current "stable" firmware or better ...

You could also try giving the MX at site C a reboot in case it has gotten into a sick state.

[ww]

What speed do you get between site a and b?

What latency you have between a and b?

What latency do you have between a and c?

Can you try a iperf session between a and c? With 1 and with 10 streams

[alemabrahao]

Was it working fine before? Have there been any recent changes that might be causing the problem (a firmware update for example)?

[mwiater]

Is the performance problem that you described restricted to the testing you conducted? Does conventional internet traffic experience the same?

Was your test to the data collection folder using windows file sharing, SMB?  Does all traffic over this VPN behave like this on the affected link?  That SQL traffic is also slow?

While fragmentation may be an issue, I'd not exclude other problems creating the slowness you described.