MX84 upload speed problem

[bsbswilson]

We have had an MX84 in production for 18 months with a 100/100 fiber internet connection.  Recently, we upgraded to a 500/500 (knowing that the MX84 would top out around 250-320 depending on the traffic and options).

We are seeing our new speed with download speed, but upload speed will not break 100Mb/s.  

We have about 20-30 hours worth of troubleshooting into this now with both the ISP and Meraki with no resolution.  Has anyone else experienced this kind of problem?  The things we have confirmed and tried are below.

Speed tests are all done with ISP provided speed test site as well as with iperf.

Firmware on latest 14.x

MX has no traffic shaping in place besides main setting for speed is set to 500 symmetrical.  (and we've tried setting this to 1G with no change)

A laptop set up in front of the MX gets full 500/500 speed.  (confirming no traffic shaping on the ISP side)

Same laptop behind the MX gets 500/100.  (with nothing else on the LAN ports)

Two laptops on two LAN ports get 1G/1G between each other.

Tried spare MX from our spare shelf.  Same 500/100.

Meraki RMA'd our unit and sent a brand new one.  Same 500/100.

Set MTU to 1452, no change.

3 separate MX84 units all have the same 100M upload limit on ISP circuits that allow for up to 500M upload on a laptop in front of the MX.

The only thing on my list next to try is the 15.x firmware and if that doesn't work I'm out of ideas and it seems the Meraki techs are as well.

[Bruce]

Have you tried creating a new network in the Dashboard and adding the MX to that and configuring for 500/500? Just wondering if the config for the network isn't updating the Upload speed correctly, and if you're continually using the same network for testing every device is getting the same 'corrupt' config pushed to it.

[bsbswilson]

We were always testing with the same network and just swapping in new devices, until last night when we created a new network and tested with that.

We finally have upload speeds of 300-400 after doing these things:

Factory reset on newest unit I got from Meraki via RMA

Built new network

Firmware update to latest beta 16.x

Tested at home (same ISP as the office)

After good upload speeds, we downgraded to the latest 14.x firmware and still got good upload speeds.

Right now, I do not know if the new network contributed to this success or if the firmware upgrade/downgrade contributed, or if the new unit simply contains some 'fix' in the hardware - or some combination of the three.

Tonight I will be deploying this good unit in place of my production unit and we'll see if it keeps performing properly or if something in my network config is corrupt.  Then I'll have to take the other two suspect units home and go through the new network path plus the firmware upgrade/downgrade to know if those units can be fixed that way or if there truly is a hardware problem with them.

Thank you everyone for your input.  Maybe this journey is almost over!

[Inderdeep]

@bsbswilson : Glad to know you got your solution. Well it shouldn't  be like upgrade and downgrade the firmware will any effect on the upload speed. May be reset sometimes will help 

Good Luck !

[bsbswilson]

The only thing I could think of with the firmware updates is that if there is a 'corrupt' config somewhere in the mix, firmware updates may re-write the config in a way that fixes that corruption.

Without more visibility into the actual config on the device though, just guessing based on the steps taken.  

After tonight's maintenance, I'll post an update with the final conclusions.

[BrandonS]

I appreciate the updates. This sounds like a painful troubleshooting gauntlet, but your methodology seems sound and you are breaking the problem down into pieces so should at least end up with a general idea where the trouble was/is.  

[BrandonS]

Your description makes me think of a duplex mismatch.  It could explain the 100mbps limit.  If you can't control that on the ISP side maybe try a dumb gig switch as a test between your WAN port and the ISP handoff.  If you can control it or have someone at the ISP check on their side you may be able to verify that port's negotiated status too.

[Inderdeep]

@bsbswilson @I am also thinking of duplex mismatch what @BrandonS expected. Did you enable auto auto in both sides 

[bsbswilson]

Forgot to add that... we did go through the various options for negotiation.  All the way from 100 half on both the ISP device and ours, then 100 full, then 1G half, 1G full, auto, and every mix we could think of... auto on our side, 1G full on their side.  Nothing changed the performance other than any 100 setting of course slowed down our download to match the upload speed.

[PhilipDAth]

Are you sure you haven't configured a WAN speed under traffic shaping?  The circuit will be limited to whatever you configure here.  Typically, you can just set it to the maximum.

[bsbswilson]

Yes, that was the first thing Meraki support thought we missed.

The maximum value allowed is 500M on our model.  With the slider or manually type in, it limits to 500 and that's how we have it set.  Along the way, I had set this down to 10 and saw it limit to 10 both up and down, then moved it back to 500 and the download moved to 500 as expected, but upload only rose to 100.  I also tried a few silly things like a download limit of 1 with an upload of 10, 50, 100, and 500.  Everything functioned as expected except upload topped out at 100 regardless of how high I went on the config.

Personally, this is what I think may be 'corrupt'.  While the slider shows it changing, internally, something in the config still has the upload side set to 100M and it won't update.  Why it would slow down, but not increase, is a mystery, but I've seen stranger things happen in code.  Nothing else I can think of would limit upload only.

[bsbswilson · Accepted solution]

We are working at full upload speed now.  Thank you everyone for your input and I hope this thread helps someone else some day.

After testing the 3 devices that we have on hand at this point again several times, we determined that something in the network config is to blame.  I'm guessing it is the WAN traffic shaping policy that looks correct in the UI, but behind the scenes was actually broken in some way.

All 3 devices, on various firmware versions, work as they should with full upload bandwidth once we built a new network.  If we apply the old network to any of them, the upload limit comes back.