MX 105 Stack

[SSAS]

Just within the last month our internet has slowed by 75%.  It worked fine up until a month ago.  Using a MX 105 that was suggest to us by Meraki to fit our needs. Using a 2 gig SFP only getting 30 to 50 Mbps download.  900 + uploads.  It looks like our Trunk ports will only handle MTU of 1500.  it needs to be 9000.  Does anyone know if we need to MX 250?  

[RaphaelL]

Have you opened a ticket to Support ?

What does the stats of the device say ? CPU , memory , client count , number of sessions and so on. They should be able to tell if you are having a hardware bottleneck ( which I really doubt )

What firmware are you running ?

[SSAS]

We have been working with support sense 09-25-2023.  They don't know what is going on.  Lots of testing.  event sent us a new MX 105.  Current version: MX 18.107.4. CPU not sure where to look, I have looked all over the dashboard.  1255 clients.  

[alemabrahao]

Check the Recommended use case.

https://documentation.meraki.com/MX/MX_Overviews_and_Specifications/MX95%2F%2F105_Datasheet

[SSAS]

Thanks for the info.  We were sold a unit designed by Meraki sales to fit our needs but it seems as through we were mistaken.  So sad.  Wish I would have stayed with Watchguard.  

[ww]

It just went bad from one day to another? Nothing changed like fw version?

What is a 2gig sfp? Like a 1Gb up/down?

[Ryan_Miles]

A 2 gig SPF? It appears you have a 10G bps transceiver in WAN 1 and a 1 Gbps transceiver in WAN 2 (physical port 4). And a 10 Gbps transceiver on LAN port 9.

How are you measuring throughput? From what source to what destination? Using what tool or website?

I just ran three speed tests using the MX tools page and got more than 1 Gbps each time. Even outside of these tests watching ongoing usage I'm regularly seeing 300-600 Mbps.

I also see in the event log almost everyday you have primary WAN link transitions between WAN 1 & 2 which you cause me to investigate the health of those links/carriers.

[kj4qwt]

Thank you Ryan,

Our issue is not on the upload side but rather the download. Is there a way to test from the Meraki dashboard to the MX105?

[Ryan_Miles]

The dashboard throughput test is a download test not upload.

[SSAS]

You will see my System Admin chiming in on this thread.  We have been testing using our backup WAN2.  Our carrier is solid.  CrownCastle has been onsite and verified the circuit.  

[Ryan_Miles]

Has all the cabling been swapped, transceivers swapped, what is the carrier handoff device? While the provider has verified their link there's no denying you're having many uplink failover events. But that might be a secondary issue.

What still needs to be addressed here is how are you measuring throughput?

Prior to the slow down a month ago what firewall was in use? The MX105 appears to be a new deployment. And was the MX105 working ok for some period of time?

[kj4qwt]

Hello Ryan,

The reason you are seeing excessive drops on our WAN1 interface is due to repeated testing with Meraki support requiring us to disonnect the fiber.

Our throughput is measured using a custom speedtest site from our ISP.

The speedtest is RFC Compliant. I'm not sure of what the RFC number is right now, I'm working on getting that information shortly.

Meraki support had shipped us an identical RMA with suspicion that it could be the previous MX105. However, the new MX105 is no different from the previous one.

We think that it could be a software update that is responsible for the degredation in router performance because we didn't notice any significant changes until the beginning of September of this year.

[kj4qwt]

I must also mention that we've tested the speeds using a MacBook Pro as a router sharing it's internet connection from an SFP+ Fiber NIC card to a GigaBit Copper Ethernet connection to a test computer. The upload and download performance outshined the MX105.

[ChristophJ]

Hi SSAS,

Is there maybe some traffic shaping in effect that is configured on your MX? A low download like that seems weird, especially because your upload is 900+ Mbps. 
What does the uplink configuration under Security & SD-WAN in SD-WAN & traffic shaping say?

What kind of SFP module are you using? Can you tell us the module name? I haven't seen a 2 gig SFP from Meraki.

I run a MX105 and I can get to 1 Gbit/s up and download without any issues with IPS enabled. Probably more would be possible too, but my uplink is just 1 Gig.
Since when are you running into this issue? Was there a firmware update and it started afterwards? Were there config changes somewhere in the network?

Also if the MX is having a hardware bottleneck that should be visible under Organization -> Summary Report and then select the Appliance network. The utilization will be visible in the bottom left corner.

Otherwise support will be able to tell if there is a hardware bottleneck. They have access to more data about the appliance.

[kj4qwt]

Hello ChristohpJ,

I've attatched some screenshots of your requested information on the MX105.

The SFP+ Fiber Module is 10G.

[cmr]

It shouldn't matter too much, but @Ryan_Miles said you have  1Gb transceiver in WAN 2, yet you have set it to 3Gb of bandwidth, I would lower it.

Also the download chart clearly shows more that 30-50Mb/s - are you only having the issue with WAN2, is WAN1 working as expected?

[kj4qwt]

Sorry. I forgot to mention that the two different fiber modules were used as a demonstration for Meraki support to rule out that optics overload is not to blame for the close proximity between our providers NID and the MX105.

[SSAS]

WAN 2 should be ruled out as we only use it as a failover backup circuit.  

[cmr]

@SSAS so if WAN1 is working at up to at least 220Mb/s download - per the graph above, what is at 30-50Mb/s?

[SSAS]

Internet is very slow 30 to 50 mbps throughout network.  All devices.  User complaints.  

[cmr]

If you go from a client to fast.com and test there, what do you see?

[SSAS]

170 Wired, 100 on Wireless

[cmr]

So not 1-2Gbps, but much better than 30-50Mbps!

[cmr]

With the wired connection are you plugged directly into one of the MX105 LAN ports, or via switches etc.?  If not direct, have you tried that?

[kj4qwt]

The one thing that we cannot see are the counters that represent the Giant Frames on interfaces 1 and 9 SFP+ 10G Fiber interfaces on the MX105.

I think the issue here is too many resends of packets because of MTU size mismatches between our LAN and the Providers NID networking equimpment and the MX105.

Our providers customer fiber interface facing our MX105 has it's MTU set to 9000 and our main switch is set to 9578.

My understanding is that the MX105 only supports an MTU of 1500 MAX, even if the LAN interface is configured as a trunk port.

[cmr]

Does your provider host servers for you, the internet itself generally never has an MTU above 1500.  If the device you are using has an MTU of 1500 and the switch has a max MTU of 9578 the biggest packet will still be 1500 bytes.  Do your devices that are accessing the internet have large frames enabled and if so why?

[kj4qwt]

Hello cmr,

The reason for the large MTU size is because we are using trunk interfaces between our switches to more efficiently handle large volumes of traffic, especially when our MX105 is being used for inter-vlan routing.

Normally it's standard to have trunk interfaces have an MTU in the 9000's range for this reason.

Are you suggesting that we need to drop our MTU on our switches and ISP NID down to 1500?

[cmr]

Unless your endpoint devices have large frames enabled, having larger frames on the switches will make no difference.  The usual reasons for having large frames is for iSCSI storage networks, though unless you have lots of sequential data it doesn't make a lot of difference in most cases.  We don't enable large frames on any networks including the iSCSI ones.  However the network switch max frame size remains at 9578.

As requested above, do any of your endpoints have large frames enabled, servers, storage systems or client devices?  If not then that is a red herring.

What speed did you get when plugging a client directly into the LAN ports of the MX105?

[kj4qwt]

Sorry cmr, I'm confused.

Are you saying that it matters to have the MTU size match between endpoints, but not between networking equipment?

When we plug in directly into an available copper Ethernet port on our MX105, we do get faster speeds than compared to going through our 10G fiber LAN interface. Like in the 200MB/s range for download and the 900MB/s range for the upload.

We get the same results when completely disconnecting our LAN network from the MX105 and just using one computer on an available copper ethernet interface on our MX105.

[cmr]

@kj4qwt what I'm saying is that the endpoint MTU size must be the same or smaller than anything in the path. 

So if you had endpoint (1500) -> switch (9578) -> server (1500) then the data transfers would be in packets of up to 1500 bytes and it should work well.

On the other hand if you had endpoint (6000) -> switch (9578) -> router (1500) -> carrier (9000) -> Internet (1470) -> server (1500) then any packet over 1470 bytes in size would be fragmented and if the client was sending 6000 byte packets then they would be broken into at least four parts when they got to the router and eight parts by time they got to the server.

If at the above the endpoint had a max MTU of 1470 there would be no fragmentation and if it had a max MTU of 1500 then it would only be in 2 fragments.

[kj4qwt]

Sorry cmr for not answering your last question.

All of our clients are using an MTU size of 1500.

[cmr]

Hmm, in that case the large max mtu on the switch shouldn't matter...  you could try reducing the switch mtu to 1500 just to see if it makes a difference (and be ready to revert if any issues discovered).

[kj4qwt]

When our ISP changes their MTU to 1500 on the customer handoff interface of the NID, errors start coming up.

[SSAS]

Not sure if this site adjust for error correction.  If I run the ISP speed test back down to 107.