MX85, not hitting 1gpbs performance

Adam2104
Building a reputation

MX85, not hitting 1gpbs performance

Has anyone else tested an MX85 and actually reached the stated 1gbps performance through it? I received mine yesterday, it's running 16.11, and I can only get 700mbps through it. I'm testing internal-to-internal only flows, VLAN to VLAN, to rule out issues with my WAN.

 

Topology:

Client --- (port 5) MX85 (port 6) --- Server

 

Client - Mac Mini 2018, Mac OS 11.6.
Server - Ubuntu 20.04.2 virtual machine, 4 vCPU, 4gigs RAM

 

Test - Copy 8gig file using scp from the client to the server.

 

Test 1 - same VLAN. Topology:
client --- VLAN 10 --- server

 

In this test, the copy runs are wire rate, 1gbps. Using Dashboard I see a reported rate of approximately 930mbps for the switchport the client is connected to. This is the expected result as the MX is not involved in this flow beyond simply switching the traffic.

 

Test 2 - different VLAN. Topology:
client --- VLAN 10 --- MX85 --- VLAN 20 --- server

 

In this test the server is moved to a different VLAN, VLAN 20. In this test the copy runs at approximately 700mbps. Using Dashboard I see a reported rate of approximately 700mbps for the switchport the client is connected to. The only difference in this test is that the traffic must be routed by the MX. All physical cabling is the same between the two tests. To move the server I just change it's VLAN assignment.

 

Because this is VLAN to VLAN, internal-only, traffic there is no NAT. The traffic-shaping configuration doesn't apply because the WAN port isn't being used. The firewall rule permitting this traffic is the first one in the list, so it shouldn't be a rule lookup issue.

 

I thought maybe this was a per-flow limitation, so I brought up a second client and ran the test concurrently with the original client. What happens there is the performance is cut in half, with each client only getting about 350mbps through the MX.

 

Note, I only have the Enterprise license so there's no threat protection (snort, amp, threatgrid, etc) enabled. The datasheet number of stateful firewall speeds of 1gpbs should apply here as that's all I'm asking the MX to do.

 

 

17 REPLIES 17
Brash
Getting noticed

I don't have an MX85 so can't be of too much help here.

 

Typically I would suggest testing using iperf rather than file copies, however given you're getting consistent results it's probably ok.

 

Seeing a drop in performance for routed traffic vs switched traffic is certainly feasible.

Typically routed traffic requires punting to the CPU for lookups where as switched traffic can be switched in hardware, often all within the same ASIC.

 

As for whether this is expected or not though, I'm not sure.

If it's an issue and doesn't meet the needs of your environment, it might be worth reaching out to your Meraki rep.

Adam2104
Building a reputation

@Brash While I certainly agree that routed operation takes more horsepower than switching, this is a question of the device performing to the specs as outlined in the datasheet, which it currently is not, by almost 25%. These are large (1500 byte) packets, in a single long lived flow. This should be one of the easiest flows to reach the max throughput of this device.

 

I will note that there is other background traffic/noise present on the network but not enough to explain the consistent impact to the performance. That noise would also impact Test1, which is consistently reporting line rate transfers on the gigabit links. The results thus far are very consistent which rules out any impact from the background traffic.

 

Interestingly, when I disabled traffic analytics I saw a small, but measurable difference in throughput, 730 vs 700. My theory was that disabling traffic analytics, and thus NBAR2, would lower the CPU-load on the device. My results suggest that this is the case. This would 100% reproducible as well.

 

I've got a case open with Meraki support but I was curious to know what the rest of the community is seeing.

Adam2104
Building a reputation

Following up on my own post here. I replaced the MX85 with a Cisco Firepower1010 running ASA 9.16(2)3. It performed both tests, Test1 and Test2 at full line rate (930mbps). This is a direct swap using the same cables, same test hosts, same WAN link, same background noise traffic.

 

The MX85 is not performing to spec on 16.12.

ww
Kind of a big deal
Kind of a big deal

I get around 800 Mbps with multiple streams. single stream is not good either

i reported issue on LAN-WAN, but LAN to LAN has the same behavior.

 

 

 

cmr
Kind of a big deal
Kind of a big deal

@Adam2104 The 16. release is considered a beta release (though the only option for newer appliances I think).  There have been some performance issues reported and fixed between versions, perhaps this is another?

 

I have an MX65 running 16.12 with the enterprise license and get the full 250Mbps on tests so clearly some devices are performing to spec on 16.12.

 

Edited to remove dumb questions that were already answered...

Adam2104
Building a reputation

@cmr You are correct, MX 16 seems to be the only supported version at the moment. I obviously recognize that it's the beta release, but I don't have any other options. This performance delta is not called out as a known issue. If it is, it should be listed.

Adam2104
Building a reputation

@ww Are you using VLANs in your configuration? I've adjusted my test setup and moved my MX85 up to my test bench. I also moved it to a fresh MX network so it has a near default configuration on it. My results are:

 

No VLANs - test client connected to port 5 - 960mbps combined upload/download throughput. So this is pretty much near line rate, which is the expected performance.

 

5 VLANs (1, 10, 20, 30, 40) - test client connected to port 5, access port, VLAN 1, consistently gets a combined throughput of ~860ish mbps.

 

The behavior is repeatable. The weird thing about it, the extra VLANs, 10, 20, 30, 40, don't have any active clients, or, any active ports. They're simply configured. VLAN1 is the only VLAN that's actually in use because that's the only client connected. In either test scenario the test client is just an access port in VLAN1. It's unclear to me why just the presence of other VLANs (everything else is MX network defaults) would drop the performance so severely. The MX67 this MX85 is going to replace had no issues hitting the 450mbps rated max, even with VLANs enabled.

Adam2104
Building a reputation

I'm going to reply to my own thread again just to keep documentation of the testing I'm doing. With the MX85 out of as my main router, I've now got this topology:

 

Google Fiber (1gbps) --- FP1010 (asa mode version 9.16(2)3) --- various MS120-8 switches --- MX85 (bridge mode) --- test client

 

In this setup, the MX85 is in bridge mode. The rest of the MX network settings are default. I'm doing a simple speedtest using fast.com. The results are all over the map. Some as low as 250mbps down, to as high as 650-700mbps. Nothing that shows a 1gbps. If I disconnect both cables from the MX85 and connect them together using an RJ45 coupler (eww), I immediately get 1gbps download on the next test on fast.com.

 

I'm not sure what else to even look at. I'll get to a point where I think I've narrowed down a set of consistent test results / conditions only to have the results change dramatically on the next test.

 

Another oddity is that I've been getting a warning in the dashboard that the MX isn't connected using TCP/443 for its tunnel to Meraki when attempting a code upgrade (16.8 -> 16.12). I never saw that on the MX67, ever. There is no firewall blocking it. Maybe there's a process on the MX85 that keeps crashing / restarting causing unnecessary processor load that would explain the wonky performance.

cmr
Kind of a big deal
Kind of a big deal

@Adam2104 I'm guessing that when you enable VLANs the MX changes from switched mode in the hardware to routed mode in software, but this is purely a guess.  Years ago when I worked for a network vendor in design and test we had a switch that used store and forward most of the time, but if all active ports were at the same speed and duplex then it used cut-through.  The latency was dramatically less and the throughput higher.  As it was a distribution switch it was quite common for customers to be able to take advantage of this mode.

Adam2104
Building a reputation

For the sake of thoroughness I ran some additional numbers on the passthrough test case.

 

Test client: 2018 Macbook Pro, MacOS 11.6, Safari 15.0.

 

I am testing using fast.com My fast.com settings are:

Connections: 8 min, 16 max

Duration: 15 sec min, 30 sec max

 

Test Case 1:

test client -- MX85 (passthrough) -- MS120-8LP -- MS120-8 -- MS120-8LP -- Cisco Firepower 1010 ASA Mode 9.16(2)3 -- Google Fiber -- Fast.com

 

Results - download / upload:

 

630/770
390/780
640/790
620/790
560/780
510/740
520/730
680/710

 

All numbers in mbps.

 

Test Case 2:

test client -- RJ45 coupler -- MS120-8LP -- MS120-8 -- MS120-8LP -- Cisco Firepower 1010 ASA Mode 9.16(2)3 -- Google Fiber -- Fast.com

 

Results - download / upload:

 

1.1gbps/710
1.1gbps/740
990/770
1.0gbps/730
1.0gbps/790
1.0gbps/730
990/720
1.0gbps/750

 

All numbers in mbps unless otherwise indicated.

 

Interestingly, upload speeds seem to be fairly consistent. Download speeds are not. I've tested this on 16.8 and 16.12. The results are the same. No other clients are connected to the MX85 except this test client.

Adam2104
Building a reputation

This appears to have been resolved as of 16.13. My MX85 is performing at least on par with expectations at this point. 

NateF
Conversationalist

Hi Adam2104, you might have already mentioned this but did you whitelist the hosts you are testing with or use a group policy on the MX to bypass any firewall, URL filtering or AMP and IDS/IPS services ? those would need to be disabled as they affect the overall performance of the MX.

Adam2104
Building a reputation

@NateF I do not have any of those features enabled. The MX is attached a completely new MX network, the config is default. There's no firewall rules, no IPS, no AMP, nothing. Just raw L3/L4 firewalling.

NateF
Conversationalist

Ok that is strange then, I only have an MX84 or MX100 to compare to or I would try and help you more.... but good to know you got it working on a FP1010 so must be the MX85 for sure, good luck with everything !

Dunky
Getting noticed

I'm looking to purchase an MX85 in the very near future - for the very reason that it is supposed to support the 1Gbps internet connection we have.

Have you had a chance to test it on v14?

Adam2104
Building a reputation

@Dunky It's not supported on v14. v16 the only supported train at the moment.

Dunky
Getting noticed

Aha, useful to know, thanks @Adam2104 

Get notified when there are additional replies to this discussion.
Welcome to the Meraki Community!
To start contributing, simply sign in with your Cisco account. If you don't yet have a Cisco account, you can sign up.
Labels