MX85, not hitting 1gpbs performance

[Adam2104]

Has anyone else tested an MX85 and actually reached the stated 1gbps performance through it? I received mine yesterday, it's running 16.11, and I can only get 700mbps through it. I'm testing internal-to-internal only flows, VLAN to VLAN, to rule out issues with my WAN.

Topology:

Client --- (port 5) MX85 (port 6) --- Server

Client - Mac Mini 2018, Mac OS 11.6.
Server - Ubuntu 20.04.2 virtual machine, 4 vCPU, 4gigs RAM

Test - Copy 8gig file using scp from the client to the server.

Test 1 - same VLAN. Topology:
client --- VLAN 10 --- server

In this test, the copy runs are wire rate, 1gbps. Using Dashboard I see a reported rate of approximately 930mbps for the switchport the client is connected to. This is the expected result as the MX is not involved in this flow beyond simply switching the traffic.

Test 2 - different VLAN. Topology:
client --- VLAN 10 --- MX85 --- VLAN 20 --- server

In this test the server is moved to a different VLAN, VLAN 20. In this test the copy runs at approximately 700mbps. Using Dashboard I see a reported rate of approximately 700mbps for the switchport the client is connected to. The only difference in this test is that the traffic must be routed by the MX. All physical cabling is the same between the two tests. To move the server I just change it's VLAN assignment.

Because this is VLAN to VLAN, internal-only, traffic there is no NAT. The traffic-shaping configuration doesn't apply because the WAN port isn't being used. The firewall rule permitting this traffic is the first one in the list, so it shouldn't be a rule lookup issue.

I thought maybe this was a per-flow limitation, so I brought up a second client and ran the test concurrently with the original client. What happens there is the performance is cut in half, with each client only getting about 350mbps through the MX.

Note, I only have the Enterprise license so there's no threat protection (snort, amp, threatgrid, etc) enabled. The datasheet number of stateful firewall speeds of 1gpbs should apply here as that's all I'm asking the MX to do.

[Brash]

I don't have an MX85 so can't be of too much help here.

Typically I would suggest testing using iperf rather than file copies, however given you're getting consistent results it's probably ok.

Seeing a drop in performance for routed traffic vs switched traffic is certainly feasible.

Typically routed traffic requires punting to the CPU for lookups where as switched traffic can be switched in hardware, often all within the same ASIC.

As for whether this is expected or not though, I'm not sure.

If it's an issue and doesn't meet the needs of your environment, it might be worth reaching out to your Meraki rep.

[Adam2104]

@Brash While I certainly agree that routed operation takes more horsepower than switching, this is a question of the device performing to the specs as outlined in the datasheet, which it currently is not, by almost 25%. These are large (1500 byte) packets, in a single long lived flow. This should be one of the easiest flows to reach the max throughput of this device.

I will note that there is other background traffic/noise present on the network but not enough to explain the consistent impact to the performance. That noise would also impact Test1, which is consistently reporting line rate transfers on the gigabit links. The results thus far are very consistent which rules out any impact from the background traffic.

Interestingly, when I disabled traffic analytics I saw a small, but measurable difference in throughput, 730 vs 700. My theory was that disabling traffic analytics, and thus NBAR2, would lower the CPU-load on the device. My results suggest that this is the case. This would 100% reproducible as well.

I've got a case open with Meraki support but I was curious to know what the rest of the community is seeing.

[Adam2104]

Following up on my own post here. I replaced the MX85 with a Cisco Firepower1010 running ASA 9.16(2)3. It performed both tests, Test1 and Test2 at full line rate (930mbps). This is a direct swap using the same cables, same test hosts, same WAN link, same background noise traffic.

The MX85 is not performing to spec on 16.12.

[ww]

I get around 800 Mbps with multiple streams. single stream is not good either

i reported issue on LAN-WAN, but LAN to LAN has the same behavior.

[cmr]

@Adam2104 The 16. release is considered a beta release (though the only option for newer appliances I think).  There have been some performance issues reported and fixed between versions, perhaps this is another?

I have an MX65 running 16.12 with the enterprise license and get the full 250Mbps on tests so clearly some devices are performing to spec on 16.12.

Edited to remove dumb questions that were already answered...

[Adam2104]

@cmr You are correct, MX 16 seems to be the only supported version at the moment. I obviously recognize that it's the beta release, but I don't have any other options. This performance delta is not called out as a known issue. If it is, it should be listed.

[Adam2104]

@ww Are you using VLANs in your configuration? I've adjusted my test setup and moved my MX85 up to my test bench. I also moved it to a fresh MX network so it has a near default configuration on it. My results are:

No VLANs - test client connected to port 5 - 960mbps combined upload/download throughput. So this is pretty much near line rate, which is the expected performance.

5 VLANs (1, 10, 20, 30, 40) - test client connected to port 5, access port, VLAN 1, consistently gets a combined throughput of ~860ish mbps.

The behavior is repeatable. The weird thing about it, the extra VLANs, 10, 20, 30, 40, don't have any active clients, or, any active ports. They're simply configured. VLAN1 is the only VLAN that's actually in use because that's the only client connected. In either test scenario the test client is just an access port in VLAN1. It's unclear to me why just the presence of other VLANs (everything else is MX network defaults) would drop the performance so severely. The MX67 this MX85 is going to replace had no issues hitting the 450mbps rated max, even with VLANs enabled.

[Adam2104]

I'm going to reply to my own thread again just to keep documentation of the testing I'm doing. With the MX85 out of as my main router, I've now got this topology:

Google Fiber (1gbps) --- FP1010 (asa mode version 9.16(2)3) --- various MS120-8 switches --- MX85 (bridge mode) --- test client

In this setup, the MX85 is in bridge mode. The rest of the MX network settings are default. I'm doing a simple speedtest using fast.com. The results are all over the map. Some as low as 250mbps down, to as high as 650-700mbps. Nothing that shows a 1gbps. If I disconnect both cables from the MX85 and connect them together using an RJ45 coupler (eww), I immediately get 1gbps download on the next test on fast.com.

I'm not sure what else to even look at. I'll get to a point where I think I've narrowed down a set of consistent test results / conditions only to have the results change dramatically on the next test.

Another oddity is that I've been getting a warning in the dashboard that the MX isn't connected using TCP/443 for its tunnel to Meraki when attempting a code upgrade (16.8 -> 16.12). I never saw that on the MX67, ever. There is no firewall blocking it. Maybe there's a process on the MX85 that keeps crashing / restarting causing unnecessary processor load that would explain the wonky performance.

[cmr]

@Adam2104 I'm guessing that when you enable VLANs the MX changes from switched mode in the hardware to routed mode in software, but this is purely a guess.  Years ago when I worked for a network vendor in design and test we had a switch that used store and forward most of the time, but if all active ports were at the same speed and duplex then it used cut-through.  The latency was dramatically less and the throughput higher.  As it was a distribution switch it was quite common for customers to be able to take advantage of this mode.

[Adam2104]

For the sake of thoroughness I ran some additional numbers on the passthrough test case.

Test client: 2018 Macbook Pro, MacOS 11.6, Safari 15.0.

I am testing using fast.com My fast.com settings are:

Connections: 8 min, 16 max

Duration: 15 sec min, 30 sec max

Test Case 1:

test client -- MX85 (passthrough) -- MS120-8LP -- MS120-8 -- MS120-8LP -- Cisco Firepower 1010 ASA Mode 9.16(2)3 -- Google Fiber -- Fast.com

Results - download / upload:

630/770
390/780
640/790
620/790
560/780
510/740
520/730
680/710

All numbers in mbps.

Test Case 2:

test client -- RJ45 coupler -- MS120-8LP -- MS120-8 -- MS120-8LP -- Cisco Firepower 1010 ASA Mode 9.16(2)3 -- Google Fiber -- Fast.com

Results - download / upload:

1.1gbps/710
1.1gbps/740
990/770
1.0gbps/730
1.0gbps/790
1.0gbps/730
990/720
1.0gbps/750

All numbers in mbps unless otherwise indicated.

Interestingly, upload speeds seem to be fairly consistent. Download speeds are not. I've tested this on 16.8 and 16.12. The results are the same. No other clients are connected to the MX85 except this test client.

[Adam2104]

This appears to have been resolved as of 16.13. My MX85 is performing at least on par with expectations at this point. 

[NateF]

Hi Adam2104, you might have already mentioned this but did you whitelist the hosts you are testing with or use a group policy on the MX to bypass any firewall, URL filtering or AMP and IDS/IPS services ? those would need to be disabled as they affect the overall performance of the MX.

[Adam2104]

@NateF I do not have any of those features enabled. The MX is attached a completely new MX network, the config is default. There's no firewall rules, no IPS, no AMP, nothing. Just raw L3/L4 firewalling.

[NateF]

Ok that is strange then, I only have an MX84 or MX100 to compare to or I would try and help you more.... but good to know you got it working on a FP1010 so must be the MX85 for sure, good luck with everything !

[Dunky]

I'm looking to purchase an MX85 in the very near future - for the very reason that it is supposed to support the 1Gbps internet connection we have.

Have you had a chance to test it on v14?

[Adam2104]

@Dunky It's not supported on v14. v16 the only supported train at the moment.

[Dunky]

Aha, useful to know, thanks @Adam2104