MX Throughput - Specifically

[Nick]

Hello,

We have been seeing some inconsistencies in Meraki documentation and real world usage on our MX85's at customer sites. With the unit failing to achieve the 750 Mbps

Can anyone confirm the correct MX85 figures?

We have been working towards and installing MX85 units based off this throughput capability

https://meraki.cisco.com/product-collateral/mx-family-datasheet/?file

However we have just seen this documentation which has been flagged to us by a customer we have suggested the MX85 unit to.

https://meraki.cisco.com/product-collateral/mx-sizing-guide/?file

Which document is correct?

Can someone from Meraki confirm what the throughput figures are on the MX85 unit?

- Stateful firewall throughput

- Advanced security throughput

I have seen the recent welcomed changes to the MX67 throughput, but it seems to us that the MX85 has lost a third of its throughput.

This would be a big issue for us and customers who have based their MX choice on this throughput calculations, only to have it reduced. Especially as I cannot see anywhere where this reduction has been communicated.

I am hoping this is just a wrong datasheet and environmental issues at customers sites.

Thanks

[alemabrahao]

Do you have any traffic shaping rule configured?

[Nick]

Nothing specific on the units that would be limiting the throughput

[CptnCrnch]

Based on my experience, you should stick to the second document -> Sizing Guide. This one is much more accurate. Especially, take a look at which feature in use impacts performance the most.

Btw.: 750 MBit for a MX85 isn't really that bad.

[Nick]

Hello,

That is the issue - we are not seeing 750 MBit

The device was originally advertised at 750 MBit but now they are saying 500 MBit. This is a big change from what was advertised initially. 

[Frank-NL]

In my opinion the Advanced security throughput and all security features enabled throughput mean the same thing

[jbright]

First thing to test is a computer directly connected to the internet connection and see what the speed tests show on it. Maybe the ISP is fibbing on the actual bandwidth they are providing? Of course that never happens 😉

[Nick]

Thanks @jbright this isn't environmental or ISP related. We are seeing this across all of the MX85 units deployed. They've been neutered 

[PhilipDAth]

I just tested an MX75 with everything turned on, and it also got 500Mb/s.  There are no speed limits configured, but it was running so close to 500Mb/s it felt like it was being software limited ...

[Nick]

Hi @PhilipDAth 

I agree - it feels like the units are being held back now. 

The big problem with this that the units were advertised as 750Mb/s and sold as that.

Now silently it seems unless I am missing something Meraki have moved the goal posts. Customers are aware of this and we already have one who is asking why their device has been limited post sale and asking for a replacement MX95 to meet their requirements. 

Anyone from Meraki able to shed some light on this? 

[Nick]

So we have been unable to gather any more information on this so far from technical support saying talk to or our local reps. Who don't know and it loops round to technical as if its a localised issue.

We are so far just getting "figures can be variable based on traffic" coupled with "MX advances mean things could be slower"

Has anyone seen or know anything further? Meraki employees? 

It feels like the devices are being limited in our testing as people like @PhilipDAth have also seen.

We haven't been told anything and this is starting to get embarrassing for us as Cisco partners with customers and Meraki's reputation with the customers. We have two sets of documentation with differing throughput now.

We have customers who have kit that has dropped 33.3% of its throughput and we have ongoing prospects we can't advance as we cannot get a straight answer. 

Can someone help please?

[DevOps_RC]

I'm sure you have already checked this, but when your customers have complained that it has gone slower, was it close to a firmware update? Just wondering if there was a particular 'fix' included in a firmware which 'inadvertently' restricted throughput of devices.....If it was, could you try reverting to a previous firmware to prove it? Have you have tested it with the advanced security features off to see if you get the 1Gb throughput?

[Nick]

Thank you - we did check this, nothing recent or changes that we think caused this.

We are getting the full 1Gb throughput with Threat Protection disabled.

Our MX firmware tests were done on MX 16.16.8 and 17.10.2 

@PhilipDAth what version was your MX75 on?

[Nick]

Hello all,

Has anyone from Meraki seen this and able to comment on the issue? Anyone on the MX product team @AI007 

Two sets of documentation that give different figures?

We're struggling to get any answers for this and it is reflecting badly on us for clients

[alemabrahao]

Have you opened a support case?

[Nick]

Hello,

Yes - we opened a case and were sent the two contradictory pieces of documentation then told to speak with the account manager. We've been at that stage but nobody can answer the issue or confirm what is going on

[GiacomoS]

Hey team,

I'm looking into this and trying to understand where it is all coming from. 

Please allow me some time to find out more. 

Just for my confirmation, I see a lot of "it dropped down to..." , so are you observing a change from the previous attainable speeds to the current cap of 750? Are you actually able to reach 750 on the MX85 or is it only up to 500? 

Did you notice at which point the effective throughput may have changed?

Many thanks!

Giac

[PhilipDAth]

I just re-tested an MX75 running on 18.107 via a wired connection direct to the MX.

When I test against an IPv4 server, I get around 500Mb/s.

When I test against an IPv6 server, I get around 750Mb/s.

I'm using a Gigabit fibre connection using DHCP for IP assignment with a 1500-byte clean MTU.

I have "everything" turned on.

[Nick]

Hello @GiacomoS 

Great thank you!

If you're able to confirm why the sizing guide differs from the datasheet now that would be great!

Yes - we're seeing a flat 500Mb/s limit on the MX85's we tested. Prior to this we were seeing more. As @PhilipDAth  said it seems like the box is being limited. 

We noticed when a customer flagged to us the sizing guide had changed from 750Mb/s down to 500 Mb/s yet the datasheet has not

Many Thanks!

[GiacomoS]

Hey team,

I'll keep this short and sweet: the expected throughput is around 500Mbps. 

I have asked the incorrect documentation to be updated. 

Please accept my apologies for the confusion, and thank you for bringing this to my attention!

Giac

[PhilipDAth]

That was brave.  Thanks for setting that straight.

[Nick]

Hello @GiacomoS 

Thank you for looking into this and coming back so quickly.

Bad and rather disappointing news! 😟 

Both for the MX lineup and the MX75 and MX85 products specifically. It also it going to make an issue for us with existing clients who purchased the MX85 unit from us with the figures and throughput previously quoted.

Why has this change been made? The units have been out for nearly 2 years at this point and the figures and throughput have always been 750 Mbps?

Is the unit being limited to 500 Mbps? Is this a product decision to do this? As it seems the hardware is capable and up until recently was performing at 750 Mbps (this is going to be another sticky point with customers)

What shall we communicate to existing customers? Clients purchased the unit based on the specifications listed and while some variances are to be expected a 33% reduction in throughput is way too much and a significant change.

We have clients who are ready to increase the bandwidth from 500 to 1000 and purchased the unit specifically as it could cope with more than 500. It is one of these clients who initially raised the issue. They want an explanation and if presented with the above are going to feel miss sold. 

As a partner and Meraki only house I think we really should have been notified about this. It has made us look silly and negatively impacted clients perception of Meraki by having differing numbers on documentation and the delay in getting an answer.

As for the MX lineup itself, in our region to have the unit limited to 500 Mbps commercially puts us at a disadvantage against alternative vendors given that the MX95 is double the cost. It also doesn't sit well with those figures agains the MX67 line up given the cost differences. Commercially it just doesn't stack up as well.

I'm very keen to know thoughts and answers to the above.

We have four deals that are currently paused waiting on the confirmation of the MX85's abilities. Sadly switching to an MX95 is going to move us away from being viable cost wise on a couple of these deals. 

[GiacomoS]

Hey @Nick , 
I'm truly sorry to hear about the impact here. 😞 I am having some conversations internally to try and find out more, but I wanted to give you all a speedy response, as I understand this had been going on for a while. 

Please bear with me whilst I get some more input, as I would prefer for you to have solid answers rather than just my opinion. 

Thank you for your patience here!

Giac

[cmr]

Thanks @GiacomoS and I know you aren't the decision maker here, but I do think this needs rethinking as significantly downgrading a product would need a very good explanation!

[Justin_Blignaut]

As a fellow Meraki partner, I really understand @Nick frustration.

We design, and implement solutions based on the knowledge gained from the official documentation. We as architects and partners work with our clients to make selected solution choices, to meet their current and future requirements this to, fit price points, address complexity, etc.. We do all this while evangelize Meraki simplicity. 

I understand that products evolve and that Meraki is all about the platform and not the "speeds and feeds" of the boxes. But, the performance characteristic of the boxes are still a significant input into the overall design considerations and longevity of the solutions we create. In most cases, these solutions are consumed for extended periods of time. Designing solution using the biggest box available such as the MX250/450 "just to be sure" that we will not run into these type of problems, is a none starter. It detracts from what Meraki is. This approach leads to competing vendor technology being better priced, better scaled with referenceable specifications which gives the partner and clients peace of mind.   

With a significant drop in the MX device performance being noted across multiple community threads affecting a range of products, I agree with @cmr I really hope that Meraki has a very good explanation and a way forward for impacted clients.

At the very least publish up-to-date and accurate performance specifications, including actual throughput at the tested software release, and the numbers of actual supported TCP/UDP flows across the device. The Meraki Support standard answer that you need to "buy a bigger box" just inflames the issue more.      

Ok rant over 🙂

[GiacomoS]

Hey everyone,

Thank you so much for all your feedback on this thread. I appreciate the impact and everyone's input. 

I unfortunately don't have much of an update to share on what has happened, but I'd like to share some guidance to move forward. 

The first recommendation I have, is to leverage the sizing guide for design decisions. The document is built to guide the decision making process on which MX would be the best one for your networks and is the one that is kept most up to date. 

The second recommendation is that there are a lot of commercial elements in this conversation; I think this should be a conversation that you expand on with your Account manager and their team, as they will be able to support you in navigating the decision making around the best product based on the needs of your network and customer.

For customers with existing deployments, I would suggest to have both a Support case open and also discuss with your Account manager, so that we can all coordinate on the best approach. @Nick , I think you may already have a case open and there was a bit of confusion on it. Please do feel free to DM me the case number, I'd like to see if I can assist steer it back in the right direction. 

@Justin_Blignaut , I hear you. At times, Support does not effectively have any other way to guide you forward when the data indicates that the device is reaching its maximum capacity, so the only recommendation that can be offered is to buy a bigger MX; I know that it can be quite frustrating and I'll take the feedback back internally to hopefully improve the process for this type of scenario. 

Thank you everyone.

Giac

[DevOps_RC]

Good Morning @GiacomoS. Thank you for the update and I appreciate that either you may not have all the information available to you currently or that you may not be allowed to release all the information. Hopefully further insights can be provided into why, what and when a change was made to 'restrict' the MX85 throughput from 750Mb down to 500Mb with all features turned on? I am curious though as to the testing feedback that @PhilipDAth stated, which seemed to indicate that if the resource being accessed is IPv6 then the MX85 will provide 750Mb but if it's an IPv4 it drops to 500Mb. It almost seems that there is a feature/function that restricts IPv4 which doesn't apply on IPv6? Are all features/functions supported on both IPv4 and IPv6. I'm thankfully not affected by this change, as I currently only have MX450s to support, but if there has been a change, of which there does see to have been, any restrictions/reductions in performance should be included in release notes or notifications in the dashboard (Sorry not meant to sound like a rant, just want to make sure this isn't another change similar to Apple battery/performance change in the background for the 'benefit' of the user). Please continue to look into this, as I'm sure all those who have taken part or read this thread are curious as to the cause.

[cmr]

@DevOps_RC the IPv6 feature set is much more basic at the moment, partly due to features still in the works and partly due to the intrinsically more simple nature of the protocol.

[spadefist]

All,

The MX sizing guide is the only document you should be using for scoping an MX, as we haven't traditionally updated the data sheet numbers post release as the results in the data sheet are only ever optimal single use testing data (RFC2544 using 1024 packets) which quite frankly are no use to anyone, however we as an industry only ever highlight data for such optimal use cases; FYI we as a PM org hate having to do this but if we don't someone who doesn't understand networking will point at vendor X and say the number is bigger and hence Meraki is worse :-(.

If you need any context on why the numbers have changed (e.g. MX75 Sec throughput) please reach out to your aligned SE/TSA and they can both explain the situation and provide options, unfortunately I can't provide the answer here for fear of it being used out of context.

Hope this clears up the situation and gives folks a path forward.

Steve

[Nick]

Hello @GiacomoS  and @spadefist (great name)

Thank you for this.

However we did follow all of those guidelines when specifying the correct model and have still ended up in this situation.

The sizing guideline did up until recently have the figures of the MX75/85 set at 750Mbps, it was recently refreshed to 500Mbps.

I did open a support case initially about this and was told to talk to the account manager. I spoke with the account manager who didn't know about this and still hasn't been able to procure an answer.

So from our side this process seems broken. We've followed the guidelines and it feels like the goalposts have been moved. Then that we're not being told or getting any information on the changes.

What is worse is that it feels like the units are being artificially held back and capped. We're sad as the MX75/85 product line performance change devalues these machines significantly.

It leaves us back in a similar situation we were in before these products were announced where there is a gap in the MX lineup to cater for environments with above 500 Mbps who don't or can't justify the cost of a unit like the MX95. This is an area where competitors don't have those sort of gaps in this market. I find it odd as this must be a pretty normal requirement as we have dozens of clients who need this sort of unit.

For existing customers we need to address the issue that they bought a unit based on the figured in the sizing sheet that had been out for over a year and now the throughput figures have dropped a third.

For new customer / sales we will also need to address the situation. We had a customer with a signed off order for an MX85 that due to this uncertainty we paused. Given that this has taken so long and with the customer pushing we had to lend them one of our MX100 units to give them access to more bandwidth while we resolved the issue. I'm not sure what the outcome here is. If we go back to the customer and say "sorry you need this next up unit that is double the costs" At best we ourselves will look incompetent, at worst it looks like a money grab.

As above Giacomo I don't have any open tickets about the issue now but am happy to open one and work with you to resolve this. Certainly the AM's we've spoken to don't know about this or anyway forward. I am very keen to get this issue resolved for existing client and clients where we have open or approved orders for the MX85 units