Performance Issue with MX250

[ShoaibAbbad]

Hi All,

We are facing an unusual issue with our MX250. When business hours are about to start and the number of clients behind the MX250 increases, there is a sudden spike in device utilization, which leads to latency affecting all traffic passing through the MX. This latency impacts both WAN links and LAN-to-MX traffic. As a result, we also experience increased latency over Auto VPN traffic.

Everything was functioning normally until mid-September. To stabilize device utilization, we rebooted the MX250, which temporarily resolved the issue for about three days. Initially, we were running MX 18.211.2, and TAC recommended upgrading to MX 18.211.3. The upgrade was stable for two days, but on the third day, latency increased again due to high device utilization. TAC then suggested upgrading to the MX 19.1.4 Beta version, but the issue persists.

All necessary logs have been captured, but no solution has been found for this behavior. We are seeing traffic of only 100-150 Mbps when the issue occurs, and after a reboot, device utilization returns to normal. We’ve even swapped the primary MX250 with a spare unit, but the behavior remains unchanged.

We tried disabling IPS/IDS, AMP and the behavior remains same.

If anyone has encountered a similar issue with any of your MX devices, could you please share any fixes or suggestions?

[RaphaelL]

Hi ,

2 questions

1- Number of "clients" ?

2- Have you tried MX 18.107.10

You can also ask support to show you the advanced metrics. CPU 99% percentile , number of flows , packets per second. 

It looks like that : 

I have almost the same issues as described with many MX68. We are exceeding the recommended limit of 50 clients ( but who cares ). With 18.2XX code and 19.X we see a big bump is device utilization.

[RaphaelL]

I don't have a clear solution to that , but I made a post couple weeks ago with all sort of info : https://community.meraki.com/t5/Security-SD-WAN/MX-Device-utilization-degradation-after-upgrading-to... It looks similar to your issues.

[ShoaibAbbad]

This is the device Utilization from Sept 15. I had requested TAC for checking CPU Utilization and they confirmed there is spike in CPU Utilization but i didn't get any report.

Average client count is under 1000 and some times it increases to 1890. The recommended client count for MX250 is 2000.

When we were running in 18.107.10 the device was much stable. After the upgrade there is in increase in device utilization. I will check with tac if we can test by downgrading the MX to 18.107.10

[RaphaelL]

Do you have the graph to compare before ( MX 18.107 ) and after ( MX 18.211 / 19.1X ) ?

The only thing different from my case is that when we disable or whitelist 10/8 from the IPS we see a huge improvement. Device utilization decreases about 10-40% depending on the type of site.

[ShoaibAbbad]

Just verified the graph. It was normal till Sept 14. We upgraded to MX 18.211.2 in month of July 2024. It was fine till Sept 14 2024.

We have whitelisted around 20 rules in Intrusion detection and prevention as the trusted traffic in AUTO VPN was getting blocked.

IDS Mode is Prevention and Ruleset is balanced.

[RaphaelL]

I see 3 suggestions. 

MX 18.107.10 , MX 18.211.4 , MX 19.1.5 ( latest Stable RC )

I would hope that going back to 18.107.10 would "fix" your issue.

[ShoaibAbbad]

Ok I will downgrade to one of these firmware and check if it helps.

[RWelch]

MX 18.211.4 improved high device utilization for one of my many MX75s.  At another location, one MX105 seems to perform better during peak utilization periods (active concurrent sessions).

[ShoaibAbbad]

Sure. I will check this firmware if it helps.

[RWelch]

A new stable appliance firmware is now available on Tue, 29 Oct 2024 

[Inderdeep]

I would recommend to upgrade the image 

[ShoaibAbbad]

I had done the upgrade twice. Still the problem exists. Will check with any other version

[Smula]

We started having the same issue which also started back in September.  Meraki has sent is another 250 and when that didn't work either, they sent us a 450 and we had the same issue with that.  They said it's a known issue but the 250 and the 450 run the same firmware so I'm not sure why they thought that was an option.

In any event, they have admitted it being a known issue and and have also said they currently do not have a fix for it as of November 13, 2024.  

I think our issue started on Sept 5 with that firmware upgrade.  Hasn't worked since.  

[RaphaelL]

Wow !   This is a spoke right ?

[ShoaibAbbad]

It's Mesh.

[ShoaibAbbad]

Whats the workbaround provided? Did you try to downgrade with the older version and test?

[Smula]

The workaround is reinstalling our old edge router. Thankfully we didn’t get rid of it. Meraki support stated they downgrade past a certain point. If memory serves we reverted to one version maybe two. Neither did the trick. It’s a flat network no VLans. Runs seemingly fine until there’s a load placed on it. Then it’s dropped packed and sever latency. Mess started around September but there wasn’t a load on it before that so I’m really not sure when the issue started. All summer we had maybe 30 staff. September we had 1600 or so and that’s when it was over. 

[ShoaibAbbad]

Understood 

[Smula]

Correction: they can’t downgrade past a certain point. Meaning we went back one or two versions but that was all we were able to. It’s a known issue and been going on for two months. Now they are refusing to generate an RMA because they want to continue troubleshooting. Obviously not an option because I’m not sure how they can do that if the device isn’t in a production environment. Complete debacle. Just warning you guys when you go down this road of the pitfalls.  I asked for the firmware it was shipped with because it worked fine for a several months. But there’s apparently no documentation as to what that version was. Even if they knew, they said they can’t install it.  Frustrating. Just don’t go chasing your network looking for something that’s not there. We spent weeks doing that. The 250/450 are the issue. Like i mentioned we now have a pile of three useless devices with no end in sight. 

[ShoaibAbbad]

I don't have option of even replacing the device with edge router as we are completely dependent on auto VPN for our intranet connectivity and the impacted device is on a major site with around 1500 to 2000 users.

[ShoaibAbbad · Accepted solution]

Hi,

I have good news! The issue has been resolved.

When the problem reoccurred last Monday, I contacted Meraki TAC. After a thorough investigation, TAC discovered that a group policy was impacting CPU utilization. Specifically, we had configured a 10Mbps bandwidth limit and applied it to a particular client. Whenever traffic was high on that client, the group policy was actively limiting the bandwidth, which caused significant strain on CPU utilization and resulted in latency issues.

After removing the group policy for that specific client, the issue has been resolved.

[CarolineS]

Hi @ShoaibAbbad - that’s great news! I’m going to mark your reply as the solution. (You can un-mark it if needed). Cheers!