Slow browsing after upgrading MXes to 17.10.2

[fudomyoo]

Anyone else experiencing very slow web browsing after upgrading your MXes to firmware 17.10.2? I know that content filtering was switched to Talos in the upgrade. I suspect that has something to do with it, some aggressive content filtering going on. Has anyone heard from Meraki or come up with some workarounds for this?

[alemabrahao · Accepted solution]

I and other members of the community had the same problem, in this case my suggestion is to downgarde to one of the 16.16.x versions.

[fudomyoo]

This is what we ended up doing. I'd rather have some content filtering on v16 code than have it completely disabled to make v17 work.

[RaphaelL]

This might be the 100th post about slow browsing speed with MX 17.10.2 despite the version being Stable.

Still no official comment from Meraki and nothing listed as known issues in the changelog is very worrying. 

[fudomyoo]

I agree. This is a bit ludicrous. I'm shocked that this issue wasn't noticed in any sort of testing by Meraki.

[BrianMorris]

I'm wasting hours of my time trying to figure out why my clients are having wifi issues and now I find all these posts. They released 17.10.2 anyway..... Sigh. (MX64W: LAN is fine, wifi is trash.)

[ww]

I had a lot of issues with content filter: advertisement.

There were also reports of slow browsing on mx build in wifi.

[Oderbang]

Hey Fucomyoo,

I have a case open with Meraki regarding this..

It is the exact same experience we had with version 15 and 16 of firmware when you enabled the "Full list" option in content filtering.

In version 17 there is no option to select "Top list" or "Full List" since they moved from "Bright cloud" to "Cisco Talos Intelligence" so my first thought is that they are now using the process for "Full list" as standard.

This heavily leans on the MX cache to speed things up (we are running MX 250) I'm not sure how long the cache TTL is (how long a record stays in the cache) or how large the cache is (how many records are kept until they are cycled in for newer results)

regardless of this though.. its the initial lookup of the site that is slow.. so cache wont make a difference until after the first lookup.

As we speak the Meraki support agent just captured a "slow" site lookup... ill keep you posted.

[Oderbang]

OK so they think they may have possibly found a bug... needs confirming BUT this is what it looks like from a flow caputre.

 We attempted to access a website not in the cache (my local gym of all things)
The category look up completed in 27ms (great nice a n quick) however the MX seems to.... ignore the result... waits about 1 second and tries again... it loops like this for about 10 seconds until it finally accepts the category response and allows the connection.

so it does not look like a cloud response time issue.. and more of a MX firmware bug... ill keep you posted when I get an update

[fudomyoo]

Thanks! You are having more luck with support than I am!

We just ended up rolling back to the v16 code. Thanks for following up!

[combobulated]

We are experiencing this also after upgrading to v17.10.2.  As a result we're rolling back to v16.x for now.  I saw that there is a v18 Release Candidate available that offers "performance improvements" for our MX appliances, but in the bug notes it also states the potential for instability.  That's a hard no for us.

[BrianMorris]

I upgraded to v18.105 on one of my problematic MX67W models and it didn't help with the wifi issue one bit. I did put an Omada WAP on it and it worked fine though.... On another site I'm rolling back to v16 tonight.

[combobulated]

Thanks Brian.  I checked the release notes for v18.05 again and it said:

Performance improvements for MX250 and MX450 appliances.

I wish this was more specific.

BTW we've just rolled back to v16 and the problem disappeared.

[lpopejoy]

We are also seeing this, but it seems to only be on MX64W (w/ wireless).

[fudomyoo]

Glad it is limited to only that model for you. Unfortunately we saw the issue on everything we upgraded. We have a variety of the MXes from MX67 to MX450.

[m_prime]

We were also seeing this issue after installing brand new MX250s in our environment. Some websites would be stuck at a white screen for more than 30 seconds on the initial load. After 30 seconds it would then go to a connection reset screen (which is the screen you would see if it was blocked) and then load the website eventually after about a minute. We upgraded the firmware to the latest (18.106) and we still had the issue. The fix for us was to combine the firewall and switch networks and turn on Unique client identifier under Security & SD-WAN > Deployment Settings > Client Tracking.

[Nomad]

Updates: The new MX 17.10.4 update (Released Feb 28, 2023) fixed my issues with MX64W. I selected ALL content and threat categories for testing and everything still runs fast and stable as it should be. Feel free to try it on other MX models.

Fixed an issue that resulted in wireless clients experiencing degraded performance for HTTP and HTTPS traffic when content filtering was enabled.

Reduced the amount of time before a URL classification request performed by the content filtering service would be considered to have failed. This may resolve latency caused by content filtering in cases when classification requests failed and needed to be retried on a consistent basis.

Best regards

[BrianMorris]

Thanks for letting us know. I'm currently testing at my own office and it appears to be ok as well.

[SteMeDi]

Does anybody know when this issue will be fixed in MX version 18?
With 18.106 I am facing the slow browsing experience, especially when the categories "Malware Sites" and "Malicious Sites" are configured. I don't want to go back to version 17.10.4

Best regards

[Oderbang]

OK, So We are still experiencing slow website lookup and external service connection timeouts on v18.106 and had to roll back AGAIN! 

I have been informed by Meraki support that in our case, we have clients behind a non Meraki L3 device (our VPN) 

and are using content filtering, and out client tracking is set to MAC address. (we use combined network dashboards.. like most users)

Meraki support has provided us a "solution" (that I don't agree with but get to that later) which is to split the network dashboard and set client tracking to IP address.

If you want to give this a go you can do this by going to Organisation -> Overview  select the network where your clients are behind a non Meraki L3 device then select "Split network" 

This reduces manageability so please research this before trying it!

So why is this not really acceptable.. well we have been operating with content filtering enabled with our non Meraki L3 VPN for many many years... no issues... so why all of a sudden do we have to loose all this functionality due to a "Upgraded" content filter??? 

[SteMeDi]

With version 18.106 the problems still existed. As of 18.107 the error is fixed (see changelog). We have tested it successfully and upgraded all our MXes to this version.

[Oderbang]

apologies it was v18.107 that we tried and had to roll back from... Were your clients behind a L3 device ?? and what tracking method are you using ?

[CyberDingo]

Commenting so I can follow this post. We are experiencing the same problem. Thank you for researching it with Meraki. 

We have one site that that has a L3 Switch (which is Meraki brand) within the LAN that processes routing that is having this problem with the Meraki. All our other sites are not set-up like this and are not experiencing this issue of slow web sites loading when a user goes to a new web site (a URL that is not yet cached).

As of version 18.107.2 we are still experiencing this problem and we had to roll back to v16.16.9

[JonP]

We finally solved this after a combination of upgrading to MX 18.107.2 and changing to "Unique client tracking", but your milage may vary.

[CyberDingo]

Hey @JonP, glad you found a way to get it working for your team. Looking at the documentation for it, it does look like "Unique Client Identifier" fits our situation:

The problem I have is that on all my other sites that are functioning that have been upgraded to v18; they do not have "Unique Client Identifier" as an option. They are of similar model as the problem site (MX100):

[JonP]

Yeah if you don't have a full Meraki network it's going to be different for you. Meraki basically shrugged their shoulders when presented with the problem, and after pushing them for a resolution they suggested this. I don't recall the exact reason as they mentioned it when they were on the phone with my colleague @CameronS .

[Oderbang]

You are in the same position as we where, except our L3 device was our VPN. Had to split the network and set it to Track by IP

Immediately resolved the issue.. but now the dashboard is a PITA to use... we are planning on going global so this is FAR from ideal.

An Meraki built in client VPN is trash..no certificate support cannot do Device VPN or always on VPN.. so that even less of an option... Meraki do seem to be pushing "features" rather than stability... 

[CyberDingo]

@Oderbang We were able to get it working by updating and switching Client tracking to "IP address." Thank you for your work on this and sharing the fix with the community. 

[combobulated]

Thank you CyberDingo.  Have you experienced any downsides from switching to Client tracking by IP address?

[CyberDingo]

Not yet. But to be honest, we changed it yesterday, so it has not been a full work day yet to see any hiccups. 

[combobulated]

Thank you.  We may consider doing this pending your experiences with it.

[Oderbang]

apart from no longer being able to use the combined dashboard.. so everything needs to be configured in 3 places instead of 1... we also noted that since switching to IP the host name of devices are less likely to be identified... and there is no way to add your internal DNS server to Meraki to be able to look it up.

we have not seen any "technical" issues... but its just frustrating UI issues

[JonP]

We're seeing internet slowness, but we thought this was an ISP issue. This issue started a few weeks ago but seems to be slowly getting worse. As @fudomyoo says this is the same issue we experienced when we turned on "all sites" filtering in the prior version.

[cmr]

The new 17.10.9 patch might fix this for you:

New MX 17.10.9 stable firmware release - IDS/IPS CPU utilization fix! (and ... - The Meraki Communit...