Upgrade to 17.10.2 | Connectivity Failures

[NWNSM1]

After our MX450 updated the firmware to 17.10.2 overnight on Thursday, we had connectivity failures.  Slow loading or web pages not loading at all.  Some services such as our AD Sync to Google said we have no connectivity and failed.  This was across all OS platforms and for 2100 users.  

It wasn't something we could troubleshoot at the time and we rolled back to 16.16 and everything returned to normal.  Does anyone have any thoughts or know where to begin?  I have any future upgrades paused on the firewall for 6 weeks but would like to be prepared for the future.

[Inderdeep]

@NWNSM1 : Not sure it is stable version yet. Did anyone else test ?

[cmr]

We run it on all sites with models up to MX250, but we only use the Enterprise feature set.

[PhilipDAth]

I think 17.x changes across to using snort V3 and content filtering based on Talos.

Are you using content filtering or IPS?

If you are using IPS, trying upgrading to 17.x, and turn IPS off and back on again.

If you are using content filtering - check the new categories (they have changed) after you upgrade.

[Nomad]

I can confirm this issue. Since some MX-Devices (With Adv Security License) upgraded to 17.10.2, all Clients (PC, Laptop, iPads, Smartphones) which are connected to the MX have performance issues with several websites including sites like Amazon, Ebay etc.

Only after removing all of the threat categories which we had applied before, the performance went well immediately.

Deactivating IPS wasn't necessary, in my case it seems to depend only on the categories, which I'm going to activate again one by one to see which of them exactly causing the issue. The problem is that even whitelisting some of the sites didn't worked..

[Bonzo]

I am receiving the same issue with the slow performance on websites. Can't really downgrade during production, can I?

Has Cisco Meraki said anything on this issue?  

[Bonzo]

I tried to remove all of the content filtering and it still didn't make a difference. Trying to disable IPS and reenable. I will let you'll know if this made a difference. 

[Adub91]

Please do. I am also having the same issue.

[EngageTechNZ]

I can confirm we have the same issue on one of our sites. Removing all the content filtering categories fixes the issue. IPS on or off makes no difference.

[Bonzo]

Tried this for my site yesterday, didn't make much difference. The only thing that fixed my issue was downgrading to 16.x

[NWNSM1]

I only have Spam URLs, Malware Sites, and Phishing & Other Frauds enabled,  We have a more granular content filter with reporting inline as well.  

[RayDz]

I see several threads in the forums on this issue now. But yes. We were bit this past weekend and unfortunately coincided with an internet service upgrade, and CPE swap, which after banging our heads collectively on our desks turned out to be red herrings. 

We are on MX250's and were at 16.16, which updated to the 17.10. last Sunday evening. 

We were seeing pages not loading and "err_timeout" messages, but then hitting reload would work. Also, pages or sites used widely in the company worked fine. Like, Office 365 pages would load. But a site that had not been previously visited would then not load or it would timeout. But then if you reloaded, or revisited the site, it would work for a time. 

We stayed late last night with SD-WAN provider engineer, as we thought it may be related, and were unable to reproduce the error at all over a 90 minute period. The difference is that it was after hours, and there was hardly any traffic on the internet connection. The next morning as people returned to the office, the problem returned. We rolled back the firmware at that time and service was restored.

My gut says this appears to be some sort of inspection or cache component failure. 

[Nomad]

Updates: The new MX 17.10.4 update (Released Feb 28, 2023) fixed my issues with MX64W. I selected ALL content and threat categories for testing and everything still runs fast and stable as it should be. Feel free to try it on other MX models.

Fixed an issue that resulted in wireless clients experiencing degraded performance for HTTP and HTTPS traffic when content filtering was enabled.

Reduced the amount of time before a URL classification request performed by the content filtering service would be considered to have failed. This may resolve latency caused by content filtering in cases when classification requests failed and needed to be retried on a consistent basis.

Best regards

[SimonReach]

Upgraded to 17.10.2 on Friday morning and had people reporting slow and problematic internet since Monday, upgraded to 17.10.4 this morning and people are still reporting the issues.

Internet loading slowly on some pages and other pages time out first time and then work fine after a quick refresh.

We've got MX100s onsite and use AMP and the Intrusion Detection and Prevention in Prevention mode and Ruleset of Security.  We also use url filtering and category blocking as well.  Might look at seeing if we can roll back to 16.x tonight if no one else has any ideas?

[Nomad]

We have the same constellation on a MX95 and have had no problems. The only difference I see to you is that we first updated to 17.10.x and then activated all the security features. Maybe you should remove and re-add everything. It's a bit of work but worth a try.

[SimonReach]

Turned off all security and left it for 5 minutes, turned on the most important bits again and just giving it a try, will turn on the rest later when we know if there's been any difference.  Thank you

edit: No difference at all, we're rolling back to 16.16.8 tonight.

[SimonReach]

Just in addition to the internet issues we're having.

Since Monday, we did the upgrades on 2 of our sites which connect via SDWAN to each other over the weekend, we've also had multiple issues with people getting corrupted Excel files and connectivity to external servers and ftp sites has also been affected.  

Last time we attempted to upgrade to version 17.x a month ago, we had major issues with the VPN so something isn't right with the 17.x updates in regards to our systems.  The VPN issue we had was that the client vpn lost it's DNS name, luckily we had another dns name registered with our DNS supplier that worked fine still but still had to get every vpn client moved over.