Content Filtering Issues?

[Duke_Nukem]

Anyone else seeing content filtering issues today?  I think it started about an hour and a half ago.  VPN users couldn't hit an internal webserver.  Internal users could.  

Page wouldn't load  -  showed blocked url in the address bar.

Added the internal server's IP and name to the Allowed URLs and it started working.  

The "Check content and threat categories" is failing no matter what URL I put in.

Error fetching categories for url lookup, please contact support

I'll open a ticket.  Curious if others are seeing issues.

[ww]

I also see the 

Error fetching categories for url lookup

But didnt notice problems yet

[rhamersley]

Yes, We had users in our Anaheim, CA VPN location having issues trying to access sites like Verizon.com and other Sites.   Now this did clear up but we had that issue for about 45 min to an hour.   I am curious if Cisco was doing anything on the "BACKEND".    I will be following this thread closely to see if you have any response from your ticket that you open.   

Me

[Duke_Nukem]

Thanks.  

No issues with normal browsing, it was only the internal web server that gave us an issue. The only content filtering we block on the MX's is the Threat categories.  

[rhamersley]

Here is what came back to one user trying to access Verizon.com

[Jameson]

We started having this issue as well about 30 minutes ago. Some internal sites are being blocked. When we attempt to check the category in Meraki, it fails. Our "workaround" has been to add the internal site to the meraki "allow list". 

Our IT Manager is on with Meraki support right now. They are saying there are no issues reported relating to this. We also submitted a ticket.

[BlakeRichardson]

https://status.meraki.net Shows all systems operational 😂

[Jameson]

When checking the network traffic in the background, I can see the following error after attempting to check the URL category. Seems related to something on the Meraki server side and their licensing of Sentry.

detail: "Sentry dropped data due to a quota or internal rate limit being reached. This will not affect your application. See https://docs.sentry.io/product/accounts/quotas/ for more information."


Also, the item before it in the loaded pages is a failed page from meraki:

Server error

The server encountered an error and could not complete your request.

A message has been sent to our engineering and support teams to notify us of the problem, and we will fix it as soon as possible.

The Cisco Meraki Dashboard is temporarily unavailable, but rest assured your Meraki equipment is still running and network operation has not been interrupted. Learn more about Meraki's out-of-band control.

[Jameson]

Meraki Just responded to our ticket:

We are aware of the issue and currently, the Engineering team is investigating the matter. When attempting to use the URL check, it returns a server error 500. Our team is working diligently to resolve this issue. If you would like to check with the category it's in right now, as a workaround Meraki leverages BrightCloud. If you'd like to use a URL checker, BrightCloud has one available - https://www.brightcloud.com/tools/url-ip-lookup.php

[Duke_Nukem]

Support just replied to me as well.  Said "The bug is affecting only the URL checker and should not prevent the content filtering engine from working as intended.".  Which isn't exactly the case.  It's blocking internal web servers (at least when accessing them via the VPN).

[Jameson]

That is our experience as well. We have multiple sites over Site-to-Site VPN as well as remote workers on Client VPNs (multiple firewalls hosting Client VPN). The local sites are blocking some internal URLs. Also, it wasn't all sites, but likely was moving that direction. It started with reports at one, then two, then Client VPNs, then another site.

[BlakeRichardson]

@Ryan_Miles what triggers https://status.meraki.net/ to show there is an issue or is it a manual process?

Not having a go at you but multiple people reporting the same issue which means it's fairly wide spread yet status page shows all systems operational...

This page was only pushed in December last year.

https://community.meraki.com/t5/Community-Announcements/Farewell-Service-Notices-Hello-Meraki-Status...

If your not the right person but know who is the right person are you able to involve them because if the status page isn't reporting issues whats the point of it?

[TNAComputers]

I noticed this yesterday as well. The lookups were failing, but everything else was working for me. Internal webpages were also working over the vpn.

I thought with 18.x code it moved to Talos instead of Brighcloud? (or ealier. I cant remember)

[CarolineS]

Hi @BlakeRichardson - I reached out to the team responsible for the status page, and learned that it is indeed managed manually. Here's a little more info:

At this time, the focus of Status Page is on service disrupting events that impede the ability of customers to operate their networks. The decision to post issues on Status Page does involve a number of factors and we are continuing to evolve our processes for when an issue will be added to the page. We have flagged this particular issue to our product teams for consideration.

They then followed up to say that the issues folks are reporting in this thread with sites being blocked is actually related to the URL checker not working:

We've confirmed most reports of this bug are purely that the URL Checker doesn't work and there's no impact.  I would suggest if any customers are having issues with content filtering actively blocking things incorrectly to open a case. We can see one of the cases thats come from that thread. 

Hope that helps!

- Caroline

[Jameson]

Wanted to follow up here. We received the below message from Meraki today. I verified that the search was working. With the working search, we could see that one of our internal domains is being flagged as "Advertisements", which we block. Yesterday, it wasn't showing a category in the Meraki "event log". It is now showing the "Advertisements" category on any domain or subdomain using that internal root domain. We do own the domain, so we are not sure how it was flagged as "Advertisement" as it's not used externally.

Response from Meraki:
Thank you for your patience as the engineering team has been able to work on providing a solution. I received word from the Engineering team that this issue has been resolved.