Google Search - Timeout across networks - MX64W and MX84s

[e39_540i]

Hi all,

I'm trying to track down an issue where Google Search will timeout multiple times a day. 

Behavior:

Open a new tab to perform a Google search with Google Chrome and the tab just spins and eventually times out. Sometimes I can open Microsoft Edge and get to google.com to search while Chrome is still spinning. This happens on the wired or wireless network.

System Specs:

• Windows 10
• Windows patches up to date
• Any desktop/laptop

I've tried doing a packet capture with wireshark but I'm not seeing the problem. I might not be looking for the right entries (just looking at DNS requests) so if you have any pointers, please let me know. It certainly feels like a DNS issue but this is happening at other offices (in different geographical locations) that have their own DNS settings. I've also tried looking at the Meraki Event Logs for the specific client but nothing out of the ordinary shows up. I can still ping google.com when this is happening.

Any ideas?

[Dylan_YYC]

Do you have the web search filtering turned on under the content filtering tab? We've also been seeing some funky things with Advanced Malware Protection if you have that enabled, it might be worth turning that off too.

[Nash]

BTW, normally if AMP is being wonky, it's enough to quickly disable it then re-enable it. That tends to clear its "indigestion". AMP only should apply to HTTP, though.

[Dylan_YYC]

Agreed, but its been acting up so much for us ive just turned it off and am waiting for the software update to fix it.

[e39_540i]

Unfortunately, we don't have Web search filtering under Content Filter. Would have been nice if that were the case then I could just disable that. We actually don't have AMP enabled at the moment (although I thought we did). We do have IDS set to Prevention and Security.

[SoCalRacer]

Any endpoint protection or managed DNS solution that could contribute to the issue?

[e39_540i]

We have CarbonBlack installed as our AV/endpoint protection. As far as DNS, we use AWS' Managed Active Directory solution so I noticed when I do an nslookup it is using one of the two servers. For instance if I say nslookup google.com, it shows me the AWS Managed AD IP address and finally the non-auth answer for google.com.

[SoCalRacer]

You might try disabling CarbonBlack and its extension or try on a machine without it installed.

[e39_540i]

I'll give it a shot. If anyone else has any ideas, I'm all ears.

[SoCalRacer]

If ultimately all your DNS requests from different locations are going to AWS AD then I would go there and see if you experience the same issues on the AD server if possible. Possibly it is an issue with how your sites are connected to AWS AD or that it is internal to AWS AD.

[e39_540i]

Figured out what was going on here. Google requests would routinely get routed to servers in Mexico. Our L7 firewall rules are currently set to only allow communication to/from certain countries and Mexico is not one of them. When the searches would time out, I could go into the search bar and change it to .co.uk and it would immediately return the search result. nslookup helped pinpoint this issue if anyone runs into this in the future.

Meraki uses MaxMind for their GeoIP lookups so it all makes more sense now. My question now is, is there a way to prevent our network users from resolving to Google's servers in Mexico? 

[rhamersley]

We contacted MAXMIND...Have to use the chat feature and they will be updating their records accordingly from the last I text them.....

My chat session with MAXMIND:

Paul:
Okay, it looks like Google was using some of the IP addresses near those for Mexican traffic, which is probably why the range including those DNS addresses was moved there. We should have the DNS addresses back in the US in next week's update.

Bobby:
To confirm the update should happen this weekend?

Paul:
Our databases update weekly on Tuesdays.

Bobby:
Ok...to confirm the updates you are performing?

Paul:
The DNS addresses should be moved back to the US. Not sure if they'll be listing them in Mountain View, CA specifically or just "US" on a country level only.

[rhamersley]

This issue directly points to an issue with google DNS servers. Currently when we encounter this latency/issue when opening google.com or performing a search we encounter this issue for a time frame of 20 to 40 and sometimes 1 minute. If you open another tab or browser and type in the following (google.co.uk - We white list the UK region) the google page comes up right away and no issues with searching on google. When performing NSLookups of google.com throughout the day I receive Mexico DNS servers it is trying to resolve to:
216.239.38.117
216.239.32.177
216.239.34.117
216.239.36.117
In our security Meraki MX appliance we block the country of Mexico. What is happening when they switch to use the Mexico DNS servers how briefly it is we encounter this latency. Google does go to the Mexico DNS locations throughout the day but only for a few seconds it looks like.

MaxMind is going to update their geolocations for those IP address that show in Mexico to show in the US.
216.239.38.117
216.239.32.117
216.239.34.117
216.239.26.117

https://www.maxmind.com/en/geoip2-precision-demo