Google.com incorrectly Geolocated

SOLVED
GiacomoS
Meraki Employee

Google.com incorrectly Geolocated

Hey Super people in the community,

 

Please be aware that there seems to be an issue with the geolocalization of the Google.com IP address and it is now showing as moved to Hong Kong. We are liaising with our provider to get this rectified, but in the meantime I'd suggest, if possible at all, to remove Hong Kong from the list of Country blocking. 

We understand that this may not be possible based on your internal policies. If the workaround is not usable for your environment, please bear with us whilst we continue to work with our provider to get this rectified. 

 

Many thanks for your patience!

Giac

Please keep in mind that what I post here is my personal knowledge and opinion. Don't take anything I say for the Holy Grail, but try and see!
Appreciate who helps and be respectful of every opinion and every solution offered.
Share the love, especially the Meraki one!
1 ACCEPTED SOLUTION

Accepted Solutions
RodrigoC
Meraki Employee

Re: Google.com incorrectly Geolocated

Hey Everyone,

 

Update on the situation...

 

We have worked with our GEO IP vendor and identified a root cause for this issue. Meraki Engineering has pushed a fix to remediate the issue. The fix will apply based on the configured list update interval settings which can be configured under SD-WAN > SD-WAN & traffic shaping.

For a more immediate update, users can change their current content rules to a full list or top sites, wait for a configuration update, and revert the changes back to the previously configured setting. These settings can be found under SD-WAN > Content filtering.

 

If anyone has any follow-up questions please let me know!

View solution in original post

68 REPLIES 68
thomasthomsen
Building a reputation

Re: Google.com incorrectly Geolocated

So that might explain my problem.

https://community.meraki.com/t5/Security-SD-WAN/No-quot-google-quot-services-all-of-a-sudden-What-th...

 

Now my wish is to have the eventlog or security center represent when something is being blocked by Geolocation, because you are completely in the dark as to what is going on.

DangerNoodle
Conversationalist

Re: Google.com incorrectly Geolocated

@GiacomoS Can I suggest that the Meraki team find another way to notify customers of major issues like this? Company wide we were unable to access Google, Google Workplace, etc you get the point.

 

I understand the categorization isn't your fault, however, seeing this earlier would have saved us a whole lot of digging as to what was going on.

 

Thanks for posting this up.

GiacomoS
Meraki Employee

Re: Google.com incorrectly Geolocated

Hey team,

 

@thomasthomsen noted, thank you for sharing! I'd definitely recommend to submit the suggestion through Dashboard as well, so our Products team can explore this too. 

 

@DangerNoodle I hear you and I'm really sorry for the impact this is having on your network. I'll make sure to pass the feedback internally, thank you!

 

Giac

Please keep in mind that what I post here is my personal knowledge and opinion. Don't take anything I say for the Holy Grail, but try and see!
Appreciate who helps and be respectful of every opinion and every solution offered.
Share the love, especially the Meraki one!
thomasthomsen
Building a reputation

Re: Google.com incorrectly Geolocated

Dont worry, already done.

But Im just one man 🙂

@everyone , please use that "Give your feedback" button ... absolutely use it.

 

Twitch
Building a reputation

Re: Google.com incorrectly Geolocated

Agreed. A banner notification on the Dashboard would be awesome. I have beat my head against the wall this morning trying to figure-out what was going on.

 

On another note, I would be very cautious about removing a block on Hong Kong just to reach Google. This could potentially be related to a cyber attack for all we know right now. 

 

Use DuckDuckGo for search instead.

 

 

NordOps
Here to help

Re: Google.com incorrectly Geolocated

Just a suggestion or "making a wish".  If we could add something to the whitelist that overrides the geo filtering that would be a super useful feature. 

 

Block a country but allow this website would be a huge enhancement for security too.

AmateurWheels
Conversationalist

Re: Google.com incorrectly Geolocated

Not just Google IP's.  Our local bank mercbank.com is coming up as Singapore (instead of USA)  This service is having some issues.

SaferNetworkSol
Conversationalist

Re: Google.com incorrectly Geolocated

Am I the only one that thinks suggesting allowing a potential threat location mitigates the value of the Geolocation blocking?  If Meraki is relying on a third party service to provide this information, it sounds like you just opened everyone to additional risk.  Will you be able to post the after action report as to why this happened and what is changing to prevent it in the future?

ToddB
Here to help

Re: Google.com incorrectly Geolocated

Too bad the geoblocks are not logged via syslog.

mpinedo31
New here

Re: Google.com incorrectly Geolocated

Out of curiosity, would this have shown within the event log in any way? Or in a packet capture? 

ToddB
Here to help

Re: Google.com incorrectly Geolocated

Doesn't show in the event logs, as the layer 7 rules are not logged. Packet capture would show it, if you did it post firewall (i.e. you would not see the packets).

StosSpiro
Here to help

Re: Google.com incorrectly Geolocated

Have there been any updates since this issue has been reported? We are experiencing the same issue and adding "China" or "Hong Kong" to the list of countries not blocked doesn't seem to work for our Meraki network. 

Christos Spiropoulos
Matthew_C
New here

Re: Google.com incorrectly Geolocated

This was a head scratcher. I assume you will update this thread when resolved? 

DangerNoodle
Conversationalist

Re: Google.com incorrectly Geolocated

For reference, do an nslookup on the non working url, then dump the ip into here to see what country it brings up: https://www.maxmind.com/en/geoip2-precision-demo

Kayakn
New here

Re: Google.com incorrectly Geolocated

We left the GeoIP in place due to security.

A reboot solved our issues immediately.

thomasthomsen
Building a reputation

Re: Google.com incorrectly Geolocated

I rebooted, downgrade , and so in, Nothing helped until I removed Hong Kong from the blocked country list. I think Meraki should make a "current status" page of their nodes, AND their partner services, so we can keep up in an online and "realtime" way.

cmr
Kind of a big deal
Kind of a big deal

Re: Google.com incorrectly Geolocated

My dashboard now has this:

 

cmr_0-1632502613901.png

 

NotCisco
New here

Re: Google.com incorrectly Geolocated

Thats what we ended up doing too. Ironically, I had my workstation on a policy that bypasses everything so all of my Google Apps were working...and thankfully from a similar situation about a month ago, I was led into trying removal/altering of Layer 7 rules. 

 

Bingo. 

StosSpiro
Here to help

Re: Google.com incorrectly Geolocated

Is this a regional issue or world wide? That message on our dashboard "only affecting a very small subset of customers" is pretty vague. Removing the "Check the servers certificate's revocation" check box under Advanced within Internet Options bypasses OCSP. We have a SaaS based application named CCH Axcess which stopped launching for our users until we removed this check box. 

StosSpiro_0-1632503133211.png

 

Christos Spiropoulos
RodrigoC
Meraki Employee

Re: Google.com incorrectly Geolocated

Hey Everyone,

 

Update on the situation...

 

We have worked with our GEO IP vendor and identified a root cause for this issue. Meraki Engineering has pushed a fix to remediate the issue. The fix will apply based on the configured list update interval settings which can be configured under SD-WAN > SD-WAN & traffic shaping.

For a more immediate update, users can change their current content rules to a full list or top sites, wait for a configuration update, and revert the changes back to the previously configured setting. These settings can be found under SD-WAN > Content filtering.

 

If anyone has any follow-up questions please let me know!

View solution in original post

NordOps
Here to help

Re: Google.com incorrectly Geolocated

Just adding feedback based on some of the comments from other users here about how to improve things so they could be identified easier in terms of using the geo filtering.  

 

  1. Allow geo blocking with the ability to add a site or IP to the "allow" even though the rest of a country is denied.  That way it's not all or nothing if there is legitimate traffic in a country you want to block.
  2. geo blocking dumps should dump to the event log like content filtering and malware so it can easily be identified.
  3. The ability to sync geo rules / firewall rules throughout the organization with Configuration sync would be nice.

 

Maybe 3 can be done through the API's but I haven't played with that yet.

StosSpiro
Here to help

Re: Google.com incorrectly Geolocated

Can you provide us with an ETA as to when this fix will be pushed out please? 

Christos Spiropoulos
BCC-SAS
New here

Re: Google.com incorrectly Geolocated

Because we use Google Workspace for everything, one of our five discreet locations was essentially down for three hours today because we were blocking traffic to/from "Hong Kong" - it seemed like a DNS issue, but it took quite some time, and eventually a call into Meraki support, to figure this one out. Next time the sooner a notice goes up the better!

 

Blair

TonyDavis
Conversationalist

Re: Google.com incorrectly Geolocated

What is the time frame on correcting this issue?  I have a car dealership that is down and cannot sell vehicles!

Matt-O
New here

Re: Google.com incorrectly Geolocated

We are still having issues with domains protected by sucuri.net which is a reverse proxy security/WAF service.  These sites seems to be hitting the geo-location rule of Signapore as mentioned by @AmateurWheels .  Will this fix be included in the roll out patch?

putt4show
Here to help

Re: Google.com incorrectly Geolocated

access to most websites seems to have been fixed with this update but there are still a couple sites not working for us.

 

how do we restore full functionality again?

 

is it possible that some websites are still being mis-classified with geoIP?

 

we still cannot access ontargetrange.com

 

this site isn't critical to the business by any means but just trying to make sure the issue is correctly identified and resolved.

DBlum
Getting noticed

Re: Google.com incorrectly Geolocated

I am just curious how it only affects a small subset of customers as well?  Every single one of my clients (100's of sites, tens of thousands of users) were affected if Hong Kong was being blocked.  I would assume every single end user that had Hong Kong blocked would be affected by this and hopefully this issue will now expedite anything could help notify us.  It would almost be impossible for a geolocation log to be created for the vast size it would be for every single site.

 

Thank you

Clint_ACISD
New here

Re: Google.com incorrectly Geolocated

"Use DuckDuckGo for search instead."

 

That doesn't help organizations like ours that rely almost entirely on Google services to function. I agree on caution removing the HK block but as of right now I have no way around it to continue functioning the rest of the day.

RodrigoC
Meraki Employee

Re: Google.com incorrectly Geolocated

Hey @StosSpiro,

 

The issue should now be resolved. Your MX will take the fix once it hits its next update interval or if you toggle your current content rules (full list or top sites) on Dashboard. The latter basically forces the update right away instead of waiting.

 

Let me know if this fixes the issue

MMcGough
Conversationalist

Re: Google.com incorrectly Geolocated

Some additional feedback. It looks like its not just Google's IPs that have been misidentified we've had some IPs of the web applications we use start identifying as Singapore today, when they are hosted in the US and for the last 4 years have always been identified as in the US.

Matt-O
New here

Re: Google.com incorrectly Geolocated

I am still having issues with itron.com which is a website reverse proxied by sucuri.net and it is being block by Geo-Location of Signapore.  For us any website reverse proxied by sucuri is still not working.  When can I expect a resolution to this?

 

Is this a separate issue that needs to be brought up differently?

RodrigoC
Meraki Employee

Re: Google.com incorrectly Geolocated

Hey @BCC-SAS,

 

Absolutely! We're firm believers in the motto "Never waste a crisis". We'll be leveraging everything we learned from this incident to improve and accelerate our response times in the future.

MMcGough
Conversationalist

Re: Google.com incorrectly Geolocated

It looks like Akamai's IPs or some of them are being misidentified as Singapore.

 

putt4show
Here to help

Re: Google.com incorrectly Geolocated

still having issue with the website

 

ontargetrange.com

 

seems like this issue isn't as specific as just Hong Kong

RodrigoC
Meraki Employee

Re: Google.com incorrectly Geolocated

Hey @putt4show@Matt-O , @MMcGough ,

 

If you are still seeing issues with incorrect geo IP mappings, please open a support ticket so we can take a look. We have an established process with our geo IP vendor to correct incorrect mappings so we should be able to help. 

StosSpiro
Here to help

Re: Google.com incorrectly Geolocated

@RodrigoC - I had just opened up case 07055940 to which I was told the fix had been applied but I am still unable to hit the following URL from any of my remote sites. This is a SaaS based platform with small footprint install locally. 

 

https://z001commonservices.cchaxcess.com/IDSAuthServer/?realm=792240&pilotcode=&isinternaluser=False...

 

 

StosSpiro_0-1632509374803.png

 

Christos Spiropoulos
Red-Five
Here to help

Re: Google.com incorrectly Geolocated

How would one determine the geolocation assigned to google.com was now Hong Kong?  Is there a lookup tool in the dashboard?

DBlum
Getting noticed

Re: Google.com incorrectly Geolocated

Can you let us know who your geolocation vendor is and see if they have a lookup tool?

Warren
Getting noticed

Re: Google.com incorrectly Geolocated


@DBlum wrote:

Can you let us know who your geolocation vendor is and see if they have a lookup tool?


URL/IP Lookup | Webroot BrightCloud

This is what it was before - so I presume it still is.

AmateurWheels
Conversationalist

Re: Google.com incorrectly Geolocated

@Warren @DBlum 

 

The Webroot BrightCloud is for the URL filtering, not the GeoIP firewall rule filtering.  The filter in discussion today uses this:

 

MaxMind IP Database

 

combobulated
Getting noticed

Re: Google.com incorrectly Geolocated

Can confirm, unblocked Hong Kong and it's working again.

RickA
Conversationalist

Re: Google.com incorrectly Geolocated

We have also confirmed, adding Hong Kong back into our Layer 7 "deny" list works fully by still allowing access to www.google.com, www.youtube.com, etc.

 

MERAKI: An email notification or updated Meraki Dashboard banner would have been helpful.

MajorReb
Conversationalist

Re: Google.com incorrectly Geolocated

I must be missing something. Does Cisco not define (in it's vendor contracts) expectations that impact our networks & security. Oh, and if I read that right, possibly a service related to or server in the CCP controlled China?

 

Again, maybe I'm missing something.

DBlum
Getting noticed

Re: Google.com incorrectly Geolocated

I 100% percent agree with us, how can a company such a Cisco be reliant on an outside third-party to provide security updates to their infrastructure and how it allows outside traffic to pass through their hardware to companies that pay for a “Cisco” service. What were to happen if this third-party was attacked and they changed the routing tables such as a ransomware or C2 site to somewhere inside the United States that was not being blocked.

CraigCummings
Getting noticed

Re: Google.com incorrectly Geolocated

Don't hold your breath.  I asked for this several years ago. 

combobulated
Getting noticed

Re: Google.com incorrectly Geolocated

Confirmed here also after removing HK from our list of whitelisted countries.  While this was going on I was unable to reach Wireshark.org as well as the googly tubes, so it definitely affected more than just Google..

MMcGough
Conversationalist

Re: Google.com incorrectly Geolocated

Just so you guys know I still have Akamai IPs that are being misclassified as Singapore instead of the US.

putt4show
Here to help

Re: Google.com incorrectly Geolocated

the support team is clueless. i opened ticket 07056658 on Friday letting them know 2 sites are still not working for us. as of 8am PDT the sites are still not working.

 

http://ontargetrange.com/

https://officesolutions.com/log-in/

 

the first reply i got back from the rep was that the issue was fixed. so i told him that this thread exists and was asked to report any additional websites that were still not working

 

the rep got back and asked if the websites were still not working and in the next sentence he thanked himself that he could assist... WTF?!?! he didnt even do anything!!!  the sites still do not work...

 

"Are you still unable to resolve those sites? The issue is reported fixed for most networks. 
 

I'm glad I could assist. If there are ever any questions or concerns that you have please do not hesitate to reach out to Meraki Support!"

 

 

Warren
Getting noticed

Re: Google.com incorrectly Geolocated

We are also seeing a security solution - SUCURI - show up as Singapore from MaxMind - but USA based on trace routes and ipaddress.com -- when SUCURI is blocked, this messed with the certificates on numerous services.  Not sure how they play a role in cert verification.

 

 

Get notified when there are additional replies to this discussion.
Welcome to the Meraki Community!
To start contributing, simply sign in with your Cisco account. If you don't yet have a Cisco account, you can sign up.
Labels