Google.com incorrectly Geolocated

[GiacomoS]

Hey Super people in the community,

Please be aware that there seems to be an issue with the geolocalization of the Google.com IP address and it is now showing as moved to Hong Kong. We are liaising with our provider to get this rectified, but in the meantime I'd suggest, if possible at all, to remove Hong Kong from the list of Country blocking. 

We understand that this may not be possible based on your internal policies. If the workaround is not usable for your environment, please bear with us whilst we continue to work with our provider to get this rectified. 

Many thanks for your patience!

Giac

[thomasthomsen]

So that might explain my problem.

https://community.meraki.com/t5/Security-SD-WAN/No-quot-google-quot-services-all-of-a-sudden-What-th...

Now my wish is to have the eventlog or security center represent when something is being blocked by Geolocation, because you are completely in the dark as to what is going on.

[GiacomoS]

Hey team,

@thomasthomsen noted, thank you for sharing! I'd definitely recommend to submit the suggestion through Dashboard as well, so our Products team can explore this too. 

@DangerNoodle I hear you and I'm really sorry for the impact this is having on your network. I'll make sure to pass the feedback internally, thank you!

Giac

[thomasthomsen]

Dont worry, already done.

But Im just one man 🙂

@everyone , please use that "Give your feedback" button ... absolutely use it.

[MajorReb]

I must be missing something. Does Cisco not define (in it's vendor contracts) expectations that impact our networks & security. Oh, and if I read that right, possibly a service related to or server in the CCP controlled China?

Again, maybe I'm missing something.

[DBlum]

I 100% percent agree with us, how can a company such a Cisco be reliant on an outside third-party to provide security updates to their infrastructure and how it allows outside traffic to pass through their hardware to companies that pay for a “Cisco” service. What were to happen if this third-party was attacked and they changed the routing tables such as a ransomware or C2 site to somewhere inside the United States that was not being blocked.

[CraigCummings]

Don't hold your breath.  I asked for this several years ago. 

[DangerNoodle]

@GiacomoS Can I suggest that the Meraki team find another way to notify customers of major issues like this? Company wide we were unable to access Google, Google Workplace, etc you get the point.

I understand the categorization isn't your fault, however, seeing this earlier would have saved us a whole lot of digging as to what was going on.

Thanks for posting this up.

[Twitch]

Agreed. A banner notification on the Dashboard would be awesome. I have beat my head against the wall this morning trying to figure-out what was going on.

On another note, I would be very cautious about removing a block on Hong Kong just to reach Google. This could potentially be related to a cyber attack for all we know right now. 

Use DuckDuckGo for search instead.

[Clint_ACISD]

"Use DuckDuckGo for search instead."

That doesn't help organizations like ours that rely almost entirely on Google services to function. I agree on caution removing the HK block but as of right now I have no way around it to continue functioning the rest of the day.

[NordOps]

Just a suggestion or "making a wish".  If we could add something to the whitelist that overrides the geo filtering that would be a super useful feature. 

Block a country but allow this website would be a huge enhancement for security too.

[MarcAEC]

I've sent this "wish" in multiple times over the last few years.  It feels like "Make a Wish" just goes direct to /dev/null.  😞

[Warren]

---

@MarcAEC wrote:

I've sent this "wish" in multiple times over the last few years.  It feels like "Make a Wish" just goes direct to /dev/null.  😞

---

While Account Reps and Support claim that "Make a Wish" has an impact on what is developed, there are tons that we have put in that we haven't seen any progress on in 5 years and pretty much don't expect it.  If it's a small UI tweak then I think it's likely.  If it's a functionality request - don't hold your breath.  AnyConnect support (Ikev2) had been wished for years before there was any progress.  

[AmateurWheels]

Not just Google IP's.  Our local bank mercbank.com is coming up as Singapore (instead of USA)  This service is having some issues.

[SaferNetworkSol]

Am I the only one that thinks suggesting allowing a potential threat location mitigates the value of the Geolocation blocking?  If Meraki is relying on a third party service to provide this information, it sounds like you just opened everyone to additional risk.  Will you be able to post the after action report as to why this happened and what is changing to prevent it in the future?

[RodrigoC · Accepted solution]

Hey Everyone,

Update on the situation...

We have worked with our GEO IP vendor and identified a root cause for this issue. Meraki Engineering has pushed a fix to remediate the issue. The fix will apply based on the configured list update interval settings which can be configured under SD-WAN > SD-WAN & traffic shaping.

For a more immediate update, users can change their current content rules to a full list or top sites, wait for a configuration update, and revert the changes back to the previously configured setting. These settings can be found under SD-WAN > Content filtering.

If anyone has any follow-up questions please let me know!

[StosSpiro]

Can you provide us with an ETA as to when this fix will be pushed out please? 

[RodrigoC]

Hey @StosSpiro,

The issue should now be resolved. Your MX will take the fix once it hits its next update interval or if you toggle your current content rules (full list or top sites) on Dashboard. The latter basically forces the update right away instead of waiting.

Let me know if this fixes the issue

[Matt-O]

I am still having issues with itron.com which is a website reverse proxied by sucuri.net and it is being block by Geo-Location of Signapore.  For us any website reverse proxied by sucuri is still not working.  When can I expect a resolution to this?

Is this a separate issue that needs to be brought up differently?

[putt4show]

still having issue with the website

ontargetrange.com

seems like this issue isn't as specific as just Hong Kong

[RodrigoC]

Hey @putt4show, @Matt-O , @MMcGough ,

If you are still seeing issues with incorrect geo IP mappings, please open a support ticket so we can take a look. We have an established process with our geo IP vendor to correct incorrect mappings so we should be able to help. 

[putt4show]

the support team is clueless. i opened ticket 07056658 on Friday letting them know 2 sites are still not working for us. as of 8am PDT the sites are still not working.

http://ontargetrange.com/

https://officesolutions.com/log-in/

the first reply i got back from the rep was that the issue was fixed. so i told him that this thread exists and was asked to report any additional websites that were still not working

the rep got back and asked if the websites were still not working and in the next sentence he thanked himself that he could assist... WTF?!?! he didnt even do anything!!!  the sites still do not work...

"Are you still unable to resolve those sites? The issue is reported fixed for most networks. 
 

I'm glad I could assist. If there are ever any questions or concerns that you have please do not hesitate to reach out to Meraki Support!"

[RickA]

@putt4show Both of your listed websites are working for our organization, and we are full Meraki Cloud deployed. If these websites are not working for you it has to be something other than Meraki.

Websites you listed are 

• http://ontargetrange.com/
• https://officesolutions.com/log-in/

[putt4show]

we still have Hong Kong blocked and i don't plan to remove or change our L7 rules. we prtty much are blocking all countries outside of the USA.

I'm assuming your probably have made changes to your L7 rules.

[RickA]

@putt4show 

We also block almost every country outside of the United States, and yes we have Hong Kong blocked in our Layer 7 rules. I just wanted to let you know and can confirm, we are able to access both of the websites you listed. I'm certain this seems as though it is a strange coincidence, but Meraki is definitely not blocking those websites and is definitely not seeing those website IPs as China/Hong Kong.

POSSIBLE SOLUTION:  Just for the measure of testing, try removing Hong Kong from your Layer 7 rules and save. Then add it back in and save.

[StosSpiro]

@RodrigoC - I had just opened up case 07055940 to which I was told the fix had been applied but I am still unable to hit the following URL from any of my remote sites. This is a SaaS based platform with small footprint install locally. 

https://z001commonservices.cchaxcess.com/IDSAuthServer/?realm=792240&pilotcode=&isinternaluser=False...

[TonyDavis]

What is the time frame on correcting this issue?  I have a car dealership that is down and cannot sell vehicles!

[ToddB]

Too bad the geoblocks are not logged via syslog.

[mpinedo31]

Out of curiosity, would this have shown within the event log in any way? Or in a packet capture? 

[ToddB]

Doesn't show in the event logs, as the layer 7 rules are not logged. Packet capture would show it, if you did it post firewall (i.e. you would not see the packets).

[StosSpiro]

Have there been any updates since this issue has been reported? We are experiencing the same issue and adding "China" or "Hong Kong" to the list of countries not blocked doesn't seem to work for our Meraki network. 

[DangerNoodle]

For reference, do an nslookup on the non working url, then dump the ip into here to see what country it brings up: https://www.maxmind.com/en/geoip2-precision-demo

[Matthew_C]

This was a head scratcher. I assume you will update this thread when resolved? 

[Kayakn]

We left the GeoIP in place due to security.

A reboot solved our issues immediately.

[thomasthomsen]

I rebooted, downgrade , and so in, Nothing helped until I removed Hong Kong from the blocked country list. I think Meraki should make a "current status" page of their nodes, AND their partner services, so we can keep up in an online and "realtime" way.

[cmr]

My dashboard now has this:

[StosSpiro]

Is this a regional issue or world wide? That message on our dashboard "only affecting a very small subset of customers" is pretty vague. Removing the "Check the servers certificate's revocation" check box under Advanced within Internet Options bypasses OCSP. We have a SaaS based application named CCH Axcess which stopped launching for our users until we removed this check box. 

[NotCisco]

Thats what we ended up doing too. Ironically, I had my workstation on a policy that bypasses everything so all of my Google Apps were working...and thankfully from a similar situation about a month ago, I was led into trying removal/altering of Layer 7 rules. 

Bingo. 

[NordOps]

Just adding feedback based on some of the comments from other users here about how to improve things so they could be identified easier in terms of using the geo filtering.  

1. Allow geo blocking with the ability to add a site or IP to the "allow" even though the rest of a country is denied.  That way it's not all or nothing if there is legitimate traffic in a country you want to block.
2. geo blocking dumps should dump to the event log like content filtering and malware so it can easily be identified.
3. The ability to sync geo rules / firewall rules throughout the organization with Configuration sync would be nice.

Maybe 3 can be done through the API's but I haven't played with that yet.

[BCC-SAS]

Because we use Google Workspace for everything, one of our five discreet locations was essentially down for three hours today because we were blocking traffic to/from "Hong Kong" - it seemed like a DNS issue, but it took quite some time, and eventually a call into Meraki support, to figure this one out. Next time the sooner a notice goes up the better!

Blair

[RodrigoC]

Hey @BCC-SAS,

Absolutely! We're firm believers in the motto "Never waste a crisis". We'll be leveraging everything we learned from this incident to improve and accelerate our response times in the future.

[Matt-O]

We are still having issues with domains protected by sucuri.net which is a reverse proxy security/WAF service.  These sites seems to be hitting the geo-location rule of Signapore as mentioned by @AmateurWheels .  Will this fix be included in the roll out patch?

[putt4show]

access to most websites seems to have been fixed with this update but there are still a couple sites not working for us.

how do we restore full functionality again?

is it possible that some websites are still being mis-classified with geoIP?

we still cannot access ontargetrange.com

this site isn't critical to the business by any means but just trying to make sure the issue is correctly identified and resolved.

[DBlum]

I am just curious how it only affects a small subset of customers as well?  Every single one of my clients (100's of sites, tens of thousands of users) were affected if Hong Kong was being blocked.  I would assume every single end user that had Hong Kong blocked would be affected by this and hopefully this issue will now expedite anything could help notify us.  It would almost be impossible for a geolocation log to be created for the vast size it would be for every single site.

Thank you

[MMcGough]

Some additional feedback. It looks like its not just Google's IPs that have been misidentified we've had some IPs of the web applications we use start identifying as Singapore today, when they are hosted in the US and for the last 4 years have always been identified as in the US.

[MMcGough]

It looks like Akamai's IPs or some of them are being misidentified as Singapore.

[Red-Five]

How would one determine the geolocation assigned to google.com was now Hong Kong?  Is there a lookup tool in the dashboard?

[DBlum]

Can you let us know who your geolocation vendor is and see if they have a lookup tool?

[Warren]

---

@DBlum wrote:

Can you let us know who your geolocation vendor is and see if they have a lookup tool?

---

URL/IP Lookup | Webroot BrightCloud

This is what it was before - so I presume it still is.

[AmateurWheels]

@Warren @DBlum 

The Webroot BrightCloud is for the URL filtering, not the GeoIP firewall rule filtering.  The filter in discussion today uses this:

MaxMind IP Database

[MarcAEC]

I believe the vendor is MaxMind.  Here is the lookup tool. 

https://www.maxmind.com/en/geoip2-precision-demo

I've had good experiences getting MaxMind to correct incorrect locations versus trying to do it through Meraki Support.

https://support.maxmind.com/geoip-data-correction-request/

Meraki support usually fights with me.  MaxMind will just make the correction. 

[putt4show]

@MarcAEC i tried filling out the form on maxmind website but how do i know what the correct geo location should be?

[MarcAEC]

I use https://www.iplocation.net/ to compare.  It shows the results from 5 different Geo IP databases.  When MaxMind has been wrong, I've used the results from iplocation.net to support my case.

For example, the two sites RickA @putt4show  is having trouble with, http://ontargetrange.com/ (192.124.249.67) and https://officesolutions.com/log-in/ (192.124.249.110), are showing as Singapore in MaxMind, but iplocation.net is showing them in the United States in all 5 databases.

[RickA]

@MarcAEC just a small correction, we are not having difficulty reaching the websites outlined. If I recall correctly, it is @putt4show which is unable to reach those websites.

As a potential solution, I would suggest temporarily removing Hong Kong and/or Singapore from your Layer 7 "deny" list and saving. Then you can immediately add it back in and save the changes again. Our organization does indeed block both of these country communications, and we can reach those websites today!

[putt4show]

@MarcAEC thanks for the info! actually it's me having the issue with those specific sites and not @RickA but i am sure your feedback will help him also.

i'm like 5 years removed from engineering work and mostly am responsible for higher level strategy and planning these days so this technical stuff is not my forte anymore.

but lately relying on our IT MSP has been unreliable and untimely!

[Warren]

---

@MarcAEC wrote:

I use https://www.iplocation.net/ to compare.  It shows the results from 5 different Geo IP databases.  When MaxMind has been wrong, I've used the results from iplocation.net to support my case.

For example, the two sites RickA @putt4show  is having trouble with, http://ontargetrange.com/ (192.124.249.67) and https://officesolutions.com/log-in/ (192.124.249.110), are showing as Singapore in MaxMind, but iplocation.net is showing them in the United States in all 5 databases.

---

The SUCURI.net service resolves to 192.124.249.22 - so it seems like maxmind just misclassified that whole block 192.124.249.0/24

Sucuri seems to be a large player in the WAF / website security space - so not as bad as blocking CloudFlare or Akamai - but still big enough to make a dent.

We typically block hong kong and Singapore - and will add them back after this MaxMind issue is resolved.

[putt4show]

@MarcAEC i got a quick response from MaxMind, within a couple of hours that the issue should be fixed in their database "next Tuesday" hopefully that means tomorrow. thanks again for your help!

Dear Requester,

We have reviewed and accepted your correction for IP range https://protect-us.mimecast.com/s/LNsiCKrG6YS7NBXuMnKkV?domain=192.124.249.0. The corrected data should appear in next Tuesday’s update. We will not send additional confirmation emails for separate corrections (you have submitted) that are accepted within the next 24 hours.

You may check whether your correction was applied via our database or web service online demos:

• GeoIP2 City Database Demo
• GeoIP2 City Precision Web Service Demo

If you have any questions please contact us at correction@maxmind.com.

Sincerely,
The Team at MaxMind

[combobulated]

Can confirm, unblocked Hong Kong and it's working again.

[RickA]

We have also confirmed, adding Hong Kong back into our Layer 7 "deny" list works fully by still allowing access to www.google.com, www.youtube.com, etc.

MERAKI: An email notification or updated Meraki Dashboard banner would have been helpful.

[combobulated]

Confirmed here also after removing HK from our list of whitelisted countries.  While this was going on I was unable to reach Wireshark.org as well as the googly tubes, so it definitely affected more than just Google..

[MMcGough]

Just so you guys know I still have Akamai IPs that are being misclassified as Singapore instead of the US.

[Warren]

We are also seeing a security solution - SUCURI - show up as Singapore from MaxMind - but USA based on trace routes and ipaddress.com -- when SUCURI is blocked, this messed with the certificates on numerous services.  Not sure how they play a role in cert verification.

[StosSpiro]

@Warren - This exactly what is occurring in my network. This wasn't an issue prior to the GEO IP issue. 

[putt4show]

@StosSpiro the 2 websites we're having an issue with still, are the same Securi ones also...

although when i checked on Friday the Maxmind database was not showing any results for these 2 websites but this morning they are showing up with results in the database.

[StosSpiro]

We never had any connectivity issues with CCH Axcess prior to this GEO IP debacle. Even though this vendor is still calling on IE, they will be doing away with that come November 30, 2021. I needed to add Singapore to my list of whitelisted countries on those Meraki networks which are not able to reach this SaaS based program. 

Also, based off of my case with Meraki that I worked with them last night, when you whitelist a client (Allow List), the SaaS based app is able to load the sign on screen. When reverting it back to the original policy of "Normal", it continues to work! Imagine how tedious it would be do this for X amount of user devices, especially for laptops during the pandemic when users are sporadically getting onto the LAN if ever? 

[GiacomoS]

Hey awesome people of the community,

Thank you for all your comments and feedback here, it is hugely appreciated. 

I've seen some of you are still struggling with some websites. Please do reach back out to the Support Team, they are aware of the procedure to get this corrected and will be able to action this for you, but please do keep in mind that it may take a few days to reflect. 

If you are unsure whether the IP is geolocated correctly, I'd recommend cross referencing different tools, like for example Maxmind's vs Neustar's or any other lookup tool you can find on the web. 

If you can send this across to the Support team they'll be able to action it without too much investigation. 

Regarding the Make a Wish button, I hear you all. I can reassure you that all of the wishes you send across get reviewed regularly and decided upon by our Product Management team, but they are weighted against many more other wishes or issues that we are addressing. They never fall on deaf hears though!

Giac

[MarcAEC]

@GiacomoS,

If something sits in a queue for years, it's the same as being ignored.  All of NordOps's suggestions are good and should be moved to the top of the list. 

https://community.meraki.com/t5/Security-SD-WAN/Google-com-incorrectly-Geolocated/m-p/129887/highlig...

Many of us have requested #1 and #2 years ago.  Years = deaf ears.

[Warren]

---

@StosSpiro wrote:

Also, based off of my case with Meraki that I worked with them last night, when you whitelist a client (Allow List), the SaaS based app is able to load the sign on screen.

---

This and packet captures from the client is how we troubleshoot things like these as well.  Just remember to move the client back to Normal and not leave them on Whitelist.

[Warren]

---

@StosSpiro wrote:

@Warren - This exactly what is occurring in my network. This wasn't an issue prior to the GEO IP issue. 

---

That is what we found as well - even though it's in the USA according to another source

Or tracert it.

49 ms 33 ms 15 ms ae12.cr2-was1.ip4.gtt.net [89.149.130.157]
20 ms 20 ms 20 ms ip4.gtt.net [173.205.46.86]
20 ms 20 ms 21 ms cloudproxy10022.sucuri.net [192.124.249.22]

MaxMind thinks 173.205.46.86 is in Austin TX and 89.149.130.157 is in Germany - so clearly they are wrong.

BrightCloud (WebRoot) says 89.149.130.157  is in Washington State, which lines up with the was1 in the url.

All in all though it's been a long time since we have seen major services interrupted by the GeoIP filter.