api.meraki.com returns fake certificate of Ingress controller

SOLVED
Expo233
Conversationalist

api.meraki.com returns fake certificate of Ingress controller

When trying to access api.meraki.com , The server is returning a fake certificate, which is causing SSL verification to fail. 


# openssl s_client -showcerts -connect api.meraki.com:443

CONNECTED(00000003)
depth=0 O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
verify error:num=21:unable to verify the first certificate
verify return:1

Any ideas on what causes this? 

1 ACCEPTED SOLUTION
John-K
Meraki Employee
Meraki Employee

Wow, thanks for raising this @Expo233 !

 

Yes, the older OpenSSL implementation does not seem to be operating as expected. This raises an interesting question, "How old an OpenSSL library should you use in production?" I honestly don't know the answer to that question but it might be a better practice to use one that was updated post-COVID.

 

Thanks again for reporting this, and please update your OpenSSL version as described above to resolve the issue.

View solution in original post

5 REPLIES 5
PhilipDAth
Kind of a big deal
Kind of a big deal

I copy and pasted your example - but I got the correct result.

 

Any chance you have a firewall or something doing an HTTPS inspection or interception?

 

PhilipDAth_0-1680464404577.png

 

Expo233
Conversationalist

The fake certificate seems to be returned depending on the OpenSSL version that is used. 

# openssl version
OpenSSL 1.1.1f 31 Mar 2020 ---> Works fine and the correct certificate is returned

# openssl version
OpenSSL 1.0.2u-fips 20 Dec 2019 ---> Returns fake cert

PhilipDAth
Kind of a big deal
Kind of a big deal

What's the bet the older version simply "connects" and does not supply the hosts header to say which web site should be accessed.

Expo233
Conversationalist

Ingress controller as per google is a reverse proxy for Meraki cloud. The response is indeed from Meraki (verified with a pcap). I tried accessing the IP of API Meraki cloud from a browser and you will get to see this fake cert being returned. 
 
It seems the response differs based on OpenSSL version. 
 
This used to work on the older OpenSSL version a few months back. So something changed on the Meraki side, probably a fix for some CVE I guess
John-K
Meraki Employee
Meraki Employee

Wow, thanks for raising this @Expo233 !

 

Yes, the older OpenSSL implementation does not seem to be operating as expected. This raises an interesting question, "How old an OpenSSL library should you use in production?" I honestly don't know the answer to that question but it might be a better practice to use one that was updated post-COVID.

 

Thanks again for reporting this, and please update your OpenSSL version as described above to resolve the issue.

Get notified when there are additional replies to this discussion.