Custom Certificates on MX Anyconnect with LetsEncrypt

[rhbirkelund]

I've been messing around a bit with Custom Certificates on Meraki Anyconnect. Since I've yet successfully argued with my wife that we need a PKI infrastructure in our home, I've been attempting with LetsEncrypt, and so far I've been unsuccessful.

However, I'm curious if any other out there have been successful with signing certificates for Anyconnect with LetsEncrypt?

I feel I've so far boiled the commands down to either one of two.

sudo certbot certonly --manual --preferred-challenges dns -d <A record pointing to MX IP> --csr MX-Anyconnect.csr

which will provide 3 files; a (device?) certificate, Intermediate CA Chain, and full CA Chain.

Or

sudo certbot certonly --manual --preferred-challenges dns -d <A record to MX IP>

which will request a certificate and require you to create a DNS TXT entry _acme-challenge.<domain name>. with a unique value for DNS-01 challenge, as well as 4 files (same as the three above + the privkey.)

Anyone else had luck with LetsEncrypt for this, or is it simply not supported?

[alemabrahao]

You cannot use LetsEncrypt. You will need to generate a CSR on the MX, take that to your certificate authority, get it signed, and then come back and install it. 

[PhilipDAth]

That is very interesting thinking.

Have you thought about going pure offline private PKI, using either openssl or Windows CA server?

I was under the impression that DNS based authentication for certbot wasn't supported anymore, but a Google seems to suggest it is.

What happens when you load the generated device certificate back into the Meraki portal?

If you open the generated device certificate on your computer - what does it look like?

[rhbirkelund]

When I add the device cert (or atleast I think it is..) and the full chain, I get the error "Failed verifying Device Cert with Cert Chain", which according to the documentation is due to an incomplete or invalid chain of trust, which I'm not completely sure why. One of the files I get should be the full chain certificate.

[KarstenI]

Keep posting with your progress and also how you would automate it with the API!

For your home PKI I can suggest https://smallstep.com/ if you have up to 20 devices.

[rhbirkelund]

So far I haven't thought about automating it. Atm, it seems only to be valid for 90 days, so I'll have to look into, to see if the expiration date can be set to 365 days.

[KarstenI]

As fas as I know there is no way to influence the lifetime. For the chain, what did you upload? In most cases it only has to be the identity and the intermediate cert.

[rhbirkelund]

They seem to want a Device Cert and the CA chain. But the ones I get from certbot don't seem to work. And I've tried replacing the CA chain with the Full Chain certificate, but that doesn't work aswell.

[PhilipDAth]

The order of the certain chain might be important.  Try these two combinations:

device

intermediate

root

or

root

intermediate

device

[rhbirkelund]

Hm.. I haven't thought about that it might be reversed order chain. I'll try that one I get the opportunity again.

[KarstenI]

based on RFC 5246 the only valid option is:

device

intermediate-that signed-the device-cert

(intermediate-that signed-the-cert-above)

(intermediate-that signed-the-cert-above)

(root)

[rhbirkelund]

Doesn't work by flipping the order in the chain, as well. Getting the same error. I can't really wrap my head around what's missing.

[KarstenI]

Have you double checked to include the correct intermediate? When you open both the device cert and the intermediate, does the Issuer key-ID of the device certificate match the key-ID of the intermediate certificate?

[RomanMD]

Although I don't have any explanations for the behavior of the Certbot, here's the workaround.

Meraki expects two certificates: device certificate and certificate chain in device-intermediate-root order.

After generating the Lets Encrypt certificate I got the three files and tried different ways to combine them but always had the same problem until I have decided to go extra mile. 

I realized that I use Letsencrypt for my Synology NAS and decided to analyze the difference between the certs issued for by Certbot manual step and the Synology automatic process and I have notice that while the intermediate certificates are the same, the root certificate are different 😱. 

After extracting the Letsencrypt root certificate from the Synology certificate and using it to create the chain for Meraki - it worked like charm. 🕺

I suspect "--preferred-chain" parameter would solve the issue, but I let others to continue the testing.

Meanwhile, here is the working root certificate.

-----BEGIN CERTIFICATE-----
MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
-----END CERTIFICATE-----

[PhilipDAth]

I would give you double-double kudos for this if I could for the massive effort and your finding.

[RomanMD]

I would not say it is massive effort, except the fact it was 01:00 AM 🙂 and that I had to come up with a solution to run the certbot on the Synology which is behind the MX appliance, which is miles away.

[KarstenI]

Did you also try to skip the root certificate? And can you also post the “wrong” root-cert?

[RomanMD]

You can't skip the root certificate, as it is the required piece of information to validate the device certificate, but now when I looked once more at both certificates I think I have another bit of explanation.

The previously posted root certificate is self-signed, while the one generated by Certbot is signed by another authority, and therefore we are missing it in the chain. If one were to add Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3 in the chain, most probably it would have worked. 

Also, a little bit of nonsense what is active and what is retired can be found here: https://letsencrypt.org/certificates/

Here is the non-working root cert: 

-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIxMDEyMDE5MTQwM1oXDTI0MDkzMDE4MTQwM1ow
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwggIiMA0GCSqGSIb3DQEB
AQUAA4ICDwAwggIKAoICAQCt6CRz9BQ385ueK1coHIe+3LffOJCMbjzmV6B493XC
ov71am72AE8o295ohmxEk7axY/0UEmu/H9LqMZshftEzPLpI9d1537O4/xLxIZpL
wYqGcWlKZmZsj348cL+tKSIG8+TA5oCu4kuPt5l+lAOf00eXfJlII1PoOK5PCm+D
LtFJV4yAdLbaL9A4jXsDcCEbdfIwPPqPrt3aY6vrFk/CjhFLfs8L6P+1dy70sntK
4EwSJQxwjQMpoOFTJOwT2e4ZvxCzSow/iaNhUd6shweU9GNx7C7ib1uYgeGJXDR5
bHbvO5BieebbpJovJsXQEOEO3tkQjhb7t/eo98flAgeYjzYIlefiN5YNNnWe+w5y
sR2bvAP5SQXYgd0FtCrWQemsAXaVCg/Y39W9Eh81LygXbNKYwagJZHduRze6zqxZ
Xmidf3LWicUGQSk+WT7dJvUkyRGnWqNMQB9GoZm1pzpRboY7nn1ypxIFeFntPlF4
FQsDj43QLwWyPntKHEtzBRL8xurgUBN8Q5N0s8p0544fAQjQMNRbcTa0B7rBMDBc
SLeCO5imfWCKoqMpgsy6vYMEG6KDA0Gh1gXxG8K28Kh8hjtGqEgqiNx2mna/H2ql
PRmP6zjzZN7IKw0KKP/32+IVQtQi0Cdd4Xn+GOdwiK1O5tmLOsbdJ1Fu/7xk9TND
TwIDAQABo4IBRjCCAUIwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYw
SwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5pZGVudHJ1
c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTEp7Gkeyxx
+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEB
ATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQu
b3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0LmNvbS9E
U1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFHm0WeZ7tuXkAXOACIjIGlj26Ztu
MA0GCSqGSIb3DQEBCwUAA4IBAQAKcwBslm7/DlLQrt2M51oGrS+o44+/yQoDFVDC
5WxCu2+b9LRPwkSICHXM6webFGJueN7sJ7o5XPWioW5WlHAQU7G75K/QosMrAdSW
9MUgNTP52GE24HGNtLi1qoJFlcDyqSMo59ahy2cI2qBDLKobkx/J3vWraV0T9VuG
WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O
he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----

[PhilipDAth]

>You can't skip the root certificate, as it is the required piece of information to validate the device certificate

Nothing should be using a provided root certificate as a method of trust.  Only internally burned-in root certificates, or root certificates that have been loaded in be trusted by a device.

>The previously posted root certificate is self-signed

All root certificates are self-signed.

>while the one generated by Certbot is signed by another authority,

That would make it an intermediate certificate.

Still, after that much effort, let's take the win as is.  🙂

[rhbirkelund]

This is aweseome info!

I has just looking at the root certificates that LetsEncrypt has on their webpage - the same page you refer to later in the post with the Non-Working Root Cert.

The Root Cert you have posted, is the one they refer to as the Self-Signed ISRG Root X1.

I'm going to test this later. 😄

[rhbirkelund]

With the hints given from @RomanMD I've managed to get it to work with LetsEncrypt.

I used the command

sudo certbot certonly --manual --preferred-challenges dns --csr MX-Anyconnect.csr -d <A record to Meraki MX>

where MX-Anyconnect.csr is the Signing Request generated from Meraki Dashboard.

This yields a Challenge that needs to be configured on a TXT record via your own DNS Admin Portal. After successfully verifying this DNS challenge, three files are created:

• 0000_cert.pem - Device Certificate
• 0000_chain.pem - Chain Certificate
• 0001_chain.pem - Full Chain with Device Certificate

However, Certbot creates the certificates with the invalid Root Cert, as pointed out by @RomanMD.

So after replacing the invalid Root Cert with isrgrootx1.pem, the Meraki Dashboard accepted the device and chain certificate.

Tested with Cisco Secure Connect Client, and not getting any certificate errors. 🙂

Kudos and thanks to @RomanMD with hinting towards the invalid Root Cert 🙂

[Hallur]

Hey @rhbirkelund. After reading through the thread, and a lot of failed attempts, I'm hoping that you can help.

I've managed to create the certificate files (0000_cert.pem, 0000_chain.pem, and 0001_chain.pem) using Certbot and the CSR file I generated on the MX page, but even if I replace the invalid root certificate in 0001_chain.pem with the ISRG Root X1 certificate, I still get the following error "Unknown Error Failed verifying Device Cert with Cert Chain" when I upload the files.

Here's exactly what I do:

- When generating the CSR file on the MX page, I input into the Common Name field the domain name of the A record that points to the IP of my MX, and I leave the rest of the fields blank

- I then use the CSR file to generate the certificate files using Certbot

- I replace the invalid root certificate inside 0001_chain.pem with the ISRG Root X1 certificate

- I upload the 0001_cert.pem as the Device Certificate, and the 0001_chain.pem as the CA certificate

Any insights into what I could be doing wrong?