Custom Certificates on MX Anyconnect with LetsEncrypt

rhbirkelund
Kind of a big deal

Custom Certificates on MX Anyconnect with LetsEncrypt

I've been messing around a bit with Custom Certificates on Meraki Anyconnect. Since I've yet successfully argued with my wife that we need a PKI infrastructure in our home, I've been attempting with LetsEncrypt, and so far I've been unsuccessful.

However, I'm curious if any other out there have been successful with signing certificates for Anyconnect with LetsEncrypt?

 

I feel I've so far boiled the commands down to either one of two.

sudo certbot certonly --manual --preferred-challenges dns -d <A record pointing to MX IP> --csr MX-Anyconnect.csr

which will provide 3 files; a (device?) certificate, Intermediate CA Chain, and full CA Chain.

 

Or

sudo certbot certonly --manual --preferred-challenges dns -d <A record to MX IP>

which will request a certificate and require you to create a DNS TXT entry _acme-challenge.<domain name>. with a unique value for DNS-01 challenge, as well as 4 files (same as the three above + the privkey.)

 

Anyone else had luck with LetsEncrypt for this, or is it simply not supported?

LinkedIn ::: https://blog.rhbirkelund.dk/

Like what you see? - Give a Kudo ## Did it answer your question? - Mark it as a Solution 🙂

All code examples are provided as is. Responsibility for Code execution lies solely your own.
20 Replies 20
alemabrahao
Kind of a big deal
Kind of a big deal

You cannot use LetsEncrypt. You will need to generate a CSR on the MX, take that to your certificate authority, get it signed, and then come back and install it. 

 

 

I am not a Cisco Meraki employee. My suggestions are based on documentation of Meraki best practices and day-to-day experience.

Please, if this post was useful, leave your kudos and mark it as solved.
PhilipDAth
Kind of a big deal
Kind of a big deal

That is very interesting thinking.

Have you thought about going pure offline private PKI, using either openssl or Windows CA server?

 

I was under the impression that DNS based authentication for certbot wasn't supported anymore, but a Google seems to suggest it is.

 

What happens when you load the generated device certificate back into the Meraki portal?

If you open the generated device certificate on your computer - what does it look like?

 

 

When I add the device cert (or atleast I think it is..) and the full chain, I get the error "Failed verifying Device Cert with Cert Chain", which according to the documentation is due to an incomplete or invalid chain of trust, which I'm not completely sure why. One of the files I get should be the full chain certificate.

LinkedIn ::: https://blog.rhbirkelund.dk/

Like what you see? - Give a Kudo ## Did it answer your question? - Mark it as a Solution 🙂

All code examples are provided as is. Responsibility for Code execution lies solely your own.
KarstenI
Kind of a big deal
Kind of a big deal

Keep posting with your progress and also how you would automate it with the API!

For your home PKI I can suggest https://smallstep.com/ if you have up to 20 devices.

So far I haven't thought about automating it. Atm, it seems only to be valid for 90 days, so I'll have to look into, to see if the expiration date can be set to 365 days.

LinkedIn ::: https://blog.rhbirkelund.dk/

Like what you see? - Give a Kudo ## Did it answer your question? - Mark it as a Solution 🙂

All code examples are provided as is. Responsibility for Code execution lies solely your own.

As fas as I know there is no way to influence the lifetime. For the chain, what did you upload? In most cases it only has to be the identity and the intermediate cert.

They seem to want a Device Cert and the CA chain. But the ones I get from certbot don't seem to work. And I've tried replacing the CA chain with the Full Chain certificate, but that doesn't work aswell.

rhbirkelund_0-1683795588026.png

 

LinkedIn ::: https://blog.rhbirkelund.dk/

Like what you see? - Give a Kudo ## Did it answer your question? - Mark it as a Solution 🙂

All code examples are provided as is. Responsibility for Code execution lies solely your own.

The order of the certain chain might be important.  Try these two combinations:

 

device

intermediate

root

 

or

 

root

intermediate

device

Hm.. I haven't thought about that it might be reversed order chain. I'll try that one I get the opportunity again.

LinkedIn ::: https://blog.rhbirkelund.dk/

Like what you see? - Give a Kudo ## Did it answer your question? - Mark it as a Solution 🙂

All code examples are provided as is. Responsibility for Code execution lies solely your own.
KarstenI
Kind of a big deal
Kind of a big deal

based on RFC 5246 the only valid option is:

 

device

intermediate-that signed-the device-cert

(intermediate-that signed-the-cert-above)

(intermediate-that signed-the-cert-above)

(root)

Doesn't work by flipping the order in the chain, as well. Getting the same error. I can't really wrap my head around what's missing.

rhbirkelund_0-1683876249483.png

 

LinkedIn ::: https://blog.rhbirkelund.dk/

Like what you see? - Give a Kudo ## Did it answer your question? - Mark it as a Solution 🙂

All code examples are provided as is. Responsibility for Code execution lies solely your own.

Have you double checked to include the correct intermediate? When you open both the device cert and the intermediate, does the Issuer key-ID of the device certificate match the key-ID of the intermediate certificate?

RomanMD
Building a reputation

Although I don't have any explanations for the behavior of the Certbot, here's the workaround.

 

Meraki expects two certificates: device certificate and certificate chain in device-intermediate-root order.

After generating the Lets Encrypt certificate I got the three files and tried different ways to combine them but always had the same problem until I have decided to go extra mile. 

I realized that I use Letsencrypt for my Synology NAS and decided to analyze the difference between the certs issued for by Certbot manual step and the Synology automatic process and I have notice that while the intermediate certificates are the same, the root certificate are different 😱

After extracting the Letsencrypt root certificate from the Synology certificate and using it to create the chain for Meraki - it worked like charm. 🕺

 

I suspect "--preferred-chain" parameter would solve the issue, but I let others to continue the testing.

 

Meanwhile, here is the working root certificate.

-----BEGIN CERTIFICATE-----
MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
-----END CERTIFICATE-----

 

PhilipDAth
Kind of a big deal
Kind of a big deal

I would give you double-double kudos for this if I could for the massive effort and your finding.

RomanMD
Building a reputation

I would not say it is massive effort, except the fact it was 01:00 AM 🙂 and that I had to come up with a solution to run the certbot on the Synology which is behind the MX appliance, which is miles away.

KarstenI
Kind of a big deal
Kind of a big deal

Did you also try to skip the root certificate? And can you also post the “wrong” root-cert?

RomanMD
Building a reputation

You can't skip the root certificate, as it is the required piece of information to validate the device certificate, but now when I looked once more at both certificates I think I have another bit of explanation.

The previously posted root certificate is self-signed, while the one generated by Certbot is signed by another authority, and therefore we are missing it in the chain. If one were to add Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3 in the chain, most probably it would have worked. 

 

Also, a little bit of nonsense what is active and what is retired can be found here: https://letsencrypt.org/certificates/

 

Here is the non-working root cert: 

-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIxMDEyMDE5MTQwM1oXDTI0MDkzMDE4MTQwM1ow
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwggIiMA0GCSqGSIb3DQEB
AQUAA4ICDwAwggIKAoICAQCt6CRz9BQ385ueK1coHIe+3LffOJCMbjzmV6B493XC
ov71am72AE8o295ohmxEk7axY/0UEmu/H9LqMZshftEzPLpI9d1537O4/xLxIZpL
wYqGcWlKZmZsj348cL+tKSIG8+TA5oCu4kuPt5l+lAOf00eXfJlII1PoOK5PCm+D
LtFJV4yAdLbaL9A4jXsDcCEbdfIwPPqPrt3aY6vrFk/CjhFLfs8L6P+1dy70sntK
4EwSJQxwjQMpoOFTJOwT2e4ZvxCzSow/iaNhUd6shweU9GNx7C7ib1uYgeGJXDR5
bHbvO5BieebbpJovJsXQEOEO3tkQjhb7t/eo98flAgeYjzYIlefiN5YNNnWe+w5y
sR2bvAP5SQXYgd0FtCrWQemsAXaVCg/Y39W9Eh81LygXbNKYwagJZHduRze6zqxZ
Xmidf3LWicUGQSk+WT7dJvUkyRGnWqNMQB9GoZm1pzpRboY7nn1ypxIFeFntPlF4
FQsDj43QLwWyPntKHEtzBRL8xurgUBN8Q5N0s8p0544fAQjQMNRbcTa0B7rBMDBc
SLeCO5imfWCKoqMpgsy6vYMEG6KDA0Gh1gXxG8K28Kh8hjtGqEgqiNx2mna/H2ql
PRmP6zjzZN7IKw0KKP/32+IVQtQi0Cdd4Xn+GOdwiK1O5tmLOsbdJ1Fu/7xk9TND
TwIDAQABo4IBRjCCAUIwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYw
SwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5pZGVudHJ1
c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTEp7Gkeyxx
+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEB
ATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQu
b3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0LmNvbS9E
U1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFHm0WeZ7tuXkAXOACIjIGlj26Ztu
MA0GCSqGSIb3DQEBCwUAA4IBAQAKcwBslm7/DlLQrt2M51oGrS+o44+/yQoDFVDC
5WxCu2+b9LRPwkSICHXM6webFGJueN7sJ7o5XPWioW5WlHAQU7G75K/QosMrAdSW
9MUgNTP52GE24HGNtLi1qoJFlcDyqSMo59ahy2cI2qBDLKobkx/J3vWraV0T9VuG
WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O
he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----

 

PhilipDAth
Kind of a big deal
Kind of a big deal

>You can't skip the root certificate, as it is the required piece of information to validate the device certificate

 

Nothing should be using a provided root certificate as a method of trust.  Only internally burned-in root certificates, or root certificates that have been loaded in be trusted by a device.

 

>The previously posted root certificate is self-signed

 

All root certificates are self-signed.

 

>while the one generated by Certbot is signed by another authority,

 

That would make it an intermediate certificate.

 

 

Still, after that much effort, let's take the win as is.  🙂

rhbirkelund
Kind of a big deal

This is aweseome info!

 

I has just looking at the root certificates that LetsEncrypt has on their webpage - the same page you refer to later in the post with the Non-Working Root Cert.

 

The Root Cert you have posted, is the one they refer to as the Self-Signed ISRG Root X1.

 

I'm going to test this later. 😄

LinkedIn ::: https://blog.rhbirkelund.dk/

Like what you see? - Give a Kudo ## Did it answer your question? - Mark it as a Solution 🙂

All code examples are provided as is. Responsibility for Code execution lies solely your own.

With the hints given from @RomanMD I've managed to get it to work with LetsEncrypt.

I used the command

sudo certbot certonly --manual --preferred-challenges dns --csr MX-Anyconnect.csr -d <A record to Meraki MX>

where MX-Anyconnect.csr is the Signing Request generated from Meraki Dashboard.

This yields a Challenge that needs to be configured on a TXT record via your own DNS Admin Portal. After successfully verifying this DNS challenge, three files are created:

  • 0000_cert.pem - Device Certificate
  • 0000_chain.pem - Chain Certificate
  • 0001_chain.pem - Full Chain with Device Certificate

However, Certbot creates the certificates with the invalid Root Cert, as pointed out by @RomanMD.

So after replacing the invalid Root Cert with isrgrootx1.pem, the Meraki Dashboard accepted the device and chain certificate.

 

Tested with Cisco Secure Connect Client, and not getting any certificate errors. 🙂

 

Kudos and thanks to @RomanMD with hinting towards the invalid Root Cert 🙂

LinkedIn ::: https://blog.rhbirkelund.dk/

Like what you see? - Give a Kudo ## Did it answer your question? - Mark it as a Solution 🙂

All code examples are provided as is. Responsibility for Code execution lies solely your own.
Get notified when there are additional replies to this discussion.