I've been messing around a bit with Custom Certificates on Meraki Anyconnect. Since I've yet successfully argued with my wife that we need a PKI infrastructure in our home, I've been attempting with LetsEncrypt, and so far I've been unsuccessful.
However, I'm curious if any other out there have been successful with signing certificates for Anyconnect with LetsEncrypt?
I feel I've so far boiled the commands down to either one of two.
sudo certbot certonly --manual --preferred-challenges dns -d <A record pointing to MX IP> --csr MX-Anyconnect.csr
which will provide 3 files; a (device?) certificate, Intermediate CA Chain, and full CA Chain.
sudo certbot certonly --manual --preferred-challenges dns -d <A record to MX IP>
which will request a certificate and require you to create a DNS TXT entry _acme-challenge.<domain name>. with a unique value for DNS-01 challenge, as well as 4 files (same as the three above + the privkey.)
Anyone else had luck with LetsEncrypt for this, or is it simply not supported?
You cannot use LetsEncrypt. You will need to generate a CSR on the MX, take that to your certificate authority, get it signed, and then come back and install it.
That is very interesting thinking.
Have you thought about going pure offline private PKI, using either openssl or Windows CA server?
I was under the impression that DNS based authentication for certbot wasn't supported anymore, but a Google seems to suggest it is.
What happens when you load the generated device certificate back into the Meraki portal?
If you open the generated device certificate on your computer - what does it look like?
When I add the device cert (or atleast I think it is..) and the full chain, I get the error "Failed verifying Device Cert with Cert Chain", which according to the documentation is due to an incomplete or invalid chain of trust, which I'm not completely sure why. One of the files I get should be the full chain certificate.
Keep posting with your progress and also how you would automate it with the API!
For your home PKI I can suggest https://smallstep.com/ if you have up to 20 devices.
So far I haven't thought about automating it. Atm, it seems only to be valid for 90 days, so I'll have to look into, to see if the expiration date can be set to 365 days.
As fas as I know there is no way to influence the lifetime. For the chain, what did you upload? In most cases it only has to be the identity and the intermediate cert.
They seem to want a Device Cert and the CA chain. But the ones I get from certbot don't seem to work. And I've tried replacing the CA chain with the Full Chain certificate, but that doesn't work aswell.
The order of the certain chain might be important. Try these two combinations:
Hm.. I haven't thought about that it might be reversed order chain. I'll try that one I get the opportunity again.
based on RFC 5246 the only valid option is:
intermediate-that signed-the device-cert
Doesn't work by flipping the order in the chain, as well. Getting the same error. I can't really wrap my head around what's missing.
Have you double checked to include the correct intermediate? When you open both the device cert and the intermediate, does the Issuer key-ID of the device certificate match the key-ID of the intermediate certificate?
Although I don't have any explanations for the behavior of the Certbot, here's the workaround.
Meraki expects two certificates: device certificate and certificate chain in device-intermediate-root order.
After generating the Lets Encrypt certificate I got the three files and tried different ways to combine them but always had the same problem until I have decided to go extra mile.
I realized that I use Letsencrypt for my Synology NAS and decided to analyze the difference between the certs issued for by Certbot manual step and the Synology automatic process and I have notice that while the intermediate certificates are the same, the root certificate are different 😱.
After extracting the Letsencrypt root certificate from the Synology certificate and using it to create the chain for Meraki - it worked like charm. 🕺
I suspect "--preferred-chain" parameter would solve the issue, but I let others to continue the testing.
Meanwhile, here is the working root certificate.
-----BEGIN CERTIFICATE----- MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4 WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+ 0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ 3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5 ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq 4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc= -----END CERTIFICATE-----
I would not say it is massive effort, except the fact it was 01:00 AM 🙂 and that I had to come up with a solution to run the certbot on the Synology which is behind the MX appliance, which is miles away.
You can't skip the root certificate, as it is the required piece of information to validate the device certificate, but now when I looked once more at both certificates I think I have another bit of explanation.
The previously posted root certificate is self-signed, while the one generated by Certbot is signed by another authority, and therefore we are missing it in the chain. If one were to add Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3 in the chain, most probably it would have worked.
Also, a little bit of nonsense what is active and what is retired can be found here: https://letsencrypt.org/certificates/
Here is the non-working root cert:
-----BEGIN CERTIFICATE----- MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/ MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT DkRTVCBSb290IENBIFgzMB4XDTIxMDEyMDE5MTQwM1oXDTI0MDkzMDE4MTQwM1ow TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwggIiMA0GCSqGSIb3DQEB AQUAA4ICDwAwggIKAoICAQCt6CRz9BQ385ueK1coHIe+3LffOJCMbjzmV6B493XC ov71am72AE8o295ohmxEk7axY/0UEmu/H9LqMZshftEzPLpI9d1537O4/xLxIZpL wYqGcWlKZmZsj348cL+tKSIG8+TA5oCu4kuPt5l+lAOf00eXfJlII1PoOK5PCm+D LtFJV4yAdLbaL9A4jXsDcCEbdfIwPPqPrt3aY6vrFk/CjhFLfs8L6P+1dy70sntK 4EwSJQxwjQMpoOFTJOwT2e4ZvxCzSow/iaNhUd6shweU9GNx7C7ib1uYgeGJXDR5 bHbvO5BieebbpJovJsXQEOEO3tkQjhb7t/eo98flAgeYjzYIlefiN5YNNnWe+w5y sR2bvAP5SQXYgd0FtCrWQemsAXaVCg/Y39W9Eh81LygXbNKYwagJZHduRze6zqxZ Xmidf3LWicUGQSk+WT7dJvUkyRGnWqNMQB9GoZm1pzpRboY7nn1ypxIFeFntPlF4 FQsDj43QLwWyPntKHEtzBRL8xurgUBN8Q5N0s8p0544fAQjQMNRbcTa0B7rBMDBc SLeCO5imfWCKoqMpgsy6vYMEG6KDA0Gh1gXxG8K28Kh8hjtGqEgqiNx2mna/H2ql PRmP6zjzZN7IKw0KKP/32+IVQtQi0Cdd4Xn+GOdwiK1O5tmLOsbdJ1Fu/7xk9TND TwIDAQABo4IBRjCCAUIwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYw SwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5pZGVudHJ1 c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTEp7Gkeyxx +tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEB ATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQu b3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0LmNvbS9E U1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFHm0WeZ7tuXkAXOACIjIGlj26Ztu MA0GCSqGSIb3DQEBCwUAA4IBAQAKcwBslm7/DlLQrt2M51oGrS+o44+/yQoDFVDC 5WxCu2+b9LRPwkSICHXM6webFGJueN7sJ7o5XPWioW5WlHAQU7G75K/QosMrAdSW 9MUgNTP52GE24HGNtLi1qoJFlcDyqSMo59ahy2cI2qBDLKobkx/J3vWraV0T9VuG WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5 -----END CERTIFICATE-----
>You can't skip the root certificate, as it is the required piece of information to validate the device certificate
Nothing should be using a provided root certificate as a method of trust. Only internally burned-in root certificates, or root certificates that have been loaded in be trusted by a device.
>The previously posted root certificate is self-signed
All root certificates are self-signed.
>while the one generated by Certbot is signed by another authority,
That would make it an intermediate certificate.
Still, after that much effort, let's take the win as is. 🙂
This is aweseome info!
I has just looking at the root certificates that LetsEncrypt has on their webpage - the same page you refer to later in the post with the Non-Working Root Cert.
The Root Cert you have posted, is the one they refer to as the Self-Signed ISRG Root X1.
I'm going to test this later. 😄
With the hints given from @RomanMD I've managed to get it to work with LetsEncrypt.
I used the command
sudo certbot certonly --manual --preferred-challenges dns --csr MX-Anyconnect.csr -d <A record to Meraki MX>
where MX-Anyconnect.csr is the Signing Request generated from Meraki Dashboard.
This yields a Challenge that needs to be configured on a TXT record via your own DNS Admin Portal. After successfully verifying this DNS challenge, three files are created:
However, Certbot creates the certificates with the invalid Root Cert, as pointed out by @RomanMD.
So after replacing the invalid Root Cert with isrgrootx1.pem, the Meraki Dashboard accepted the device and chain certificate.
Tested with Cisco Secure Connect Client, and not getting any certificate errors. 🙂
Kudos and thanks to @RomanMD with hinting towards the invalid Root Cert 🙂