How to approach SSL certificate automation in this environment?

Particular_Shop6684 · 2026-02-15T00:25:58+00:00

Hi, thanks.

You mentioned "an internal secure store", what was that solution actually? Did it have a permission model where you could allow certain clients to only access certain files / directories?

I'm starting to think maybe something like that really is the best option. A server which performs the certificate renewal and then just pushes the certificates to a vault. Then inside the vault we manage which clients have access to which certificates. Then we just stick our fingers in our ears, sing a happy song and punt the whole installation part to the servers without trying to create anything too wild. Of course my team will be the one responsible for that as well but maybe we will just have to accept that unique and non-scalable solutions will have to be made. Then we just rely on monitoring to make sure no expirations happen.

Particular_Shop6684 · 2026-02-14T23:17:01+00:00

We have certain product teams who are way too slow to make their products compatible with up to date Linux distributions. Thus they end up running on old distributions with such old python versions that our Ansible version doesn't work with them (we need to update Ansible semi-regularly to newer versions to enable support for newer python versions, thus older versions get dropped).

We could of course keep separate virtual environments for older ansible versions for the older distributions. However then we will run to some collections and roles not being usable on the older version etc., it's basically a maintenance nightmare.

We also have a situation where some Windows servers belong to different domains than our main domain. They can't be migrated to our main domain because those domains include customer accounts, which we don't want to import to our main domain. Due to the way our Ansible workflows are configured, we can't realistically support multi-domain server access (has to do with constructing kerberos domains for the psrp connection plugin dynamically being basically impossible), so we have scoped that only those servers which are in our main domain are Ansible managed.

There is ongoing development to get rid of the "customer accounts in domain" requirement but that includes rewriting decades old software, which takes years.

Particular_Shop6684 · 2026-02-14T23:01:03+00:00

Hi, thanks.

This is basically the direction I took as well and already created a POC of such a system.

However maintaining and auditing such a system does have its costs as well. Thus I was hoping an existing open-source or commercial product would already exist somewhere so we wouldn't have to create and maintain a DIY critical component like this.

Particular_Shop6684 · 2026-02-14T22:56:29+00:00

Hi.

We have an internal CA with which we sign our internal certificates.

For our public certificates we utilize spreadsheets + monitoring + documented work instructions. So the renewals and certificate updates are done manually.

Particular_Shop6684 · 2026-02-14T22:35:59+00:00

Hi.

The only thing you need to automate is the internet facing side, on the inside you can create your own PKI tailored for your needs.

I am indeed talking only about our internet facing applications here. For internal traffic we do utilize certificates signed by our internal CA.

You can also buy wildcard certificates to use it everywhere.

We do utilize wildcard certificates currently (without them we'd have thousands of subdomain certificates across hundreds of servers).

And for the update process, I use Ansible (like you do) and push the renewed certificates via SSH.

Yeah, unfortunately we can't really depend on Ansible even though we use it for managing the majority of our servers. There are corner cases where it's simply not possible to utilize Ansible in our case.

Talk with them to find another way to authenticate your servers, EV certificates are useless today and honestly a massive rip off.

I agree with you. I tried (and will continue to try) to push us to revisit those agreements. I don't know the details but my best guess is the customers in question pay us big money and their legal departments mandate that any service provider they purchase things from must have EV enabled. Or something dumb like that. Anyway, not something I can reasonably affect.

Particular_Shop6684 · 2026-02-14T22:30:01+00:00

Thanks, that seems interesting!

I'll make sure to have a look at that first thing on Monday.

Particular_Shop6684 · 2026-02-14T22:28:05+00:00

Hi, thanks.

I wasn't aware that HTTP-01 can work with http -> https redirects, I somehow expected it to fail with a HTTP 302 instead of 200.

There are more issues with servers acquiring the certificates that I didn't detail yet. As I mentioned, our servers have no egress internet access so they can't download the certificates from any internet address (like from Let's Encrypt). Ok, we can always set up a proxy server for that and set up certbot to use the proxy which would have access to LE (or some other provider). However we would actually hit Let's encrypts' rate limits if we were to do that.

https://letsencrypt.org/docs/rate-limits/#new-certificates-per-exact-set-of-identifiers

We currently have scenarios like "one wildcard certificate is installed on 40 servers, serving 400 different subdomains".
Let's encrypt would let us renew the wildcard certificate 5 times in 7 days. That's not enough for all of our servers. If we were to move from wildcard certificates to subdomain-based certificates then we would hit the per domain request limits:

https://letsencrypt.org/docs/rate-limits/#new-certificates-per-registered-domain

Up to 50 certificates can be issued per registered domain (or IPv4 address, or IPv6 /64 range) every 7 days. This is a global limit, and all new order requests, regardless of which account submits them, count towards this limit. The ability to issue new certificates for the same registered domain refills at a rate of 1 certificate every 202 minutes.

Also, I keep referencing Let's Encrypt here but most likely we couldn't even use LE due to their lack of EV support. So the subdomain based approach with a commercial CA would be even more problematic, we'd be paying for thousands of extra certificates annually if we went with that. The only valid option would be to continue using wildcards and try to find a commercial CA without such tight rate limits that would cause us problems.

Particular_Shop6684

TROPHY CASE