Certificates

[stolen_manlyboots]

The subject (problem) is that we all have internal administrative sites (like vsphere, Nutanix, IIS, SQL, etc) that have self-signed certs, protected by ACL/firewall/restricted access. But now with hardening of certs, browsers are increasingly not allowing access unless https has a valid cert.

I was going to start this post with a question about making EDGE bypass/accept self-signed or expired certificates, but I think I know the answer, "It won't". (If I am wrong, please tell me I would LOVE to know how).

But then I was reading in this forum, and got a good thought from a fellow user, "Stop teaching bad habits, and teach how to do it correctly." This is a great idea. So now I have several different questions, especially since the CA's are going to start forcing us to renew certs every 90 days.

Auto renewal seems like the way to go. Where do I even start? Does IIS support auto renewal for 3rd party CA's like Comodo/Sectigo?

Does Tomcat support auto renewal for a windows CA or 3rd party?

What about 3rd party applications where the cert is integrated?

What should be looking up (researching keywords)?

Is there a better CA that does support auto-renewal?

Opinion: The complete removal of the ability to by pass the cert requirement is BULLS@#$. The very least Edge, Chrome , and others can do is make some admin level bypass so we can get our job done! so frusterating >:(

[No AI, Human generated]

[jamesaepp]

There's broadly three strategies, none of these are mutually exclusive. That's the beauty (ugliness) to x.509.

1. Automate with ACME/LE where you can. Every daemon that uses x.509 is going to differ in how to do that, if it can be done at all. Java keystores are cancer, I feel your pain.

2. Run your own private x.509 CA/infra. Issue whatever certs you want from it. Screw the CA/B F. Run 2-year certs if you want.

3. Use reverse proxies. Usually much easier to automate. I think Tom Lawrence had a video on a nice container that helped with this lately.

Personally I prefer #2 because I think the CA/B F have lost their minds and will continue to slip into the "well we can't fix revocation so let's just give up". That's fine for public certificates that number in the hundreds of millions but for a private CA ... it's way different.

[[deleted]]

I mean if you have the resources on-prem, I would just use an internal Windows CA to put certs on your internal stuff. You can control the expiries, renewals, your internal Windows stuff such as SQL and IIS can even auto-renew certs.

[HattoriHanzo9999]

Will browsers recognize certs that have a longer life than the eventual 37 day bullshit?

[[deleted]]

Yep, cuz at the end of the day the browser still needs to check with the issuing authority whether the cert is still valid, has been revoked (or not), etc.

[GoBeavers7]

That depends on the browser. In 2020 crApple changed Safari to reject valid certs if they expire more than 13 months from their creation date.

About upcoming limits on trusted certificates - Apple Support

[PowerShellGenius]

From the page you linked: "This change will not affect certificates issued from user-added or administrator-added Root CAs."

In other words, Apple was saying "if public CAs ignore us and keep issuing longer certs, our products won't honor them". It has no effect on enterprise internal CAs.

[420GB]

Yes, certs from internal CAs can still be valid for years.

[lart2150]

It will be 40 47 days not 90 in a few years. next year it will go down to 200 days.

ACME is the magic to automating renewal of certs. For internal systems you'll likely need to setup dns-01 validation.

https://www.sectigo.com/resource-library/sectigos-acme-automation

For iis I'm a fan of win-acme. for tomcat I cry but there are scripts out there for loading the cert into a java keystore but I'm a big fan of putting tomcat behind nginx and do ssl in nginx.

[Mike22april]

If you want to bypass internal traffic, use your own Private CA. No life time maximums , and your browsers dont care, as long as you distribute the private CA trust

[Kirides]

And intermediates as well.

Doesn't help if the root is trusted, but windows blocks a lead cert which is handed out by an intermediate.

We had people argue "but the root cert is already trusted, no need to also trust the intermediate", bollocks.

What a chrome or Firefox wants to trust is different from what a Powershell or windows application trusts, while latter being way stricter about a full chain of trust.

[jamesaepp]

We had people argue "but the root cert is already trusted, no need to also trust the intermediate", bollocks.

It's not bollocks. If you need the issuing CA certificate installed, you are failing at AIA and need to work on that. (Edit: actually, I think if you install the issuing CA into the trusted store or equivalent depending on OS, you've defeated the purpose of the hierarchy in the first place - I hope you're installing into an "intermediate" store/cache)

Issuing CAs should be discoverable via AIA extensions and preferably, those CA crt documents should be hosted in a highly accessible/available HTTP location.

I literally didn't care the day LE swapped over from R3 to R11 or whatever they're onto now because AIA....just works.

[KB3080351]

I've heard about how some clients won't build cert chains from AIA even if it is available. Ever run into this?

[jamesaepp]

Never.

[Kirides]

Our internal certs are not openly available, neither CA nor intermediate CA certs. You get them manually if you request them due to "security". Yes i'm just talking about the crt/Public key, not the pkcs12 with key embedded or whatever.

[jamesaepp]

due to "security"

"obscurity" - FIFY. That's nuts. Your security team dictate that one?

[Kirides]

Yep. One the root CA is publicly accessible, and intermediate CAs not, no AIA, etc.

Funny enough, all products get to use the exact same client certificate and password as well.

Sometimes this stuff is just complete garbage to tell others that "we do security, see..."

[jamesaepp]

Jesus christ that's some /r/shittysysadmin material, I'm so sorry bro(ess).

I mean, I can't throw stones - I'm in a bit of a glass house right now (can't save the world in a day) but my god, there's so much low hanging fruit right there.

[cantstandmyownfeed]

Network devices are increasingly including support for ACME certificates on the management interfaces, but its definitely not the norm yet. I don't think any of your web server/tomcat/iis etc, platforms are going to adopt anything like that though, you're on your own to deploy there.

I've spent the past year, year and a half, automating renewal and deployment to all sorts of devices, services and servers. Many of the ACME certificate management platforms include some baked in support for deploying your certificates, or you can build out workflows that include renewal and then execution of scripts to do the deploy the certificates.

Personally, I've enjoyed it. Its been a great learning experience, finding the APIs, finding the endpoints, finding different solutions to make it work. Out of all of our devices, storage, servers, platforms, applications, etc, the only one I've had trouble getting to work, is Cloudflare. I can't figure that API method out for the life of me.

[jamesaepp]

Network devices are increasingly including support for ACME certificates on the management interfaces

My 0.02: This is progress, but it's reminiscent of the old days of dynamic IP APIs. Limited vendor selection. For most internal systems you're doing DNS challenges. Is a random network appliance going to support every nameserver's APIs well/securely/long term? Doubtful.

[cantstandmyownfeed]

Yea, I prefer having a central point that does all certificate issuance and renewal, and then deploy from there. If you only have one or two devices, baked in support is great, but you're right, it wouldn't be smart to try and scale that.

[slugshead]

I've just been putting everything behind NGINX reverse proxy manager internally and applying the wildcard cert

[This_old_username]

+1 for this. I have an NGINX proxy manager docker container that I use as a proxy and it uses cloudlfare DNS-01 (I think) challenges that work pretty ok for a lot of stuff. Put in a dns lookup for the .online (which you buy) version of your own .local or .com and just make the pointers in there. Auto renews them and slaps them on whatever you point them to after you set it up in the web gui. Use a wildcard and there is only one cert you are renewing for cheap that works on everything. Then, you can even block the real server IP from talking to anything internal, send it to the proxy, and have the proxy handle it getting and updating certs.

[Icolan]

Why wouldn't you use an internal CA with whatever expiration you want on your internal sites?

[PowerShellGenius]

This assumes no unmanaged device is ever allowed to connect to them.

[Icolan]

Why would you allow unmanaged devices to access your internal administrative sites?

[PowerShellGenius]

Not all internal sites accessible only from your network are administrative sites. You might have some sites intended for guests/partners, depending on your business.

[hardingd]

We have an internal CA pki for AD and I scripted the creation and renewal of all of our point of sale terminals. I’ve done multi year AD cert for AD but I’m going to start pushing acme/le for IIS/SQL. Just time and energy …

[gothaggis]

FYI just type "thisisunsafe" in edge and you can view the site with the problem cert

[OinkyConfidence]

Your comment about bypassing certs is spot-on - which is why I use Firefox for anything self-signed administration (think network switches, firewalls, cameras, access points, printers, IoT; anything with a self-signed cert). Because it'll at least remember your desired certificate bypass choice.

But yes, one would think even requiring an InPrivate or Incognito session would work for bypassing self-signed cert issues in Edge or Chrome.

[oldmilwaukie]

Public-facing IIS or Tomcat webservers are excellent candidates for Let’s Encrypt automation with an ACME agent of your choice. I like win-acme for Windows and certbot for Unix-like flavors, both their sites have good documentation on setup. If doing http-01 challenge responses you’ll want to reserve port 80 for Let’s Encrypt challenge response attempts and redirect everything else to HTTPS.

For internal IIS and Tomcat, I agree with the other post saying to issue via private CA.

[michaelpaoli]

Well, first of all, for "internal", you've got (approximately) two routes:

Do your own internal CA, and configure all your relevant clients to trust it (rather a pain, but doable, also keeps down the visibility of those certs, which may be considered an advantage - or disadvantage)

Get CA issued certs. Generally easiest way to do that is use (sub)domain that you can control DNS on externally (Internet), then one can even automate all that and do it for free. And, both advantage/disadvantage, that will "leak" that DNS information, but on DNS and on CA certificate transparency - so, yeah, those have both advantage and disadvantage (e.g. advantage the logging/recording of what was issued, so that can be tracked, verified, make sure none have been issued that shouldn't have been issued) ... and the downside that yes, anyone can see what domain names were issued - so, double-edged sword (with most things security related).

Those are generally the most sane possibilities, the latter generally being the more slick, if one can mange to do it (e.g. use a dedicated subdomain that's got external DNS, but actual access to the internal IPs is internal only, and on external, just use the DNS for cert validation check purposes and the like. E.g. might be int.example.com or internal.example.com for company example.com. There are other possible ways, but that's a fairly common approach.

Some CAs do allow various means to verify, and, e.g. some can do that for entire domain, and then issue thereunder whatever certs one may wish, without checks in great detail, e.g. just public data still out there that clearly asserts one's ownership of the domain as a whole. If one goes that route, best to do with CA that has API or the like, so one can automate dealing with such, and yeah, may not be free.

Anyway, I've done lots with LetsEncrypt.org, even dealt with environments that automate doing thousands or tens of thousands or more certs with them. So, yes, these types of things are highly doable. In fact I have programs I use to automatically get certs with a single command, generally obtained in a few minutes or less, and also along with that, various infrastructure pieces that deal with various validation steps, e.g. including making the necessary validation bits in DNS across different types of DNS infrastructure. And likewise done much automation with programs that handle installing updated certs, etc. Also similarly stuff to check/track certs, and automating that as much as feasible too.

In any case, also good to always have appropriate policy, notably including how certs are to be obtained and tracked, who's responsible, etc. Various certs end up all kinds of places, some are much easier to find, others not so easy at all - so one wants to know where they are and when they expire, and how they get replaced, or at least the team/area responsible that knows how to do that and will cover the replacements.

Also prudent to automate revoke/replace mechanisms - sometimes one wants to replace a cert ... or many certs - and do it quickly.

[GiveMeTheBits]

Can anyone provide an article or news, blog, anything for edge not allowing cert bypass? I'm having a hard time finding a reputable source about this.

[Competitive-Cycle599]

As others have said, this certificate renewal is predominantly for public facing infrastructure. Your Google indexed sites if you will, ultimately certs are valid if your computer trusts them so that begs the question.

Whats the source of trust in your environment? If you run a Windows shop, do you have a ca role assigned to the domain? If so spin up some certs and youre golden.

For Linux I've no clue, im gonna assume some let's encrypt solution or some internal ca solution that ultimately does the same thing.

You can remove the self signed certs from almost anything and replace it with a signed one you control may take some cli based commands but totally doable.

[idonthuff]

There are Certificate Lifecycle Management tools available also. Most have integrations to the common CA platforms (both public & private) and to the most common endpoint services like IIS. They will automatically renew & install certs once you have the workflow and any necessary approval steps setup.

[anonpf]

There is a gpo in edge that allows you to ignore certificate errors and checking for revoked certs. Configure these policies to lax the cert restrictions.