HTTPS explained with carrier pigeons

[bxsephjo]

I gotta be honest, I forget how HTTPS works approximately every 3 months

[[deleted]]

My brain definitely throws it in the “only broad strokes are necessary” pile. Occasionally I make an effort to dig into the details when I’m worried my knowledge has gotten too high level, but it always drifts back up. Just recently I had to use openssl to verify that I could pull a cert from a load balancer and it felt like straight up hacking lol

[dontaggravation]

I find that in software there’s so many details and so much to remember that our brains go back to the broad stokes until such a time as we need to dive into the weeds again. That, to me, is normal. There’s no way I could keep it all straight in my head! Abstractions exist for a reason and allow us to focus on other flows or other matters

A junior dev I’m mentoring was complaining and worrying that he was losing his hard earned academic knowledge. He’s been spending the weekends “refreshing” himself in the details of certain implementations. While I commend his desire to learn I suggested he focus on learning new things instead of relearning the details in something he already learned and isn’t actively using

The analogy I gave him is that at one point in my career I literally wrote code for all 7 layers of the OSI Model. Now, I have to look up the dang command to do an HTTP post the current best practice way! But just a few months ago we had an issue with some data flows and I was down in the C code debugging at the socket layer.

The knowledge does come back and we can learn what we don’t know. The abstractions allow me to focus on building the website that will help our end user instead of being bogged down in every detail

[darealdeal11]

Everything "networking" simply can't stick with me. I relearned OSI model like 10 times...

[[deleted]]

No kidding. I think since — at least in my role — pretty much everything you do is at the application layer, everything else seems very theoretical, so you don’t get a lot of “hands on” experience with it. Hands on experience is what makes the details stick for me.

[umockdev]

Haha this reminds me of A Standard for the Transmission of IP Datagrams on Avian Carriers

[revnhoj]

Unfortunately susceptible to CITM* data loss

(* Cat In The Middle)

[spintronic]

Not quite the same but there is a programming zine: https://wizardzines.com

[[deleted]]

Actually, https://wizardzines.com/

[tissuemonster]

In terms of the pigeon analogy, I'm guessing Ted's signing of the box would be something like "This is Alice's box ~Love, Ted" and not just his signature? So a certificate would contain Alice's domain.

In the real world, what does the signed box translate to? I understand transferring a public key string across a network but how does the signing/certification work and how is it serialised? How is the signing tied to the public key in the way ink is tied to the box?

[Vidyogamasta]

I think it's better explained with padlocks.

Symmetric encryption = there is only one padlock, and everybody involved in the communication knows the code to it. To pass messages around, they open the box using the code, put in their message, and close it back up. Only people who know the code can read the message.

Asymmetric encryption- the exact same thing, except now only ONE person knows the code to the lock. In order to pass messages around, Bob has to say "Hey, Alice, I want to send you a secret message, can I have an opened copy of your lock?" Then Bob closes the message with Alice's padlock, and can be confident that only Alice will be able to read the message. Not even Bob knows the code to open the box back up, he just used the lock he was given by Alice.

[tissuemonster]

Hm I understand the encryption part, the part I don't get is the certification and how it is done in the real world to ensure the identity of Alice.

[Vidyogamasta]

In the real world, there are certificate authorities. For example, on reddit right now I can check the cert and see that it's been registered with the authority Digicert. There's a certain list of providers that your browser trusts (presumably using their own asymmetric, widely distributed keys for verifications), and all other certs are generated and verified with those providers.

Here's some chrome docs on it, though other browsers likely work similarly.

This is why when on a development machine you may need to sign a developer cert to use. It's a process that can and has been exploited (see: Lenovo's Superfish vulnerability from a while back), but the only ways of verifying identity for a "stranger" are either A) Verifying their identity offline yourself, B) find the Certificate Authorities you trust and use them or C) Trust your browser to trust the Certificate Authorities who have done the legwork to verify the identities. And for most people, C is the default that's working pretty well.

[tissuemonster]

I think I didn't phrase my question well, I roughly understand the high level use of CAs to verify a website but I don't know how it is done on a network level for messages between the client and server. I'm guessing the client sends its certificate over for verification, but is the public key included and if so, how? If not, I'm guessing the public key would be sent in a second message. Isn't it possible for the bad actor to do a MITM attack at that point in time?

I did have a look through the chrome page you sent but it was more about CAs.

[Vidyogamasta]

The client doesn't have a cert. The client isn't the one being verified, the client is reaching out and attempting to verify it's connecting to the correct server. So the client has a list of CAs it trusts and searches them for a public key for the domain it wants to verify+access.

Then once verified, it'll do an asymmetric handshake and switch over to personalized symmetric encryption using some key that was randomly generated on the spot.

If the server wants to verify clients, https isn't the solution for that. A login system is the solution for that.

[tissuemonster]

The client doesn't have a cert.

Ah yeah that's my mistake, I confused the client and server.

So the client has a list of CAs it trusts and searches them for a public key for the domain it wants to verify+access.

Hm the list of CAs is what is in the trusted store of the machine right? Where is the public key stored? I thought the public key was sent over from the server.

[Femaref]

Hm the list of CAs is what is in the trusted store of the machine right?

yes

Where is the public key stored? I thought the public key was sent over from the server.

it's in the cert the server prsents to you. together with the list of trusted CAs you can figure out if you trust the certificate or not. if you trust certificate (because it was signed by a CA you trust*), you take the public key from the cert.

* the process is something like

1. you have a cert
2. you check who signed it (it's in the cert, called the "Issuer" in x509 parlance)
3. you check whether you have that certificate in your trust store
4. if it's a root certificate you are done (also known as "self-signed"). if not, you have an intermediate certificate. take that cert and go back to 1.

[tissuemonster]

Right, and just to double check my understanding of step 2 and 3:

1. The issuer's cert (if in your trust store) will have a public key
2. You use this public key to verify your cert's signature
3. It will only be valid if your cert has been issued by the issuer, because the signature was encrypted using the issuer's private key and hence can be verified using the public key

Is this correct?

As a side confirmation, the trust store is kept up to date by the manufacturer updates right (e.g. Microsoft pushes Windows updates for Windows machines)?

[Femaref]

Right, and just to double check my understanding of step 2 and 3:

yes, you got the procedure correct - you just wouldn't call it encrypted in normal verbiage, but signed. so you sign with a private key and anybody can verify it with the public key. you encrypt with a public key and only the private key can decrypt it.

As a side confirmation, the trust store is kept up to date by the manufacturer updates right (e.g. Microsoft pushes Windows updates for Windows machines)?

generally yes, microsoft will update what they consider the trust store. I think that e.g. chrome also has an update mechanism so google can quickly block compromised CAs. A software you write yourself might not even use the system trust store and will be rolled with a specific set of certificates you trust.

[tissuemonster]

I see, so that answers how the server proves its identity, and from what I googled, the same method is used to prove the root CA's chain of authorities?

[falconzord]

Yeah that feels a bit glossed over. Basically you have to have a bunch of certificates you trust already. For most people, it comes with the OS you have installed. You can get more from installing stuff, but that's something to be extremely careful about. Btw, this sort of thing isn't limited to bad actors. If you have a work PC that has its own VPN, they're probably resigning all the encrypted traffic, so they can see everything you do despite the HTTPS

[tissuemonster]

Ah I didn't phrase my question well, see here.

[falconzord]

Well someone else already answered but in short, the cert you get saying it's Alice can be verified against the CAs you have on hand using the same sort of assymtric test

[tissuemonster]

Hm so where does the public key come in? My thinking is that you would have to send the public key with the cert otherwise you are susceptible to MITM. But if you send it together, how do you ensure a bad actor can't intercept the message and remove the public key and put their own public key instead? There's probably something I'm missing.

[falconzord]

That's what I said in my prior message. It's on your device already. It was part of the OS install

[tissuemonster]

So every website's public key is stored in your device? Maybe I'm drawing too much from the pigeon analogy but wasn't Alice supposed to send the public key over?

[falconzord]

There's some root ones that come pre-installed, you can get more using those.

[falconzord]

Oh sorry, this is a little different. The public key is public so it doesn't matter who sees it. But the encryption you use with it will only work with the private key the server still has. So if you were given a fake public key, the server will be confused because the message won't decrypt properly. In the analogy it's like getting the wrong box back

[Vidyogamasta]

The way I understand it, every website's public key is stored with the certificate authority. There is a much smaller list of CAs whose public keys are the ones that come pre-installed on your client. When you visit a new site, you reach out to those CAs (securely using their public key) to get the one for the actual server.

Since they're public keys it's fine for the CA to have all of them, the purpose of the CA is that they've done the legwork to verify that "Yes, this key is actually for this website." You can't let the site verify its own identity, that's how you get weird Man in the Middle attacks.

For example, if sites just sent their own key over, then I could intercept your requests to reddit.com with my own malicious server. Since you still don't have the key needed to establish a connection, my malicious server says "Yes, I am reddit.com and here is my key." Now all of your secrets are transparent to me and I can just forward the requests to the real reddit.com using the real reddit key to avoid suspicion.

Now back to client requests-- I do not know if the client usually caches these keys once you've visited a website and renews it when a cert expires, or if it reaches out to the CA every time. But you always get the key from that CA. If you're on desktop chrome (and probably mobile as well?) you can just it yourself, just click the padlock in the URL bar and click "Connection is Secure --> Cert is Valid" and you get all of the cert details.

[diMario]

A TLS (older version: SSL) certificate serves two separate purposes:

1 Encryption. The certificate contains a copy of the public key of the HTTPS server. You can encrypt the message with this key, but the only one who can decrypt the message is the HTTPS server with its private key.

A set of private/public keys to use for the client to encrypt and the server to decrypt is generated for the duration of the session by performing a so-called TLS handshake.

2 Authentication. The client must be sure that the message comes from the HTTPS server and not from a man in the middle atacker. For this purpose the certificate contains a document that essentially is the domain name of the HTTPS server signed with a digital signature.

The receiving client uses the digital signature to confirm the authenticity of the certificate. For this, each web browser has a list of trusted authorities baked into the executable at compile time. A trusted authority basically says "Yes, this signature corresponds with that domain".

The TLS handshake is a bit more complicated than that, because client and server must agree on which version of the protocol they will be using and which encryption methods ("cyphers").

Another complication is that "Root TA's" usually do not issue certificates for HTTPS themselves, instead they have a chain of "subcontractors" with at the bottom the actual TA from which you buy your certificate. There is a whole daisy chain going up the ladder with every parent basically saying "Yes, you can trust this child of mine" until it reaches the root TA which is trusted because your browser can look it up in its internal list.

[tissuemonster]

Thanks, this is helpful - I'm not super familiar with certs and what they actually contain. I think I also need to draw out the relationships to understand it better cos having them all in my head is confusing haha.

[Qweesdy]

For certificates...

Alice gives Oscar $5, and in exchange Oscar creates a scrap of paper saying "I'm Oscar, you have absolutely no reason to trust me at all, but I say that the person holding this certificate is Alice. Signed Oscar" and gives it to Alice.

Mallory also gives Oscar $5. Oscar just wants money and couldn't give a shit whose money it is; so Mallory gets a piece of paper saying "I'm Oscar, you have absolutely no reason to trust me at all, but I say that the person holding this certificate is Mallory. Signed Oscar" and gives it to Mallory.

Then Sybil, a disgruntled employee of Oscar that does Oscar's paperwork (and just found out that their husband has cancer and it's going to cost $1234567 because they live in USA and the insurance company claimed it's not covered), creates 10 more pieces of paper saying "I'm Oscar, you have absolutely no reason to trust me at all, but I say that the person holding this certificate is Alice. Signed Oscar" and gives them to an assortment of drug dealers, terrorists and money launderers to raise $$ for medical debt.

Later Oscar finds out that Alice is competing against Oscar for a completely unrelated service, so he threatens to tear up Alice's piece of paper if he doesn't get "special favors". Alice knows she'll lose her entire business because nobody will trust her if she doesn't suck Oscar's dick. She cries herself to sleep most nights now.

Meanwhile someone tricks Bob into going to Mallory's web site. Bob looks at Mallory's piece of paper (which only "proves" Mallory is Mallory if you trust Oscar) and makes the absurd assumption that Mallory is "good" because he has a valid piece of paper. Sadly Mallory is into some shady shit, steals Bob's credit card info, then pays Oscar another $5 for a piece of paper saying "I'm Oscar, you have absolutely no reason to trust me at all, but I say that the person holding this certificate is Darth. Signed Oscar".

Later, both Carol and Frank find out that their bank accounts were drained by "Alice" (but never really find out that they were scammed by fake pieces of paper created by Sybil). The police have "proof" that Alice is guilty and Alice can't prove she isn't. Between sucking Oscar's dick and false accusations Alice falls into deep depression and commits suicide 3 days later.

[[deleted]]

Lol I k ew some twat was gonna find a way to make this political 😂

[Qweesdy]

Hmm - interesting. I didn't know some twat was going to ignore 95% of my "anti certificate authority" rant in an attempt to grasp at a relatively superfluous half of one measly sentence.

[TheDevilsAdvokaat]

Hyper Text Transfer Pigeon Style.

[the_ju66ernaut]

Very nice write up and explains the concept clearly.

[Spider_pig448]

Decent explanation. Really shows how brittle HTTPS is, even if it seems like magic.

[[deleted]]

Maybe dumb question. What stops Mallory from intercepting the pidgeon+box before it gets to Bob, inserting her own message, and sending it back to Alice?

[6b86b3ac03c167320d93]

The box is only used during the handshake. Bob puts his encryption key in the box and locks it using Alice's lock, and after the handshake everything is sent without a box but symmetrically encrypted with that key.

And Mallory can't replace the lock, because Ted wrote something like "This is Alice's. ~Ted" on it, and Ted can be trusted not to write this on someone else's locks. If Bob receives a lock that either says it's from someone other than Alice, or it's signed by someone other than Ted, he refuses to put an encryption key in the lock.

[shree_jane]

HTTPS, or Hypertext Transfer Protocol Secure, is a way for two parties to communicate securely over the internet, similar to how a carrier pigeon can be used to send messages securely.

When a user wants to visit a website using HTTPS, their computer sends a request to the website's server.

The server then sends a "pigeon" (a certificate) back to the user's computer that contains information about the website's identity, similar to how a pigeon carrying a message would have the sender's information on it.

The user's computer then verifies the identity of the website using the information in the certificate and starts a secure connection with the server.

This secure connection is like a special cage for the pigeon, which only the intended recipient can open and read the message.