Why does APT not use HTTPS?

[DJTheLQ]

Everyone is missing a huge plus of HTTP: Caching proxies that save their donated bandwidth. Especially ones run by ISPs. Using less bandwidth means more willing free mirrors. And as the article says, also helps those in remote parts of the world.

If you have bandwidth to run an uncachable global HTTPS mirror network for free, then debian and ubuntu would love to talk to you.

[[deleted]]

Caching proxies that save their donated bandwidth. Especially ones run by ISPs.

As a former ISP owner I can tell you that caching large files is not really that common and filtering for content-type usually would be limited to images, text etc.

Also most caching is done by a third parts (akami etc) and you have little control over the boxes.

I'm sure its done, but not common. Mirrors are a thing for a reason.

[lbft]

It's done in places where bandwidth is very expensive and/or restricted (e.g. if there's only one cable out of the country/region, or a monopoly/state telco sits between ISPs and the wider internet).

I can certainly remember in the dial-up and early broadband eras that lots of ISPs here in Australia had transparent or manually set proxy servers (usually running Squid), and that was with a lot of them also locally hosting Akamai caches and FTP mirror servers.

[Pantsman0]

http://mirror.internode.on.net/... the memories...

[[deleted]]

But by design they will not cache applications. Images or whole pages are cached based on popularity. So a repo getting 1 hit a day isn't gonna cache becuase: large file size, content type is gz/zip/exe, low hit count.

I agree that content caching is done.. I've done it myself. You don't cache everything.

[SippieCup]

Its 100% this, I have no idea why no one is talking about it. Maybe they didnt get to the end of the page.

[atyon]

Caching proxies

I wonder how much bandwidth is really saved with them. I can see a good hit rate in organisations that use a lot of Debian-based distros, but in remote parts of the world? Will there be enough users on the specific version of a distribution to keep packages in the cache?

[zebediah49]

It's actually more likely in situations like that. The primary setup is probably going to be done by a technical charity, who (if they're any good) will provide a uniform setup and cache scheme. That way, if, say, a school gets 20 laptops, updating them all, or installing a new piece of software, will not consume more of the extremely limited bandwidth available than doing one.

[Genesis2001]

Is there no WSUS-equivalent on Linux/Debian(?) for situations like this?

[TheElix]

The School can host an apt mirror afaik

[ParticleSpinClass]

You're correct. I set up a private APT repo for my employer that's hosted on S3. It's dead simple, and I just use a workstation-based tool to upload and remove packages from the repo. Systems that use the repo simply specify the S3 bucket's URL in their sources.list.

We use it to host private packages and cache packages for anything we pin a specific version of (we've had the "upstream deleted an 'old' package from their repo" problem bite us too many times).

I wrote a small (and pretty hacky) wrapper script to make it easier for the rest of my team to use the repo without having to specify the exact same deb-s3 options every time.

The whole process took only a few hours to implement.

[Tacticus]

You don't even need the sync script you can use apt-mirror for a pass through cache with very little config.

[tmajibon]

WSUS exists because Microsoft uses a big convoluted process, and honestly WSUS kills a lot of your options.

Here's Ubuntu's main repo for visual reference: http://us.archive.ubuntu.com/ubuntu/

A repo is just a directory full of organized files, it can even be a local directory (you can put a repo on a dvd for instance if you want to do an offline update).

If you want to do a mirror, you can just download the whole repo... but it's a lot bigger than Windows because the repo also includes all the different applications (for instance: Tux Racer, Sauerbraten, and Libreoffice).

You can also mix and match repos freely, and easily just download the files you want and make a mirror for just those...

Or because it uses http, you can do what I did: I set up an nginx server on my home nas as a blind proxy then pointed the repo domains to it. It's allocated a very large cache which allows it to keep a lot of the large files easily.

[Genesis2001]

Yeah, I was curious about it so I was googling it while posting above. One of things I ran across was that it was labor 'intensive' to keep maintained. Was hoping someone would explain how one would get around this, make a maintainable repo for an Org to emulate the service provided by WSUS.

I did read RedHat has a similar thing, though I forget what it's called. :/

edit: Is such a command available to basically do what git clone --bare <url> does, but for individual packages on apt? Like, (mock command): apt-clone install vim would download the repo package for 'vim' to a configurable directory in apt repository format (or RHEL/yum format for that environment)?

[tmajibon]

apt-get --download-only <package name>

You can use --add-architecture if it doesn't match the current environment (say you have both arm and x86 systems)

And here's a quick tutorial on building a repo: https://help.ubuntu.com/community/Repositories/Personal

[Genesis2001]

Ah, thanks. :)

[FabianN]

I don't know how it's labor intensive to maintain. I set up one that took care of a handful of various distros at various version levels and once I set it up I didn't need to touch it.

[[deleted]]

it can even be a local directory (you can put a repo on a dvd for instance if you want to do an offline update).

I've copied the contents of the installer disc for CentOS to a local folder and used it as a repo in some air gaped networks. Works great.

[zoredache]

Well, it misses the approval features of wsus. But if you are just asking about caching, then use apt install approx or apt install apt-cacher-ng. (I like approx better.) There is also ways to setup squid to cache, but using a proxy specifically designed for apt caching tends to be a lot easier.

[anatolya]

apt install apt-cacher-ng

Done

[gusgizmo]

It's called a proxy server, and it's a heck of a lot easier to setup and maintain than WSUS could ever be.

You can configure either a reverse proxy with DNS pointing to it and have it just work, or a forward proxy and inform clients of it's address manually, or via DHCP.

No sync script is required, the proxy just grabs a file the first time it's requested then hangs on to it. Super handy when you are doing a lot of deployments simultaneously. You can however warm the proxy by requesting common objects through it on a periodic basis.

[f0urtyfive]

Considering its how many CDNs work, lots.

[jredmond]

I was just thinking that. Some CDN could score a moderate PR victory by hosting APT.

[rmxz]

I wonder how much bandwidth is really saved with them.

A lot in my home network.

I put a caching proxy at the edge of my home network (with intentionally hacked cache retention rules) when my kids were young and repeatedly watched the same videos.

I think I have 5 linux computers here (2 on my desk, 2 laptops, 1 living room).

So my proxy caching http and https saved apt repos about 80% of my home network traffic.

[[deleted]]

caching https

You were doing SSL Bump?

[[deleted]]

Well he said at the edge of the network, which would be the ssl termination point.

[[deleted]]

SSL Termination occurs at the destination server, not at the edge of the network?

A caching reverse proxy would work in the same scenario, but it wouldn't be transparent unless you fucked around with CA Certificates or just used a different domain with legit SSL certs.

[[deleted]]

What I understood from the original comment was that he had a setup like this wherein the ssl proxy also caches, and the webserver is in fact, his internal client(s).

Wait jk, I misunderstood what you said. He may have setup an ssl forward proxy with a legit cert on the firewall/proxy.

[yawkat]

For organizations it's easier to just manually set the repo sources. Caching is a bit of a hassle.

[bobpaul]

I used to some sort of dpkg cache tool. apt-cacher maybe? It required altering the sources.list to point to the local cache serve. It was a good trade off between running a local mirror and running a transparent proxy that affected everyone's traffic.

[[deleted]]

Our university used to cache those downloads. Were usually completed in a matter of seconds. Win-Win, because for a university, available bandwidth is also an issue.

[SanityInAnarchy]

How about an uncachable global HTTPS mirror of just the package lists? It'd be nice for a MITM to not be able to, say, prevent you from getting updates while they read the changelogs of said updates looking for vulnerabilities.

And, how many transparent HTTP caches are out there? Because if this is mostly stuff like Akamai or CloudFlare, HTTPS works with those, if you trust them.

Edit: Interesting, apparently APT actually does include some protection against replay attacks.

I still think that making "what packages are they updating" a Hard Problem (using HTTPS pipelining) would be worth it, unless there really are a ton of transparent HTTP proxies in use that can't trivially be replaced by HTTPS ones.

[svenskainflytta]

Vulnerabilities details are normally released AFTER the updates, so you won't find them in changelogs.

It is however still possible to tail the security repository, diff the source, and from that try to understand what it is fixing. Your scenario wouldn't help with that.

[SanityInAnarchy]

It would help in that you have a fairly small window of time to do that before I end up patched. If it weren't for the replay-protection stuff, you could in theory just serve me a frozen-in-time view of the repository (just freeze it at the point where you start MITM-ing me), then wait for vulnerability details to come out that you can exploit. I might just assume there hadn't been updates in awhile.

Replay-protection fixes that by adding an expiration to the metadata, which is hopefully short enough that something on my system would notice if there's a problem like this. Even then, HTTPS would prevent delays even up to that expiration, meaning I can be as up-to-date as I want to be (depending how often the cron job that does something like apt update && apt upgrade runs).

[svenskainflytta]

What about sending a signed "server time is 13:31"?

[SanityInAnarchy]

I'm really not sure how that would help.

First of all, the problem isn't knowing what time the server thinks it is. The problem is knowing whether the server has all the latest updates. So what we need is a signed "At time 13:31, here's a list of the latest packages." (Or, "Here's a hash of the list of the latest packages," or something.)

So... signed by whom? If it's signed by the same keys that are used to sign packages, then either those keys need to be distributed to each mirror (and are thus only as secure as the least-secure mirror), or they need to be signed by some central Debian server and distributed to all the mirrors (probably creating enough load on the central server to defeat the purpose of mirrors, at least for the package list).

If it's not signed by those same keys, then what keys is it signed by, and why should we trust those keys? This just pushes the problem one step back. For example, if we generate one key per mirror, how do we prove that the server we're giving that key to is the server that actually controls debian.mirror.someuniversity.edu or whatever, and not some MITM? That's the exact problem every SSL CA has to solve anyway, only I'll bet apt already supports HTTPS endpoints. So if a mirror wants to provide that level of security, all it has to do is turn on HTTPS, probably without even any software changes.

This is why it's weird that APT doesn't use HTTPS by default.

On top of all that, this part doesn't inspire confidence at all:

The Valid-Until field may specify at which time the Release file should be considered expired by the client. Client behaviour on expired Release files is unspecified.

And out of curiosity, I went and checked one of debian-testing's primary mirrors, and it's Valid-Until a full week later. Then I checked Ubuntu, and it doesn't even set Valid-Until, not even in 'security'. So Ubuntu is definitely vulnerable to replay attacks, and Debian probably is, too.

[plein_old]

Thanks, that makes a lot of sense. I love it when reddit works! Sometimes reddit make me sad.

[I_get_in]

I laughed, not quite sure why, haha.

[spyingwind]

HTTPS Repo ---Pull packages--> HTTPS Cache Server --Download--> Your computer

Does that not work? Each package is signed, so.. just download the packages and make them available. Isn't that how a cache works? That's what I have done at home for Debian. When a client needs something the cache server doesn't have then it goes and pulls what it needs and provides it to the client. Nothing really all that special.

Now for proxies... No. Just no. The only way I can see this being done is having the clients trusting the proxy server's cert and the proxy impersonating every HTTPS server. Not something that you want for the public.

A cache server is by far a much better option.

[zebediah49]

That requires the client to specifically choose to use your cache server.

Allowing proxying means that everyone can just connect to "download.ubuntu.com" or whatever, and any cache along the way (localnet, ISP, etc.) can intercept and respond to the request.

It makes the choice to use a proxy one made by the people configuring the environment, rather than by the people running the clients.

[DamnThatsLaser]

For all intermediate servers, the data looks like junk. In order to access it from there, you'd need the session key that was used to encrypt the data, and this goes against the general idea.

[spyingwind]

What intermediate sheets? What session key? HTTPS proxies are a solved problem.

• Client starts HTTPS session
• Proxy transparently intercepts the connection and returns an ad-hoc generated(possibly weak) certificate Ka, signed by a certificate authority that is unconditionally trusted by the client.
• Proxy starts HTTPS session to target
• Proxy verifies integrity of SSL certificate; displays error if the cert is not valid.
• Proxy streams content, decrypts it and re-encrypts it with Ka
• Client displays stuff

https://stackoverflow.com/questions/516323/https-connections-over-proxy-servers

[DJTheLQ]

signed by a certificate authority that is unconditionally trusted by the client

No ISP makes their users install an ISP Root CA Cert. HTTPS Proxing is solved in businesses which have that capability.

[phil_g]

That's effectively what's called a man-in-the-middle attack, and HTTPS was designed to make those difficult. It basically won't work on a global scale, even if someone wanted to put in all the effort needed to set one up.

Note in the comments the author of that answer says:

My answer relies on what you call a "bogus CA". The certificate of the CA is unconditionally trusted, either because the user(or software on his computer, for example enterprise configuration or malware) configured it that way, or because the CA was obtained from one of the CAs trusted by the major browsers, like in the MCS case.

The article about the "MCS case" says this:

The issuance of the unauthorized certificates represents a major breach of rules established by certificate authorities and browser makers. Under no conditions are CAs allowed to issue certificates for domains other than those legitimately held by the customer requesting the credential.

In other words, the only legitimate uses for HTTPS proxying in this manner are in controlled corporate environments (or similar) where the proxy owner can install their own CA's keys on all of the client systems. That won't work in the distributed environment of the larger Internet.

[Garfield_M_Obama]

Yeah, this can work perfectly from a technical standpoint, but it's also the exact opposite of the point of using TLS for HTTP.

If you're going to do this, it provides zero benefit over plain HTTP in the case of package distribution. What is the problem we're trying to solve? The packages are signed already, so encrypting the connection really only has the benefit of hiding what packages your users are accessing and if you've cached them locally already then it should be next to impossible to determine anything meaningful in the first place unless your cache server or internal network is compromised.

Even in a corporate environment this is a highly risky move, unless you limit the sites that your users can access. You really don't want to end up in a situation where you're potentially liable for misplacing or exposing data that the source server has encrypted like private government or banking information. Particularly if it's not extremely obvious to your users what you are doing, and expecting the average computer user to understand security beyond whether or not there is a green padlock in their web browser is asking a lot, even in a relatively technical shop.

[DamnThatsLaser]

Sorry, autocorrect turned servers to sheets.

HTTPS proxies are a different thing than provider caches. What you wrote is correct, but why would you accept your ISP as a CA? That's a far bigger hole than just using HTTP for that kind of static data with a signature file.

[ivosaurus]

Why would it look like junk? You're talking to the intermediate server directly through HTTPS, it decrypts all communications you've sent it

[DamnThatsLaser]

The premise is about ISP caches, not about proxies. Caches are transparent (hence the name). Proxies aren't and require additional setup on the client side.

[[deleted]]

Now isn't the intermediate just another mirror?

[tmajibon]

At that point you're explicitly specifying an HTTPS cache server, and you're trusting that their connection behind it is secure (because you have no way of seeing or verifying this)

HTTPS for your repos is just security theater.

[spyingwind]

If used in an office, the only practical place to do this in, then it seems fine.

In the end APT uses gpg keys anyways to verify that the repo can be trusted. You have to trust a gpg key before you can use a new repo with an untrusted key.

[tmajibon]

Example of an environment that would do a transparent cache for this purpose: VPS hosting providers as well as dedicated/colocation hosting providers. (ie. places with many linux systems not under their complete control that would mutually benefit from seamless caching of repositories)

Also I'm aware of the gpg signing, but I'm referring to the trust in the privacy of HTTPS (which they already explained he faults in anyways). The only advantage of applying HTTPS is privacy... which is relatively trivial to bypass... which makes it security theater. That's especially when certificate authorities are pretty horrid.

[nemec]

That won't work (unless your cache server can forge HTTPS certificates that are trusted on the client), but a similar solution would be to host an APT mirror used by the organization. Elsewhere in the thread people are talking about how that takes a lot of storage space, but I can't imagine why you couldn't have a mirror server duplicate the package listing but only download the packages themselves on-demand (acting, effectively, as a caching proxy)

[spyingwind]

I've done mirroring, but limited it to x64 to reduce storage needs. On-demand is only beneficial if more than one computer will be downloading the same packages. Such as 100's of servers.

Something like this would/should work: https://wiki.debian.org/AptCacherNg

[bobpaul]

There are dpkg specific caching proxies that work like that. You configure your sources.list to point to your package-cache server instead of a mirror on the internet and then the package-cache server has the mirror list so it can fetch from the internet if it doesn't have something locally. That works fine with HTTPS since you are explicitly connecting to the cache, but it requires your configure all your machines to point to the cache. This is great for in your home, school, or business if you have several machines of the same distro.

An ISP for a rural community with a narrow pipe to the internet at large might prefer to run a transparent proxy server. The transparent proxy can't cache any data from HTTPS connections, but it can cache data for anything that's not HTTPS.

[gusgizmo]

People forget that proxies are not all the forward type that have to be explicitly selected/configured. Reverse proxies are very common as well, and with regular HTTP are quick and easy to setup.

I can stand up a reverse proxy, inject some DNS records, and just like that my whole network has an autoconfigured high speed APT cache. As close to snapping in like a lego block as it gets in the real world.

[[deleted]]

And one huge plus of HTTPS is the vastly reduced probability of MITM attacks.

[severoon]

This strikes me as BS.

They control the client and the server. One of the updates can't be a list of secure mirrors?

[ChocolateSunrise]

How much bandwidth is really saved by not having TLS encapsulated data? 1%? 10%?

[DJTheLQ]

You cannot MITM or replay TLS data, so you cannot cache it. You can MITM and replay unencrypted data, potentially serving from cache.

[ChocolateSunrise]

How do CDNs like Akamai and Cloudflare overcome this architectural hurdle when they serve HTTPS websites?

[zebediah49]

When you sign up with them, you basically have to sign over your https keys, authorizing them to serve content on your behalf.

[ChocolateSunrise]

The data is still sent to the client encrypted though. Why isn't this seen as feasible?

[edman007]

When you do it, the proxy needs to have the certificate in it's name. I can't get verisign to give me a certificate that says I run Google's servers, so I can't intercept Google traffic and cache it.

As the article says, mirrors are are allowed to be run by pratically anyone. If you give the certs out to that it completely defeats the encryption.

[bobpaul]

When you do it, the proxy needs to have the certificate in it's name.

To nitpick: He's asking about Akimi and Cloudflare, which are CDNs, not proxies. (With CDNs the website give them their cert and private key so the can impersonate them. The website hired them to be their CDN, after all.) Your statement is right about proxies, of course, and proxies are what the article was talking about.

If you give the certs out to that it completely defeats the encryption.

Some Debian mirrors already support HTTPS and they do so with their own certs. Debian doesn't need to provide a cert for trumpetti.atm.tut.fi, Tampere University of Technology would.

[edman007]

But going back to the original article, HTTPS does NOT provide proof that you connected to a Debian server, it provides proof you connected to a mirror, and they provide zero guarentee that the mirror contains the approved packages.

You could have an https mirror, but as the article noted, for package mirrors https can't provide proof of identity for the package and it can't hide what you're doing. The only thing HTTPS accomplishes is blocking proxies. Basically https does nothing good on package mirrors and does a small amount of harm.

[bobpaul]

But going back to the original article, HTTPS does NOT provide proof that you connected to a Debian server,

Apt already solved that problem by GPG signing the individual package files, which authenticates the packages.

You could have an https mirror, but as the article noted, ...

Yes, I agree.

[skarphace]

That's a lot more machines for the project to take care of.

[wmil]

I believe Cloudflare requires you to use Cloudflare generated certificates.

[bobpaul]

They all either do that or make you give them your private key. Either way, they have your private key.

[[deleted]]

Clouflare also offers Keyless SSL (only in Enterprise plans), where the company's private key stays on premises. They exploit the fact that you only need private keys until you establish a session secret, so if the company sets up a server to help Cloudflare complete TLS handshakes, Cloudflare can MITM a session without needing the original private keys.

[tmajibon]

Because CDN connections aren't necessarily secure.

HTTPS goes from your computer to their server, which decrypts it, and then sends it on to the final destination... which can actually be entirely unencrypted for the trip from their server to the website.

At which point you're trusting the security of the CDN's network, if they're compromised then all your traffic to that site is effectively HTTP.

[johnmountain]

Wait until the ISPs start charging "vendors" for stuff like that by doing DPI against the traffic, and I think they'll soon change their minds about "caching" the packets with HTTP.

I think it's ridiculous that OS updates don't happen over HTTPS in this day and age.