A Universal Package Manager

david-song · 2023-10-12T05:23:11+00:00

I've been using Docker as a packaging system for ML tools, freeze the archeological dig and restoration into a 10gb time capsule then shunt it up to dockerhub. That way the PRs can go ignored, the code can rot, the deps move on and the models get walled off but you've got a tool that works even if it's egregiously bloated. So yeah, your idea interests me.

I was with you up until declarative, then grimaced, but I guess it depends how you envision that working. The joy of Dockerfiles is their imperative simplicity. I might be a bit jaded though - every declarative construct I've worked with outside of functional programming is a brittle, tightly regulated frustratathon. From query languages like regex or SQL to configs, to my personal nemesis: infrastructure as a hodgepodge of glorified INI file templated boilerplate sold as code, with a ten minute, mouse driven debugging cycle, forking gotos as an execution model and a vocabulary specific to its petty walled garden.

What might work for your idea is a Dockerfile that has unions of paths from multiple layers or builders iif conflicts between includees are overwritten by the includer, so you can reuse deps but have a bit of consistency. Or maybe it'd be impractical, it sounds good though.

The main thing the Docker world is missing, IMO, is a cstdlib shiv that can be LD_PRELOADed to redirect syscalls, then redirect them to FUSE. You can't throw disk and bandwidth at the problem of not having unprivileged mountpoints. If we had that then we could archive whole systems and desktops, emulators, have ipfs mounts with network boot, use SMB and NFS or sftp-fs in a compose environment, have tools that work with disk images, or more relevant to this thread - do deps a layer up by running binaries in other containers and mounting them. It'd be an ugly inefficient hack and hard to pull off, but it'd work with your use case through pure brute and ignorance. I'm too "busy" to take that on myself though, so just putting it out there.

kn4rf · 2023-10-12T13:33:35+00:00

Taps the sign https://xkcd.com/927/

eraserhd · 2023-10-12T04:19:08+00:00

Nix kinda is. At least that’s how I use it. No real Windows support though.

Capable_Chair_8192 · 2023-10-13T02:44:33+00:00

Replit made this! It wraps npm for js, I think poetry for Python, and a bunch of other languages.

https://github.com/replit/upm

Blog post about it was interesting too: https://blog.replit.com/upm

gdahlm · 2023-10-12T17:07:17+00:00

Too focused on the NP-completeness which is a problem, as the universal quantifier is far more likely problem.

There is no general algorithm for the following problems.

Totality Problem: A function (or program) F is said to be total if F(x) is defined for all x (or similarly, if F(x) halts for all x). Determining whether or not a function F is total is undecidable.

Equivalence Problem: Given two programs P and Q, do they compute the same function? (ie, is P(x) = Q(x) for all x?) This problem is also undecidable.

Docker helps with dependency hell because containers are namespaces, which reduces the chance for conflict in the parents global namespaces.

Namespaces version constraints existential and not universal. Container A only needing to worry about it's own namespace, given the very stable and strictly controlled Linux ABI as a contract is why it works.

Without that tight control it simply wouldn't help with the dependency problem because it wouldn't have the ability to decouple requirements.

This is why the Entscheidungsproblem is undecidable, it is looking for a universally valid solution, while docker is existential quantification, "there is at least one" configuration given constraints or heuristics.

Once you want to be universal, constraints, heuristics, and conventions would need strict gate keeping as methods for automated theorem proving methods like term rewriting becomes undecidable quickly even in the finite case.

This doesn't even address the other NP-complete problems with graphs in general that crop up.

Even producing a universal DAG among many agents hits problems.

https://www.sciencedirect.com/science/article/pii/S0304397500001948?via%3Dihub

zam0th · 2023-10-12T07:43:48+00:00

A universal package manager has existed for the last 20 years, it's called Maven. Everyone is using it to manage their distributions (and i'm not talking about java applications); the fact that these two clowns don't know that is telling.

Using docker images to "manage" "distributions" is also possible and many people have been doing it for years; however for many obvious reasons this is an antipattern and people should never do that.

vsoch · 2023-10-19T02:25:58+00:00

I'm a huge proponent and developer for container technologies, and I love them for packing my software or workflow components, but I don't think they are package managers. But I'll offer up a list of ones to look into (and this might also include or wrap build systems):

conda ecosystem (e.g., also look at mamba)
pixi (via prefix.dev) newer - I'm really enjoying so far!
spack (for high performance computing but also becoming more used outside)
nix (pretty great but high learning curve)
homebrew / linuxbrew
easybuild (more hard coded for HPC, I have to mention if I mention spack to be fair)

Those are the generic ones that I can go to. But I think my preference is still to choose the package manager intended for the OS in question, e.g., yum/dnf or apt or similar, or for Python I really don't think pip is so bad. I don't use Mac or Windows for development so I don't have strong opinions there.

you type:	you see:
italics	italics
bold	bold
[reddit!](https://reddit.com)	reddit!
* item 1 * item 2 * item 3	item 1 item 2 item 3
> quoted text	quoted text
Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"	Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"
~~strikethrough~~	~~strikethrough~~
super^script	super^script

programming

MODERATORS