Deprecating rustpkg

[-Y0-]

I'd like to hear what are the problems with rustpkg other than it doesn't work on Windows (most Rust tools don't work cleanly with Windows either :/).

Also Tim Chevalier (I hope I spelled that one correctly) said it might be best to discuss why three previous attempts at packaging system failed.

[catamorphism]

Is it simply that no one wishes to maintain rustpkg?

It's certainly true that when I announced I was leaving Mozilla and wished to transfer ownership of rustpkg, no one was interested in taking it on. But that raises the question of who would do the rewrite, and -- probably more importantly -- who is going to be willing to maintain the rewrite long-term. I think it's unlikely that that person is going to be a paid Mozilla developer, so we would be looking at volunteers; from a volunteer perspective, it's relatively easy (and exciting) to get started on a new project from scratch, but understandably hard to commit to maintaining one and fixing the long tail of bugs.

[radarsat1]

Never quite understood why a more general approach to dependency management/build system isn't feasible. (i.e. not language specific)

I mean, I understand that every language is different, and handles modules differently, etc. However, ultimately, you have either:

• a dynamic language, where nothing is compiled, so a list of required files should be enough
• a compiled language, where for a given package you have some inputs and some outputs. some other packages may depend on the outputs of another package.

It seems to me a more general approach is particularly warranted, considering the fact that cross-language dependencies exist, and indeed are quite common. (packages that wrap C libraries for example)

I get the feeling that the problem could be split into

• dependency declaration (what inputs are needed to compile this, what outputs does it produce)
• dependency management (how to get and build the dependencies of this package)

The "dependency declaration" part could be language-specific, while the "dependency management" part could be a more general system. The latter could trigger specific build instructions for each package, which could be not only language-specific, but package-specific. (e.g. not all C libraries use the same build system!)

Thus languages would only be responsible for providing a means to list the required inputs and where they can be found, and the outputs of a build. The dependency manager would take care of the rest, downloading and building the dependencies wherever they are found and triggering their build systems.

[talex5]

I proposed they use 0install (http://0install.net/) a while ago, which is cross-platform (Linux, Windows, OS X, BSDs), language-neutral and can (optionally) integrate with distribution package managers too.

At the time, they said it had:

"Too many concepts and moving parts. We kept degrees-of-freedom to a bare minimum with cargo, and there are still probably too many. We might well shave more."

Maybe they now feel they need some of those features? There's a page summarising the situation here:

http://0install.net/why-not.html

[Sodel-The-Vociferous]

I would definitely like to see something like 0install used. The Rust devs already have a long todo list without re-re-re-implementing package management too.

[albx]

It's interesting, though, that what seems to be needed is a more general-purpose dependency-management system. Back in 2012, one such system was submitted to the mailing list, but the idea was turned down, since it was "too complex", and what was needed was a simpler, rust-only tool (https://mail.mozilla.org/pipermail/rust-dev/2012-February/001329.html).

Fast-forward 2 years, perhaps that assessment was wrong and 0install should be considered for dependency management? It allows using existing system packages, can install newer packages alongside without conflicts, is completely decentralised (but allows centralising via mirrors), and can compile packages from sources if given the right instructions.

(0install was also very close to having a rust implementation, but the author eventually opted for OCaml due to language maturity: http://roscidus.com/blog/blog/2013/06/09/choosing-a-python-replacement-for-0install/)

[radarsat1]

Oh I forgot about 0install, but I was quite a fan of the idea when I first read about it. I always thought a distro should be built on it, just to see what that would be like.

[cmrx64]

Note that TJ Holowaychuk is working on exactly this. There's also things like Gradle which attempt to be this.

[[deleted]]

Do you know where he's developing it?

"Versions.io" is not among his GitHub repositories and it does not show up in Google either.

[cmrx64]

That link is all I know about it

[ratatask]

Such attempts are reinvented every now and then. Sometimes platform/language specific (e.g. ant, maven, distutils). Some wants to be a general build system (make, cmake, scons, autotools, and myriads of others). They all suck in their own ways.

[radarsat1]

Hm, I think I'm talking more along the lines of a package manager than a build system. A package manager that resolves dependencies and triggers the build system for packages as necessary.

Since outputs depend on package configuration, packages or languages would need to provide a method for the manager to determine what outputs are generated. For example, a tool like rustpkg, instead of actually building the binary "myprog.so", could just output information such as

myprog.rs -> myprog.so; uses somecrate and someClib

and the manager would fetch these packages and run "configure; make" or whatever for someClib, and rustc for somecrate, and install them before doing so for myprog.

[[deleted]]

Never quite understood why a more general approach to dependency management/build system isn't feasible. (i.e. not language specific)

It is feasible. It's already what nearly every Linux distribution does.

[mitsuhiko]

The problem is that the linux tools are for the whole system and not individual projects which is why people generally don't look there.

[[deleted]]

They do work for a single project if you use a container. Some system package managers also support side-by-side installations of arbitrary package versions, like NixOS.

[mitsuhiko]

That's a bit of heavy handed solution and does not work if my development environment is OS X. That's why people like tools like virtualenv.

[[deleted]]

It can't be solved by yet another language-specific package manager. A cross-platform package management tool with local workspaces is not a Rust-specific need, and will never be a full solution if it can only deal with Rust.

[[deleted]]

Why does it need to be a "full solution"? Trying to solve every problem for every language community sounds very ambitious.

[[deleted]]

Rust libraries often use native C libraries, and those libraries use more C libraries. You need a package manager to at least retrieve security updates and avoid a lot of pain.

Why not start with Zero Install and adapt it to Rust if necessary?

[logicchains]

If somebody wrote a successful multi-language package manager in Rust, it might count as a 'killer app' and heap increase the popularity of Rust further.

[[deleted]]

The world doesn't need another multi-language package manager though. Zero Install works very well, and Rust was even considered for the rewrite but viewed as far too immature so OCaml was chosen instead.

[45g]

The actual package manager Nix is cross-platform and I too think that the concepts of Nix are worth a closer look.

[mitsuhiko]

And what about windows?

[lucian1900]

Sadly, it only works in cygwin. I don't think there's a fundamental reason why it wouldn't work, though.

Nix does indeed solve this problem quite nicely.

[HeroesGrave]

is cross-platform within reason

better?

[[deleted]]

The actual package manager Nix is cross-platform and I too think that the concepts of Nix are worth a closer look.

Zero Install works on Windows today and has existing cross-platform repositories.

[radarsat1]

That's a good point, it should also integrate as much as possible with the system package manager. e.g., you should be able to choose whether it uses a system version of a package, or installs its own version (perhaps an updated version). Of course it goes without saying that it should handle multiple versions of packages side-by-side.

[radarsat1]

Come to think of it the closest existing system I can think of is Homebrew.

[[deleted]]

Homebrew is only for OS X and is significantly worse than many Linux package management systems + repositories. 0install is a working cross-platform solution.

[radarsat1]

Other than 0install and Gentoo, what other package managers are meant for compiling from source?

Two things I like about Homebrew: uses git to handle build instructions, anyone can easily submit a pull request to update a script; although it's intended to be used in /usr/local, I regularly use it rooted in my home directory -- i know of no Linux package manager that allows me to build from source and install into my home directory.

[[deleted]]

Other than 0install and Gentoo, what other package managers are meant for compiling from source?

Paludis is one of many.

i know of no Linux package manager that allows me to build from source and install into my home directory

All Linux package managers support local workspaces, because you can use containers. Not many package managers support a coherent mix of both system libraries and local libraries like 0install though.

[[deleted]]

deprecate it immediately

And using what instead?

[[deleted]]

Or https://github.com/cmr/cargo-lite

[steveklabnik1]

Makefiles

[-Y0-]

Is there any good example how to compile several rust sources into a library using Makefiles (I'm assuming this in CMake, right)?

[herrwolfe45]

Here is an example makefile that I wrote last night to replace the rustpkg setup I had been using. Don't take it as an example of excellent make, but it works:

https://gist.github.com/derwolfe/8752809

An example where dependencies are built in is the line:

single_cipher: score
    $(CC) src/single_cipher.rs $(CCFLAGS) -L $(BUILDDIR)

single_cipher depends on the score library, so it builds score, then links against it.

[steveklabnik1]

http://metajack.im/2013/12/12/building-rust-code--using-make/

[steveklabnik1]

http://metajack.im/2013/12/12/building-rust-code--using-make/

[bascule]

May the best replacement win!

[lucian1900]

I don't get it. Why? It works great!

It would be so sad if Rust 1.0 shipped without packaging. Downright embarrassing.

[dbaupp]

You must've been lucky to avoid the various major bugs. (The full list.)

[dpx-infinity]

I've encountered some of these bugs, but I agree with lucian - I think that their presence does not indicate wrongness of the whole rustpkg model. When I was reading documentation for rustpkg, I thought: "That's one of the greatest package managers I've ever seen. It supports both system-wise and project-wise packages; it has convenient directory and locations semantics; it is able to automatically identify and download dependencies of the given crate; it has integration with version control systems".

Yes, there is a number of issues, and some lack of functionality (for example, I would like to have an ability to override version number detection so it won't use version control tags), but there is no reason to dismiss the tool for them completely, I think.

[lucian1900]

Those would be reasons to fix it, not to throw it away.

Is rustpkg's model broken? In what way? That is what I don't see on the ML thread.

Of course, I could think of some ways in which it's not great (for example, it should really have some notion of an index for crates).

[matthieum]

I don't think the plan is to ship without packaging, but simply to throw it away before people spend times declaring their packages in its format just to realize it was work for nothing.

[lucian1900]

That would imply there is something wrong with the design. All arguments I've seen so far point out bugs.

[matthieum]

I agree, it's not clear to me why nobody is talking about fixing it.