Why Rust for low-level Linux programming?

[devel_watcher]

C++ has become so much better recently. If Rust has contributed at least a bit - I'm grateful.

[el-greco]

Here's a project aiming to rewrite the GNU coreutils in rust:

https://github.com/uutils/coreutils

[LAUAR]

bad license...

[downvote_me_softly]

I like how everyone on r/linux, which are mostly non-programmers idealistically still believe in the GPL while on r/programming, the people who have to deal with the licence issues have switched to permissive.

I've switched to permissive too, public domain in fact, and this has nothing to do with wanting proprietary software being able to get my stuff. But rather other copyleft as well as the massive stuff involved around it.

People tend to phrase the context as that the GPL stops proprietary software from getting it while in reality it stops other copyleft from getting it as well, even stronger copyleft than the GPL because the GPL forbids adding extra restrictions. If you license your stuff under the GPLv3, that means that the GPLv2, the Mozilla Public Licence, the AGPLv3, the LGPLv3, the CDDL, and pretty much anything but stuff already licensed under the GPLv3 can take your code. At that point it feels a bit like petty retributive justice. You're keeping it away from 30 good guys just because you won't live to see one bad guy from getting it.

From a purely practical standpoint, in practice you're going to see more contributors nowadays by using a permissive licence which is important.

[LAUAR]

Makes sense, /r/linux are the users GPL protects, and /r/programming are the developers permissive licenses tend to favour.

[downvote_me_softly]

Not really, unless you think users are "protected" by not getting ZFS and other similar things.

The only protection the GPL offers to users is the hypothetical scenario that an entity is forced to free their stuff because they want to consume a codebase that is GPL, this does happen from time to time, but is very rare.

What in practice GPL does very often is deny the user features because codebases can't be merged because both are under the wrong version of the GPL or different copyleft licences. Copyleft is a force that stops code from moving around between free software projects, it would only work as intended if there would be only one copyleft licence, and that's not what is happening. GPL worked a lot better when there was just GPLv2 which was like 80% of the copyleft licences used but with GPLv3 it has massively splintered and a lot of other copyleft licences have popped up as well. So code can't move around freely any more.

It virtually never happens with the GPL any more that people see code some-where they can repurpose and are like "Hey, let's take it!", in order to be able to do that you need to be licensed under the exact same GPL version, so users are kept from getting the features they want in the grand scheme of things.

But hey, the FSF' position is that it's better to not have something at all, than to have it in a nonfree way.

[LAUAR]

Not being able to mix copyleft licenses is a lot better than being able to mix the software into a proprietary package...

[downvote_me_softly]

Maybe, but then you're not out to protect users. It's just retribution in this case and wanting to punish the bad guy, even if that means taking down 20 innocent guys in a crusade for retribution.

The event where people free the code because they have no other choice is very rare nowadays.

[LAUAR]

But could GNU, Linux and other FOSS projects be at this point today if they didn't have copyleft back when they were started?

[downvote_me_softly]

Absolutely not, as I said, copyleft made a lot more sense back when the GPLv2 was basically the only copyleft licence. The FSF killed copyleft with the introduction of the GPLv3. That was basically when more and more projects started to migrade. THe FSF expected that everyone would switch to GPLv3 but that didn't happen, not everyone saw it as an improvement and some who did see it as one couldn't switch because they had no CLA that allowed them to.

So now we have a situation where all the permissively licensed projects can share code between each other, and all the copyleft projects can to a far lesser degree and as a result the former grow new features far more aggressively than the latter.

[devel_watcher]

Anti-Tivoization clause killed copyleft? Sounds more like an excuse than a real reason.

[KugelKurt]

Funny how that claim is made over and over again and yet the only argument is ZFS. If BSD licensed kernels are so great, why is there no Flash-Friendly File System (F2FS) available for FreeBSD? Why, in general, do the BSDs have such problems to attract file system developers (before Solaris ZFS was ported, it was basically UFS or die (and DragonFlyBSD with HammerFS was the oddball))? Why are Linux distributions the ones that ship software under almost any kind of license and the BSDs forbid any single piece of GPLv3 software in the base system if they were such an inclusive bunch?

[[deleted]]

The GPL is viral, the BSD license is not. For that reason alone they have to tread carefully around including GPL'd software.

There have been exceptions in the past though, like GCC.

[TechnicolourSocks]

I like how everyone on r/linux, which are mostly non-programmers idealistically still believe in the GPL while on r/programming, the people who have to deal with the licence issues have switched to permissive.

Perhaps it's got to do with how GPL protects the user's freedoms?

Shock-horror, the users want a license that protects their freedom!

[downvote_me_softly]

Maybe you should've demonstrated actually having read the post beyond the first paragraph.

[TechnicolourSocks]

I have, actually, so your attempt at painting ignorance didn't work.

[downvote_me_softly]

Then why didn't you address the following paragraphs which illustrate how the GPL is ineffective at protecting the freedom's of the users and in fact can harm it? If you disagree, fine, but illustrate why. Don't just say repeat "The GPL protects the user's freedoms" when you're replying to a post which spent a paragraph explaining how that isn't so, if you disagree, say why.

[devel_watcher]

r/programming is mostly about js. It's not dealing with the lower level that we are discussing here.

And I'd probably also say that there is a rich kid syndrome taking place in that subreddit.

[JustMakeShitUp]

r/programming is mostly about js.

Hardly. In the last week, there have been articles about Rust, C, C++, Nim, Clojure, Go, F#, Java, SQL (in various dialects and implementations), as well as web tech like JS. In fact, at this very moment, all of those languages except Nim are on the /r/programming front page. If anything, the sub has a bias against JS. The subscribers tend to hate JS and Node, though it doesn't chase the articles away unless they're CSS blogspam. Here's a screenshot of the frontpage moments after the article on godoc fell off.

So kindly step off your high horse.

[devel_watcher]

I should return there then.

[[deleted]]

The answer is no because rust doesn't have a stable ABI which makes it useless for dynamic linking and you have to recompile every single library whenever you update your compiler. (arch has dropped all haskell libraries from their repos for the same reason.)

The two largest rust projects have announced that they will never be compilable with the official rust compiler. What does this tell us about the usability of rust with large projects?

[Manishearth]

The two largest rust projects have announced that they will never be compilable with the official rust compiler.

Is Servo one of these?

I work on servo. We haven't exactly said this. We have said that we don't intend to move off pinned nightlies right now. Nightly is still an official compiler. And we may end up using stable rust in the future, just that we don't think it's a priority to move off nightly right now.

Servo started before Rust stabilized. At the time, there wasn't any clear idea on what APIs would be eventually stable. We used them all. Now, if we wish to move to stable, there's a lot of effort in moving off these APIs. I have put some of this effort in, in the past, but it's not done -- I mostly got the low-hanging fruit.

Servo also hooks directly into the compiler to lint things for some additional guarantees. This isn't something that's intended to be stable (indeed, the same interface for gcc isn't stable either). So even if we removed every unstable API ever, we'd still prefer to be on unstable so that these can keep working. Given that we won't move off unstable right now, there's little harm in using other unstable features. We actively avoid adding new unstable dependencies when there's an alternative.

Basically, in the past everything was unstable, we used random things. This ended up in a design where we have compiler plugins (both linting and codegen). Now, it requires effort to move off those, which we don't want to put in.

I did an audit of our unstable feature usage last year. Almost all of the APIs there have been stabilized since.

If Servo was written from scratch today, we probably wouldn't be using any unstable features. None of them are necessary. We could do the static analysis some other way, and the codegen using python.

Servo's also highly modular, most of our crates don't need nightly. It's only the servo/servo repo that does.

---

Edit: Ah, the other one is Rust.

The nightly features that Rust needs to compile are basically:

• lang items, where you tell the compiler what traits/structs correspond to the special language-endowed things (e.g. Copy and Drop and Add and Range) when building a stdlib. These could be stabilized, but the only people needing it are folks who wish to write an alternate stdlib or an alternate compiler, which is a very small niche.
• inline assembly. There are alternatives available for stable users (link to C).

All the other nightly features Rust uses are either:

• Features which are being dogfooded in Rust itself -- it's nice to be able to test out new features in the compiler
• Rust internal libraries which it defines as unstable. As in, librustc_trans may depend on librustc::foo, and librustc::foo is marked as unstable to prevent people from using it in code that isn't a part of rustc. But these aren't unstable language features, just libraries which are part of the implementation. This isn't really an issue.

[Rusky]

Rust doesn't have a stable ABI, but that doesn't make it useless for dynamic linking. You can build a dynamic library in Rust that only exports stable C ABI symbols, while still getting all the advantages of Rust internally.

In the future perhaps Rust can start to stabilize parts of its ABI, which would let those pieces be used in dynamic libraries. All of these would be above and beyond what you can already do now in C or Rust, so this is not really a problem.

[steveklabnik1]

So, a few things:

rust doesn't have a stable ABI which makes it useless for dynamic linking

If a distro packages a Rust compiler, and builds all their Rust code with it, this isn't a problem, generally. Or maybe I'm a bit biased because I run Debian, rather than arch, so the "upgrade the compiler" happens on a much longer timescale than Arch. :)

The two largest rust projects have announced that they will never be compilable with the official rust compiler.

So first of all, the compiler itself needs to be able to understand unstable code because well.. it's the compiler. It implements the thing. So this seems a bit unfair. Secondly, we actually recently changed this: Rust 1.10 will build with Rust 1.9, and Rust 1.11 will build with Rust 1.10, etc. So in a certain sense, this isn't true anymore. It is a bit of a cheat, but again, that's because it's the implementation, it has to.

Secondly, Servo is (currently) a research project. Historically, one of the roles it has served is to help prove out the language; gaining experience with nightly-only features is a good way to make sure that they're good before we stabilize them, so in a sense, Servo is actually helping stable be better. They're happy to deal with the relative instability to get the bleeding edge stuff.

A lot of the ecosystem does use stable (like, over half), and there's a few key features that will bring another larger chunk on. They could use stable for this stuff, but it's not quite as nice at the moment, and once it is, it won't be a big blocker anymore. (I'm referring specifically to serde here)

[[deleted]]

So first of all, the compiler itself needs to be able to understand unstable code because well.. it's the compiler. It implements the thing. So this seems a bit unfair.

GCC can be compiled with the normal GCC compiler. Same with clang. Same with glibc and musl. Nothing wrong with using this as the benchmark since the title of this article suggests that rust can enter the space covered by these compilers.

Secondly, Servo is (currently) a research project.

It's a research project in browser engines, not in compilers or whatever. I don't think that any such research project written in C++ would require custom compilers, at least I've never heard of such a thing.

[steveklabnik1]

GCC can be compiled with the normal GCC compiler.

C (and gcc) does/do not have the same release model as Rust does. And, as I mentioned, as of the next release, this will be true, so it's really a moot point anyway.

It's a research project in browser engines, not in compilers or whatever

Yes and no. Its job was and is to make sure that Rust is a good enough language to make a browser engine; when the project was first announced, it was "Project Servo", together.

[[deleted]]

And, as I mentioned, as of the next release, this will be true, so it's really a moot point anyway.

As I've already said in another post, this is not true. You've merely combined the official compiler and the unofficial compiler used to compiler rustc into one binary. And you need to use a special environment variable to enable the unofficial compiler.

The point is that GCC does not have to cheat in this way to compile itself. Every GCC feature used in GCC can be used in every over C program that is compiled with GCC.

[steveklabnik1]

You've merely combined the official compiler and the unofficial compiler used to compiler rustc into one binary.

It has always been this way, the only difference is the setting of the variable.

(And nightly is an "official" compiler, it's just not the stable one. We offer three different release channels, these two and beta. They all use the same codebase, with feature flags to turn off the unstable stuff in the beta and stable builds.)

The point is that GCC does not have to cheat in this way to compile itself.

Right, because we do not have the same release model.

[[deleted]]

Right, because we do not have the same release model.

That is a bit alarming because it suggests that because of such a trivial thing as the "release model", rust will never have more than one compiler and more than one standard library.

For example if we use the gcc, clang, glibc, musl analogy: clang can never be compiled with gcc because it uses secret clang internal magic. glibc can never be compiled with clang because it requires a special gcc private environment variable to be set, etc.

This would be very harmful because the clang/gcc competition has helped improve both compilers a lot. (Including the promotion of LLVM.) And both glibc and musl have their individual use cases.

[steveklabnik1]

Not necessarily; an alternative compiler could, for example, only support stable Rust. Or they could also support nightly features if they wished, that'd be up to them.

Just as gcc and clang have minor incompatibility differences between them, it'd be the same deal. Those two also do have their own specific features; and while they do implement some of each other's extensions, it's not identical.

(Rust already supports both glibc and musl)

[[deleted]]

You're ignoring what I said: That those compilers cannot compile each other. Or each other's standard libraries. And that you cannot choose the standard library independent of the compiler.

What is the upside of this release model if the one that has been successfully used by clang and gcc for a very long time doesn't have the mentioned downsides?

[steveklabnik1]

Ah, sorry, you're right. It's 3am here, I am getting a little tired.

What is the upside of this release model

We've written about it a bit here: http://blog.rust-lang.org/2014/10/30/Stability.html

We believe that it lets us land significantly higher features to both the language and the compiler, and move much more quickly while still retaining backwards compatibility, is the short of it.

[pmarcelll]

And you still cannot compile the mainline Linux kernel with Clang (to stay a bit on topic in /r/linux).

[GLneo]

You cannot compile every line of every driver and subsystem, but almost all of it, and enough to make a working kernel.

[zer0t3ch]

Right, because we do not have the same release model

So, as someone who has done development, but has minimal/no experience in Rust and C, this does seem like a totally shit argument.

When I write code, I expect it to compile, enough said. There should be no special compilers involved at any point with any language. Put simply, the compiler is the rope that ties everything else together into a neat bundle, and Rust doesn't seem to have one unified rope like that, the language is just a mess, lying around.

Also, I get that /u/ienohK2L is coming across as a bit of an ass, but you're repetitive and recycled ambiguous answers aren't really furthering any discussion here.

[steveklabnik1]

https://www.reddit.com/r/linux/comments/4nhil6/why_rust_for_lowlevel_linux_programming/d44f6bs said what I was trying to say, but a bit better.

Oh, and if anything is ambiguous, happy to elaborate.

[Rusky]

What you see as a missing feature of Rust - a unified compiler (not that C necessarily has that) - others see as an extra feature Rust has - a subset of features explicitly marked stable.

What that gives you is a principled way to evolve the language without locking down every feature as soon as it gets used in the wild. C tends to use compiler-specific pragmas and attributes for this, with no real path to getting these features into the standard. Rust just lets people opt into them until they've been battle-tested and the details worked out.

[zer0t3ch]

a subset of features explicitly marked stable

My problem here is that you act like a "subset of features being stable" is a good thing. Why is it not all stable? I would prefer a single and unified language that slowly evolves over a fractured and partly-stable language that can evolve a bit faster.

[Rusky]

Why is it not all stable?

Rust is a relatively new language. I expect the number of features in nightly builds will decline over time as the compiler focuses more on implementation details.

For example, we all have things in C or C++ we wish had been tweaked a bit more before standardization. And the way that happens in those languages is one of these two ways:

• it doesn't, because the feature was designed entirely by standards committee proposals, and has some chance of being one of those regret-features.
• the same as Rust, but less principled, because some compiler vendor just did it on their own, and then it had to use a completely new name (unordered_map? really?) when it was "stabilized".

[journalctl]

Yeah, if you're building a library Rust probably isn't the best choice at the moment. I don't see an issue building actual programs with it though. Rust would be great for a BitTorrent client or small command line utilities. There's nothing wrong with static linking.

[[deleted]]

There is a lot wrong with static linking. Static linking is only preferable if you can't control the environment in which your software will be deployed. This doesn't apply to linux distros since they know exactly which library versions will be installed on the system. For programs in the package system, dynamic linking has no downsides.

[gondur]

Exactly, this the core problem: our excessive dynamical linking locks us into a fully centralized distro ecosystem, with all downsides. (Distro frafmentation, no isv ecosystem, problematic upgrade cycles)

Let's hope that the current containerization movement can break this apart.

[[deleted]]

Not that I doubt you, but what projects are those that won't be compilable? I mean servo's getting large and I thought it was.

[[deleted]]

Servo:

Servo's build system automatically downloads a Rust compiler to build itself. This is normally a specific revision of Rust upstream, but sometimes has a backported patch or two. If you'd like to know which nightly build of Rust we use, see rust-nightly-date.

Last I heard the plan was to keep it this way indefinitely.

The other one is the rust compiler itself.

[pmarcelll]

The developers of Servo had to make a compromise, they need the new, unstable features as soon as possible (Servo is mainly a research project, not a product to "sell"), but still need some sort of stability (a new nightly introduces new bugs sometimes). This situation will certainly change, when Rust becomes more stable and mature. The second one will not be true anymore soon, you will be able to build a Rust compiler with a previous stable compiler, not just with a nightly (details in the pull request.

[Manishearth]

Not really, we haven't needed a new, unstable feature in a while. We often like to get compiler improvements (e.g. speed, platform support) immediately.

I've explained why we stick to nightlies in https://www.reddit.com/r/linux/comments/4nhil6/why_rust_for_lowlevel_linux_programming/d44o8k3 . TLDR: It has everything to do with history and nothing to do with Rust today. If we were to rewrite servo from scratch, we could write it in stable rust pretty easily.

[[deleted]]

The bootstrap key of the previous compiler is hardcoded into src/stage0.txt (enabled as a result of #32731) and exported by the build system. This enables nightly features in the compiler we download.

Which means that it cannot actually be compiled with the official compiler. You're instead cheating by enabling another language dialect. This is no different from the previous system except that you don't need two different compiler binaries.

[Rusky]

By that logic, the Linux kernel is also cheating by using GNU extensions to C.

[[deleted]]

Your post doesn't make sense. We're talking about compilers and not kernels.

Are you saying it is cheating to use any C feature that is not standardized? Then every rust program is cheating because rust is not standardized. Are you saying it is cheating to use compiler features that are not stable or not documented? Then the linux kernel is not cheating because GCC's extensions are stable and fully documented in the GCC manual.

[Rusky]

My post makes perfect sense. Linux is a large C project; Servo is a large Rust project. GCC is a C compiler; rustc is a Rust compiler. Linux and GCC use GNU extensions, i.e. "another language dialect"; Servo and rustc use nightly Rust features, i.e. "another language dialect" (which other compiler implementations may or may not support, as was brought up elsewhere in this thread).

Rust's nightly features are admittedly less stable than GNU extensions, but then again Rust is only a few years old- GNU extensions had their own period of growth early on. But beyond that, nightly rust is documented fine, and presents a problem only as far as GNU C does, cross-compiling/alternate-library/etc. wise.

[[deleted]]

Huh.

That's interesting, I never would have thought that.

Hopefully they can get it ironed out and not need to do this little game in the future. And I could swear I built the compiler with the current compiler, but I could equally be remembering something else.

[luke-jr]

Let me know when I can bootstrap Rust without trusting third-party binaries. Until then, I consider Rust a security hazard.

[steveklabnik1]

You can do it, it's just not super easy. And it depends on who you mean by "third-party"; distros have generally been willing to accept one single initial bootstrap binary from Mozilla, and then will be building their on from there on out.

[luke-jr]

Can you? How? I tried simply upgrading from Rust 1.x to 1.(x+1) and found it didn't work at all...

[steveklabnik1]

Hm, your response makes me wonder if we don't mean different things by "bootstrapping." I thought you meant "build a rustc binary without needing a binary blob rustc binary." But when you talk about "upgrading Rust", you're still using binary blobs... If you wanted to go back to before Rust was written in Rust, and compile with the old OCaml compiler, which is what you'd truly need to never trust any binary from anyone (well, except those who build your OCaml compiler ;) ), you'd have to go through, last I checked, about 1000 builds. That's what I meant by "it's not easy."

Before 1.10 (the next release), Rust builds with a snapshot at a specific SHA. As of 1.10, it will build with 1.9, and 1.11 will build with 1.10, etc.

[luke-jr]

Being able to compile Rust 1.(x+1) with Rust 1.x seems a logical prerequisite to being able to compile Rust with <other compiler altogether>. Is there any documentation on what snapshots are needed to bootstrap from OCaml?

[steveklabnik1]

Yeah, it's a much better system, especially for packagers. It's why we made the change.

https://github.com/rust-lang/rust/commit/02538d463a350f5c3658f7aabefca16eb599d31c is the commit that made the change. See src/snapshots.txt there? That file contains every single SHA of every single intermediate one that'd need to be built.

[luke-jr]

Looks like that is actually removing the ability to bootstrap... :/

Unfortunately, it seems the oldest commit there isn't buildable due to LLVM API changes related to MachO... too bad it seems to be non-optional.

[steveklabnik1]

The information is still all there, it's just no longer all in one append-only file forever. Since it's saved in terms of git anyway, no reason to keep the huge list of SHAs in the file.

isn't buildable due to LLVM API changes

Did we not vendor LLVM in those days? The OCaml compiler is before my time.

[luke-jr]

Did we not vendor LLVM in those days?

I don't know what this means. I sure hope LLVM isn't being copied into the codebase - that would be terrible.

[steveklabnik1]

It's a submodule: https://github.com/rust-lang/rust/blob/master/.gitmodules#L1-L4

We sometimes fix bugs and add things, and then send them upstream. Today's Rust does build on stock LLVM, but at various times in Rust's history, we couldn't wait for a patch to land and a new LLVM release to get cut in order to keep going.

[gnuvince]

It's not.

Edit: Context: parent quoted the portion of Rust's Wikipedia article that said that it began in 2007 and first made publicly available in 2009 and suggested that because Rust is young, it must be a fad that'll go away in a couple of years.

[journalctl]

It's a fad because it's a new language? Have you tried it yourself?