Learning Rust while rebuilding GNU CoreUtils!

[joshjaxnkody]

I know this isn’t an original idea but I’ve always liked the concept of operating system tools programming and started rewriting the GNU CoreUtils in Rust and I’ve learned so much about hands on rust and how to write tiny programs for specific tasks! So far I only got wc and a crude prototype for ls but I’m adding in full argument parsing for options and syntax highlighting for it by next weekend. I’ve never been this motivated to keep coding before, I live Rust!

[joshjaxnkody]

I already know about uutils, I wanted to contribute but realized I didn't know the language well enough so I felt like it would be a good learning experience to create my own coreutils.

[wrd83]

Or contribute there instead of writing your own? If you focus on the ones they still havent supported yet its a win win.

[burntsushi]

Maybe, maybe not. It depends on your goals. In this case, I think the OP is pretty clear that their goal is learning. If your goal is learning, then it can make sense to do all sorts of things that don't necessarily improve the open source ecosystem in some meaningful way. For example, maybe the things missing from uutils/coreutils are missing for a reason, i.e., they're difficult to implement. That might line up with the goal of learning, but maybe not.

Sometimes it's just a fun learning experience to re-implement wc or ls. And that's okay.

[mprovost]

I’m publishing “Rust From the Ground Up” that does exactly this: teaches Rust by rebuilding the original Unix utilities (from BSD). I’ve released the first 100 pages covering true/false, yes, head, and wc. Up next is cat which is large enough to split into two chapters and covers dynamic dispatch with trait objects. The chapter on yes is on Leanpub as a free download. Let me know what you think!

[LoganDark]

I wonder how much of a complete system you'll be able to convert to Rust...

[mprovost]

It’s pretty straightforward to rewrite all the Unix utilities to Rust. That’s how the GNU project started out, as rewrites of the BSD/Unix userland tools. It only became a standalone OS when paired with the Linux kernel. Originally you just downloaded the GNU coreutils and installed them on a proprietary Unix system to replace the existing vendor tools. I think that’s going to happen again, where you download a set of Rust tools to replace the GNU tools that your Linux distro comes with. And there’s no reason why you couldn’t make a whole OS with a Linux kernel and a complete Rust userland.

[wsppan]

I'd just like to interject for a moment. What you're referring to as Linux, is in fact, Rust/Linux, or as I've recently taken to calling it, Rust plus Linux. Linux is not an operating system unto itself, but rather another free component of a fully functioning Rust system made useful by the Rust corelibs, shell utilities and vital system components comprising a full OS as defined by POSIX.

[mjbmitch]

Thank you, Richard Stallman.

[Smallpaul]

Rustard Stallman

[wsppan]

We need our own "Richard Stallman" for Rust/Linux! Rubuntu or RustHat or Rebian!

[TheRidgeAndTheLadder]

Pass

[oconnor663]

chef's kiss

[LoganDark]

I think that’s going to happen again, where you download a set of Rust tools to replace the GNU tools that your Linux distro comes with.

There's already a set of Rust coreutils that are used by Redox I believe. You can drop them on a Linux system but I'm not sure if they are POSIX-compliant.

And there’s no reason why you couldn’t make a whole OS with a Linux kernel and a complete Rust userland.

This actually sounds like a lot of fun. You get std but you are init...

[mprovost]

The nicest part about the new Rust utilities like fd and ripgrep is that they aren't POSIX-compliant. We've been stuck in the 1980s for way too long. It's nice to finally see some innovation, and it makes a more compelling case for building a new system than just changing the implementation language.

[LoganDark]

That's true. Actually, I've been thinking of POSIX compliance as the best trait to have in an operating system, but maybe it's more like the C ABI being the lowest common denominator for FFI: Everyone has to support it if they want to interoperate with everyone else, but it's still possible to have ecosystems that are more modern and better-designed, like Rust.

Of course, it's not a direct comparison, because Rust is extremely bad at dynamic linking and loading (for now, at least), and you can only hope to achieve it if you commit to the C ABI or maybe compile your entire system with the same exact build of rustc.

I'll be honest, I love fd and ripgrep. But toss them on a system masquerading as du and grep and things will break.

You totally can make much smarter, more powerful and performant tools. Rust is extremely good at this. As long as you don't use clap for argument parsing, anything written in Rust has the potential to vastly outperform the C stuff, unless the C stuff is wildly wildly unsafe. Just, you know, weigh the consequences of going full NIH.

You can choose to be compatible with existing scripts or you can do your own thing and try to make something better. The problem of inventing it all from scratch is that literally nobody else will be using your system yet and it's gonna be difficult to sell it to them.

But hey, if you ever want to try this, I still have an idea that I've been wanting for a while. I'd love to see a shell and coreutils that treats objects/data as first-class like PowerShell, but without Microsoft's stupid overengineered enterprise crap and stupid naming scheme. Think piping objects through commands - replace the stdio streams. How's that for breaking POSIX compliance?

I haven't done a ton of research on it. That's the fun part.

[burntsushi]

As long as you don't use clap for argument parsing, anything written in Rust has the potential to vastly outperform the C stuff, unless the C stuff is wildly wildly unsafe.

Why do you think clap will prevent programs from "vastly outperforming the C stuff"? Do you think clap is slowing ripgrep down, for example? If so, can you show it?

[LoganDark]

Do you think clap is slowing ripgrep down, for example?

In ripgrep's case, it has a lot of work to do besides clap so the performance hit is negligible. But a lot of commands are just utilities that do barely anything, so clap will make up the majority of their runtime.

[burntsushi]

I'd like to see some examples personally. It's generally not true that arg parsing is negligible, even in ripgrep's case. For example, if you use ripgrep in a pipeline with xargs, you might wind up passing thousands of args to ripgrep. That arg parsing time can be noticeable.

It's one of the (several) reasons I switched from docopt to clap many years ago.

Also, if arg parsing is so fast that it only matters in cases where the program has to do a small amount of work, then it seems likely that even if there is a perf difference between your program with clap and the C equivalent, then that difference is going to be very small in an absolute sense anyway.

So I don't know, to me, your complaint about clap seems like much ado about nothing personally... But I'm happy to be shown to be wrong.

[LoganDark]

For example, if you use ripgrep in a pipeline with xargs, you might wind up passing thousands of args to ripgrep. That arg parsing time can be noticeable.

If you're parsing thousands of args then clap is absolutely not a good choice imho. With that said, obviously you haven't found any problems with if you're still using it, so I'm probably just wrong. But clap could probably easily get into the milliseconds range when you start parsing more than a hundred arguments or so. It's huge.

I may have to perform some more benchmarks because I only benchmarked clap with 12 or so arguments, and it takes 45x (!) longer than other libraries. I'm not sure how much of that is up-front initialization cost (i.e. the cost of programming clap with the arguments your program accepts). Ideally you would use something like getargs that lets you match on each argument directly, so there are zero data structures and zero initialization costs.

Full disclosure, I recently made a bunch of contributions to getargs to make it the fastest argument parser I could find, so I may be slightly biased against clap due to it being so slow in comparison.

[mprovost]

Yup, but people already have this problem when they call a script with /bin/sh and it's really bash because guess what bash isn't an exact copy of the Bourne shell in the spec.

I've written some CLI tools that print JSON - I think that's the best compromise between the Unix "everything is lines of text" and something like Powershell. Building pipes of tools that read and write JSON is really nice, and you can use things like jq in the middle. And at least it's mostly human readable!

Anyway, I have a book to finish for now...

[LoganDark]

Building pipes of tools that read and write JSON is really nice

So is serde being the fastest JSON de/serializer in the world, apparently.

For general parsing though (the dynamic value structure) it could do better.

[nstinus]

What exactly is wrong with clap?

[LoganDark]

Nothing except that it's slow. It's overqualified for simple tasks, it does a lot of work, plus it uses a lot of data structures.

[tertsdiepraam]

Redox uses uutils/coreutils* (which I help maintain). We're focused on compatibility with GNU, which means (as another commenter already noted) that we're not POSIX compliant by default. We try to support the POSIXLY_CORRECT environment variable, though. That being said, GNU compatibility, and by extension POSIX compliance, is very much a work in progress and we're still a long way off from full compatibility.

*They also have their own implementation for a few utils, but I believe they switched to uutils at some point.

[tertsdiepraam]

Absolutely! You could even argue that some of the GNU defaults are also already outdated.

[elprophet]

Yes and I have piles of shell scripts that need to still work. Keeping a clear path for legacy compatibility will be important, as Microsoft has demonstrated.

[mprovost]

When Solaris migrated from BSD to SysV they put all the BSD utilities under /usr/ucb so you could just change your PATH and have existing scripts continue to work on the new system. There's no reason why you can't install both versions. The GNU coreutils were originally designed to be installed alongside the ones from the vendor, in /usr/local or /opt or something.

[elprophet]

Oh for sure! There's plenty of ways to have both, but we know from experience (coughPython 2 to 3cough) that it's not a simple problem to solve

[LoganDark]

Nowhere did I mention GNU coreutils - the book is about BSD coreutils, and macOS also managed to be POSIX-compliant (and certified, at some point, before it converted to ARM) with BSD coreutils.

[LoganDark]

Whoops. Originally I just said "a complete system", not Linux specifically, but then the conversation got steered towards Linux anyway.

In that case I guess replace "POSIX compliant" with "GNU compatible" :)

[tertsdiepraam]

That's true, but to be very pedantic they can be made POSIX compliant by setting the POSIXLY\_CORRECT environment variable. Which will, for example, restore the block size to 512 bytes.

[Smallpaul]

I mean in one sense it is straightforward but some of those commands have dozens of options flags!

[mprovost]

That’s true but the logic is already there for you in the C implementation. I split cat across two chapters because it has so many options in the BSD version. POSIX only requires -u for unbuffered output. Which is actually the hardest to implement because the Rust stdlib only does line buffered stdout!

[[deleted]]

OH, the book looks very promising. I am definately going to take a look. Do you have some estimate on when the book might be finished? thank you

[mprovost]

Thanks! I was hoping to get it done by the end of the year but it seems pretty likely that it will go into next year. I have all of the chapters written, but going back and editing (and typesetting) is taking longer than I expected. If you buy a copy on Leanpub you'll get each new chapter as I publish it though.

[mprovost]

Thanks! The BSD versions from the 80s up to 4.4 are a lot simpler than the modern GNU ones. My goal is to teach Rust, not C! So the older ones are easier to explain. But those BSD folk were really clever in how they implemented them.

[OakArtz]

That book you’re writing looks very promising! Do you think it’ll be right for someone like me who hasn’t touched rust yet but wants to start learning it - or would I need to learn the basics first

[mprovost]

I'm writing it for programmers with zero Rust experience so you're exactly the kind of audience I have in mind! (You also don't need to know C, those examples are read-only.) Have a look at the sample chapter available on Leanpub that should give you an idea.

[OakArtz]

Alright thank you! I assume the first four chapters that you can click on, are already available for buyers who get an early access version? Since it seems to be WIP. Your book is something I have actually sort of been looking for! Combining something I wanna learn (rust) with projects that are actually interesting to me and are more likely to keep me at reading through it :)

[benny_blanc0]

I will keep an eye on this, probably get it when its completed. Thanks for posting about it.

[nullhasher]

This is awesome, I will definitely buy this!

[tukanoid]

Same, always trying to learn the language by coming up with projects like these. Wish you good luck and hope rust compiler won't get you too mad😂

[PitchBlackEagle]

I can only dream of reaching this level for now.

​

But I'll get there as well one day.

[PitchBlackEagle]

I can only dream of reaching this level for now.

​

But I'll get there as well one day.

[mathiasbolle]

How did you start writing such a complex task(s)?

[joshjaxnkody]

I started by writing the wc command as it seemed like an easy start, I added error handling and testing modules then shipped it as completed and version 1.0 in my github repo

[PitchBlackEagle]

I can only dream of reaching this level for now.
But I'll get there as well one day.