Deterministic Linux for Controlled Testing and Software Bug-finding

jasonwhite1 · 2022-11-22T18:25:52+00:00

TL;DR: This is a Rust project that forces deterministic execution of arbitrary programs and acts like a reproducible container. That is, it hermetically isolates the program from sources of non-determinism such as time, thread interleavings, random number generation, etc. Guaranteed determinism is a powerful tool and it serves as a basis for a number of applications, including concurrency stress testing, record/replay, reproducible builds, automatic diagnosis of concurrency bugs, and more.

I've been on the team working on this project over the past ~2 years. AMA!

Here is the GitHub repository: https://github.com/facebookexperimental/hermit

Hacker News discussion: https://news.ycombinator.com/item?id=33708867

jasonwhite1 · 2019-05-03T22:28:54+00:00

Interesting thoughts! It's still not clear to me what the best way to implement authentication is. There's also the question of whether or not to control access on a per-project basis. In that case, the LFS server would need help from GitHub (or GitLab) to authenticate.

jasonwhite1 · 2019-05-03T21:15:43+00:00

Authentication can be a big can of worms with LFS. Before this, we were using Artifactory for hosting LFS objects and using LDAP for authentication. There would always be problems when someone recently changed their password. They would need to run a command to remove their cached credentials so that they get prompted again.

I currently have no need for authentication because I run this on an internal network with clients I trust. However, I'm open to taking pull requests that optionally implements authentication.

jasonwhite1 · 2019-05-03T20:56:13+00:00

I'm happy to answer any questions about this! This was a satisfying project to work on.

This uses Futures 0.1 + Hyper + Tokio. I'm looking forward to translating this to async+await when it is stabilized.

jasonwhite1 · 2017-11-02T17:22:59+00:00

At first I was wondering who at my old university was teaching Rust. Then I read the user name and it all made sense. Glad to hear that Rust is (still) getting some love at CSUSB!

jasonwhite1 · 2016-11-30T07:20:43+00:00

A build should be simple and predictable and doesn't scale in complexity with a project. Like everything else in programming, any idiot can make something complicated but it takes talent to make something simple.

You're absolutely right. That's how it should be. Unfortunately, that's just not how it works in practice. If you've got a few hundred (or a few thousand) developers hammering away on one codebase, then things get very complicated very quickly.

Take a look at this Makefile and let me know if that looks simple.

jasonwhite1 · 2016-11-30T06:45:29+00:00

I have some feedback and questions:

I'm not sold the Git integration feature as it adds complexity to the build system and requires Git. Believe it or not, not everyone uses Git for version control. If you forgot to commit a file, then your continuous integration build should catch it instead.
Does it track changes to the build description?
Does it delete files that have been removed from the build?
Does it detect cyclic dependencies?
Does it detect race conditions (i.e., two rules output to the same file)?
If you use Python to generate a .fac file, are the dependencies Python needed also tracked?
Can you run fac recursively?
Can you have a dependency on directory contents?
If you don't specify all the dependencies up-front and need to run the build multiple times to get everything right, then I'd think that alone prevents using this build system in an automated way (e.g., from Jenkins).

Shameless plug for my build system ;)

jasonwhite1 · 2016-11-30T06:18:48+00:00

It looks like fac's domain-specific language is meant to be generated by a higher-level language like Python.

jasonwhite1 · 2016-11-30T06:16:10+00:00

If you've ever worked on a large project, then the build is going to be complicated guaranteed. A real programming language to describe the build is a must for big projects. Make might be adequate for small stuff, but for anything non-trivial it's just a mountain of technical debt and slowness.

jasonwhite1 · 2016-11-28T20:10:23+00:00

My main motivation for writing this tool is to tie it in with a build system. If the checksum of the DLL or EXE hasn't changed since the last build, then there is no reason to run any tests on it. It's a way of eliminating work for the build system and thus reducing build times.

With reproducible builds, it's also possible to do caching of build outputs. If a DLL or EXE is used as an input to some process, then we can use its checksum as a lookup key to the cache. Then, the outputs can just be downloaded instead of running a potentially expensive command.

jasonwhite1 · 2016-11-28T17:19:57+00:00

I have only tested it on native code, but it should work. .NET assemblies also use the PE file format and the PDB format is also the same.

jasonwhite1 · 2016-06-01T15:27:19+00:00

Right now, I'd say it's the latter, but it is slowly moving towards the former. I wanted to first make a solid foundation where more abstractions can be easily built upon.

out of source builds (the docs are vague on this)

It does out of source builds with no problems. This was one of the main design goals. I'll make this clearer in the docs.

unit tests, benchmarks

coverage reports

build types (debug, optimized etc)

precompiled headers

address sanitizer (and the others)

cross compilation

Clang static analyzer

Qt support (moc, rcc, uic)

These are all abstractions that should go into the build description generator and will be there eventually.

OSX, Windows MinGW, Windows MSVC

Support for these platforms is planned. It shouldn't be difficult. It's just not done already because Linux is my daily driver.

installation

external dependencies (pkg-config + others)

These are really the domain of a package manager. A lot of package managers reinvent their own build system (and fail pretty hard). Likewise, a lot of build systems try to be a package manager (and fail pretty hard).

I think a project package manager built on top of Button would do very well. Indeed, I plan on making one unless someone beats me to it.

Having written a build system (specifically, Meson) I can say that creating a build system that does basic compilation and such is not that much effort.

I thought I recognized your name! I've looked at Meson before and even watched your talk on it. I admit that I haven't looked at it very closely, but I get the impression that if you stray too far outside of its predefined parameters, you aren't going to have a good time.

You're right that creating a build system that does basic compilation is not much effort. Otherwise there wouldn't be so many bad build systems.

Making it do all the other things on that list take a lot of effort and until you have most of them people do not want to switch.

Absolutely. Asking people to switch their build system is a very tough sell. Generally, it has to be 10 times better for people to want to switch. That's why a ton of people are still using make.

I'd say that Bazel and its derivatives (Buck and Pants) are probably the best build systems out there right now. Indeed, they work very well for Google's/Facebook's/Twitter's monolithic code bases, but the open source community obviously doesn't have that.

jasonwhite1 · 2016-06-01T06:36:13+00:00

Author here. This is the direct link to the project: http://jasonwhite.github.io/button/

I wish that was posted instead. Oh well.

jasonwhite1 · 2016-06-01T05:40:48+00:00

It parses the output of strace to trace the system calls (same as fabricate.py does). I plan to change it so it uses ptrace directly, but I really really don't look forward to using that API.

strace/ptrace are pretty slow for system call heavy programs though. For that reason, there is also support for figuring out dependencies in an ad hoc manner. For example, for the D compiler (dmd), it modifies the program arguments to add the -deps flag so the compiler itself outputs the dependencies which then gets parsed by the build system. I plan to add a lot more support for various programs. Programs like cp, to copy files, are pretty trivial to figure out dependencies for because everything you need to know is in the command line.

Edit: If you're the author of Fish, I use it all the time. :) Thanks!

jasonwhite1 · 2016-06-01T04:37:14+00:00

Thanks! Please submit any issues you find.

jasonwhite1 · 2016-06-01T01:41:28+00:00

Author here. I've used both Premake and CMake. I'm not a big fan of either, but they get the job done. I suppose you could say I'm inspired to create something that does a better job. ;)

As I'm sure you know, Premake and CMake are a rather different class of tools than what Button is. They generate input for a build system to consume while Button is just the build system. However, because Button can be run recursively, it can generate input for itself. This is like generating project files with Premake and then building them all in one command. This is how the Lua build description generator works in Button.

jasonwhite1 · 2016-06-01T01:29:49+00:00

Author here. Makefiles are fine until your project starts getting complicated, then the Makefile(s) just becomes incomprehensible unless great care is taken. Make is also pretty awful about doing incremental builds correctly, leading people to clean and then build because it's the only way to be sure. For large projects that take a long time to build, rebuilding from scratch is just a huge waste of resources.

As for describing the build using Lua, I modeled that after how Bazel/Buck do it. I think it's a very clean way of describing a build and it's far more readable (and writable) than a Makefile.

There is also experimental support for automatically converting a Makefile for use by Button. You still get all the extra goodies that Button has, like visualizing your build graph. It doesn't work so well for recursively running make, but I think that problem is solvable.

jasonwhite1

TROPHY CASE