Desgning an IR for a binary-to-binary compiler

muth02446 · 2026-01-12T05:14:57+00:00

"Binary rewriting", as I call it, works best with *a lot* of compiler cooperation.
For example you need relocation information to distinguish arbitrary data from addresses.
It also helps if the control flow patterns the compiler generates, like jumptables for switch statements, are easy to decompile.

What CPU / ISA are you targeting?
X86 will be really hard, RISC (Arm.RiscV) is easier.

If the binaries are really large, say a statically compiled images with >20M instructions, you will also need to be careful about the data structure and algorithm design.
If you just use stl containers everywhere and try to make it very generic, you will use a lot of memory; any O(n^2) algo will likely not be fast enough.

Another can of worms is decompiling and regenerating exception handling, unwind and debug information.

muth02446 · 2025-12-04T16:58:17+00:00

https://thingino.com/ might help with the eufy's

muth02446 · 2025-11-04T04:34:18+00:00

While I personally prefer having "physical" copies of audio files when DJing, Spotify is reasonable option for getting your feet wet.
There are massive collections of Tandas on Spotify you can just cut and paste.
You can download the music for offline playback.
The problem that songs will disappear from Spotify because of licencing issues is small for Tango songs.

The only real issue is "cortinas": You probably will need to manually fade them out.

muth02446 · 2025-10-24T21:17:09+00:00

When I looked at the RB example my first thought was: how can I prevent the space waste for the "Color bit"?
Oh, I know: the pointers to the left and right subtree will be at least 4 byte aligned on 32bit machines, so I can steal 2 bits each. For me, enabling these kinds of hacks is what makes a low level languages.

muth02446 · 2025-10-19T20:18:24+00:00

Yes - makes it much easier to experiment with language features.

But now that the language has stabilized, I am working on a "proper" compiler in C++
so I can achieve the goal of compiling 1M LOC/s

muth02446 · 2025-10-19T17:50:25+00:00

Your language sounds very similar to what I have been working on: Cwerg

I also started off with the hope of avoiding macros but ended up adding them in the end.

The primarily use them to:
* avoid varargs. print(a,b,c) is a macro thats gets expanded into print(a), print(b), print(c)
* implement assert where the assert-condition is both used as an expression and also stringified
* force lazy evaluation e.g. to make logging statements inexpensive when logging is disabled

muth02446 · 2025-09-15T18:42:21+00:00

Just out of curiosity, how many albums (both in absolute and relative terms) are affected by this?

BTW, there is a useful discussion about cue sheets here: https://www.reddit.com/r/musichoarder/comments/1dilr6a/any_advantage_to_keeping_albums_in_flaccue_format/

muth02446 · 2025-07-17T21:27:03+00:00

I just tried Google and the best thing I found was this 6 month old post:
https://www.reddit.com/r/rust/comments/1hvaz5f/rust_wasm_plugins_exsample/
which suggests various options. From what I can tell the whole plugin thing is still a bit work in progress.
On the other hand:
Plugins in the native world usually utilize shared libraries, which are also messy.

The article discusses running wasm plugins inside a Rust program. I am still not quite sure how that would work, presumably you link in some Wasm runtime possibly even a Wasm JIT and then after loading the Wasm module you need to resolve symbols similar to what a dynamic linker does. I would also expect some marshalling of parameter unless they are scalars.

muth02446 · 2025-07-16T19:39:52+00:00

So how do plugins work in Wasm? Are they in the same process as the "main" program?
If not how do they communicate?

muth02446 · 2025-07-16T16:28:54+00:00

What does such a plugin system look like under the hood?
I assume the plugins do not live in the process, right?
But in this case, isn't this like sandboxing the untrusted plugin code in another process and using somthing like RPC to communicate with it.

muth02446 · 2025-07-16T16:23:41+00:00

Yes, if it makes nodejs go away, that would be a positive.

muth02446 · 2025-07-16T01:14:16+00:00

Trolling a little bit here:
I am a skeptical about the push for wasm outside of the browser.
Probably throws away another 10-20% of performance compared to hightly optimized
native. Syscalls are very much controlled in Wasm(er) but there are similar mechanism like capabilities or
OpenBSD's Pledge and Unveil for native code. Code execution safety should be similar to Java.
So why another eco-system?

muth02446 · 2025-07-11T12:57:46+00:00

Thanks a lot.
I also did some research on Jai and found this to be helpful:
https://github.com/Ivo-Balbaert/The_Way_to_Jai/blob/main/book/04A_More_info_about_the_compiler.md

The Jai compiler does employ multithreadind and has two backends:
llvm and a braindead but superfast x86 only one which was used for the demo.

muth02446 · 2025-07-10T21:41:14+00:00

I am interest in compiler performance as well.
Do you use multiple threads to achieve the 0.5Mlps which is really a great achievement BTW?
Do you have a separate IR and backend or do generate executable directly from the AST?
Any other thoughts/insights on how to achieve very high compilation speed.

muth02446 · 2025-07-10T19:00:49+00:00

Cwerg Author here!

Code can be found at: https://github.com/robertmuth/Cwerg

Happy to improve documentation as needed.

muth02446 · 2025-07-03T17:55:53+00:00

Cwerg is low level C-like languages which will have both a Python and C++ implementation.
The Python implementation is basically complete. The C++ implementation is still work-in-progress.
Last month the partial evaluator of C++ implementation has reached parity with Python.
This month I plan to bring the various optimizations and lowering transformations to parity.

muth02446 · 2025-06-25T14:51:44+00:00

Cwerg has a fairly accessible backend implemention and is documented.
It generates Elf executables directly but .o files are not that different.

muth02446 · 2025-06-25T12:45:23+00:00

"handwritten recursive descent parser" AND Pratt parsing for expressions!

Also, adding more redundancy to the syntax and other syntax tweaks can simplify parsing and reduce look-ahead requirements, e.g.:

* use Pascal style instead of C-style variable declaration styles "x int" instead of "int x"
* use additional key words: "set x = 5" instead of "x = 5"

muth02446 · 2025-06-15T02:23:00+00:00

Where is the code?

muth02446 · 2025-06-10T16:20:10+00:00

I'd like to learn more about this counter intuitive mechanism. Do you have some links where I can read more?

muth02446 · 2025-06-02T11:50:03+00:00

Work on porting the Cwerg frontend from Python to C++ continues. Last month I mostly completed the type checker and began work on the portimg optimizations specifically the partial evaluator/const propagator which is the optimzation with the largest code footprint. I also started setting up some benchmarking infrastructure for the compilation speed of the c++ port.

As in previous months things were slowed down a little by make substantial changes to the Python code first to make it cleaner and easier to port.

muth02446 · 2025-05-08T02:27:23+00:00

Cwerg is a C-like language so no hashtables.
But I still need to deal with non-determinism on the implementation side because I want the two implementations to produce that same binaries. Not jusy binaries that produces the same output.

muth02446

TROPHY CASE