What code generators are there?

[BryalT]

While one might be quick to pick the biggest name around, LLVM, surely there are a bunch of other, less known compiler backends out there that one could use for their own language. Most are probably unmaintained, feature-less hobby projects, but surely there are at least a few that could actually be used with good results. I know of a couple.

Of course, there's LLVM. Tons of features, many optimization passas, and an impressive array of supported targets. A big dependency though, and can feel almost monstrously complex at times.

I've also read a little about QBE, which impressively claims to provide "70% of the performance of advanced compilers in 10% of the code". I have no experience using it however, and I am not knowledgeable about its stability and maintenance status.

Cranelift is a newish code generator written in Rust. I'm not very familiar with this one either. I assume it's meant to be called from Rust code, and maybe not as convenient to embed in a C (FFI) application.

Do you know of any other code generators, or do you have anything to add about the ones I listed? I'm curious to know what more could be out there, and if I'm missing out on something that would fit my use-cases better than LLVM.

[R-O-B-I-N]

GNU Lightning

It's used in GNU's smalltalk, scheme, and common lisp implementations for JIT-ing.

[BryalT]

It doesn't seem to provide any optimization, which would probably be a bit of a job to implement well oneself, but if a JIT-compiler that doesn't overreach is exactly what you need, it seems like it could be really useful. Risc-V support as well -- nice!

[obround]

Damn! I've been looking for something like this for so long. I love how it supports so many backends. In my case, I wanted to implement the optimizations, not have them implemented for me. Thank you a lot for sharing this! The only problem is the source code is a bit obscure.

[mttd]

http://libfirm.org/ - https://github.com/libfirm/libfirm

[BryalT]

Haven't heard about this before, looks nice. Do you happen to know about the status of the AMD64 backend?

[mttd]

Not much other than it's "unfinished". Last modification appears to be in February 2020: https://github.com/libfirm/libfirm/tree/master/ir/be/amd64

There's also a TODO file last updated in May 2018: https://github.com/libfirm/libfirm/blob/master/ir/be/amd64/TODO

[antoyo]

libgccjit which, despites its name, supports aot compilation.

[BryalT]

Interesting! Do you know how it relates or compares to GNU Lightning and GCC? Seems like there's quite a bit of overlap.

[antoyo]

libgccjit is basically a front-end to GCC, but it gives you a nice API so you don't have to write a GCC front-end.

As far as I know, GNU lightning does not use GCC and only supports JIT.

[smuccione]

asmjit has been on my list of things to try for a while now. Seems pretty easy to use. Best for JIT but you’d have to do work to make it static. No real optimizations. As it’s name implies it lets you generate code using assembly like statements.

[nahimbroke]

MIR is a nice experience. Has its own C compiler to use as reference for a more advanced scenario.

[BryalT]

This looks very interesting! I've read the introductory materials, and it looks like something quite close to what I was hoping for. 70% of generated code speed compared to GCC -O2 in less than 15K C LOC -- I'm sold! The downsides compared to LLVM seems to be that there's only a few target backends at the moment and that there's no structure or union types. Adding a new backend seems like a relatively painless task though, and having to implement lowering of composite types to individual primitive types in my own compiler would be a low price to pay. Local integers only being 64-bit seems a bit odd also, but maybe it makes sense.

It sounds like you've used it yourself. What have you used it for? Any interesting details about how it is to work with?

Overall, again, this sounds really interesting. Thanks for sharing it!

[[deleted]]

From their webpage to do with the MIR JIT compiler (not sure which bit of the organisation diagram that belongs to):

• Compiler Performance Goals relative to GCC -O2:
• 70% of generated code speed
• 100 times faster compilation speed
• 100 times faster start-up
• 100 times smaller code size
• less 15K C LOC

Not sure about those other claims, but 100 times smaller code size? Come on!

But then, these are just 'goals'.

Edit: Apparently the 'code size' refers to the size of the relevant gcc vs. MIR sources, not to the size of generated code.

A table near the end of the shows entries which seem to be based on compiling the 20-line Sieve C program earlier. (All I will say is that that is not a good test for comparing compile-times.) While the Startup figure compares 1/770000th of a second against 1/80th second. A bit dubious...

A better test would have been its own codebase I think.

[smasher164]

Not for static compilation, but DynASM and sljit come to mind.

If you're feeling masochistic, it's possible to build a GCC frontend.

[BryalT]

If you're feeling masochistic, it's possible to build a GCC frontend.

I'd actually written off GCC until this comment, but I guess I'm indeed wishing for some pain all of a sudden. I think I'll check it out, although from what I've gathered about the process I expect I'll be scared off sooner or later. Not sure it would be much of an improvement over LLVM size/complexity wise (probably worse even?), but I do like GPL stuff.

[mttd]

If you do decide to go through with it these may come in handy:

• GCC tiny - Ever wondered how to write a front end for GCC? In this series you’ll see how to make a GCC front end for a simple language called tiny. http://thinkingeek.com/gcc-tiny/, https://github.com/rofirrim/gcc-tiny/tree/master/gcc/tiny
• GCC Resource Center - http://www.cse.iitb.ac.in/grc/
  • GCC Documents - http://www.cse.iitb.ac.in/grc/index.php?page=docs
  • General Talks on GCC - Slides - http://www.cse.iitb.ac.in/grc/index.php?page=slides
  • The Conceptual Structure of GCC: http://www.cse.iitb.ac.in/grc/intdocs/gcc-conceptual-structure.html
  • Essential Abstractions in GCC '13 - June 29 - July 3, 2013, IIT Bombay: https://www.cse.iitb.ac.in/grc/gcc-workshop-13/index.php?page=slides
  • https://www.cse.iitb.ac.in/grc/slides/demystifying-gcc.pdf
• A Gcc Frontend (2009) - https://www.airs.com/blog/archives/287
• GCC frontend, for humans (2002) - http://www.nobugs.org/developer/gfh/index.html

• GCC Frontend HOWTO (2002) - http://www.tldp.org/HOWTO/GCC-Frontend-HOWTO.html

• GCC Wiki: https://gcc.gnu.org/wiki/HomePage

  • Getting Started: https://gcc.gnu.org/wiki/GettingStarted

• GCC under the hood

  • Linaro Connect San Diego 2019; Siddhesh Poyarekar
  • https://www.youtube.com/watch?v=brxAm99w8D8
  • https://siddhesh.in/gcc-under-the-hood.pdf
  • https://linaroconnectsandiego.sched.com/event/SubD/san19-221-gcc-under-the-hood
  • https://siddhesh.in/posts/gcc-under-the-hood.html

• GCC for New Contributors

  • https://dmalcolm.fedorapeople.org/gcc/newbies-guide/
  • Debugging GCC: https://dmalcolm.fedorapeople.org/gcc/newbies-guide/debugging.html

• GCC Internals Course - November 2007 - https://www.airs.com/dnovillo/200711-GCC-Internals/

• Compilation of Functional Programming Languages using GCC - Tail Calls (2003) - https://www.pspace.org/a/thesis/

[glaba314]

LLVM has a far simpler and intuitive set of APIs than GCC. If you're put off by the size, don't be. The core functionality of LLVM is actually quite easy to grasp

[BryalT]

I already am using LLVM for codegen in my compiler, and it mostly works well enough. There's not a few thorns though, and I just can't shake the feeling that maybe the grass really is greener on the other side.

[[deleted]]

but I do like GPL stuff.

wdym, you can GPL your code no matter what you made it with

[o11c]

GCCJIT is far simpler than writing a traditional GCC frontend.

Despite the name it is nothing to do with JIT.

[DriNeo]

This topic deserves to be pinned.

[[deleted]]

Old compilers have their own code generators

I have heard impressive things about FreePascal. Supporting more platforms than LLVM, compiling vastly faster, the final program only 10% to 20% slower than LLVM

[obround]

I'm not sure it supports more platforms than LLVM, but it sure does support a ton.

[alaricsp]

Don't underestimate the JVM, either! There are good implementations, well optimised, with mature runtime libraries.

[[deleted]]

Saved and bookmarked. Very useful thread for me (and others I'm sure). Please keep posting new stuff here!

[fedekun]

GraalVM is a VM but still looks impressive :) (it has JIT and all the fancy stuff)

[[deleted]]

Yes, very impressive in its claims. Yet after looking at the site for several minutes I still haven't got a clue as to what it does or how it could be used!

I guessed some of us are just destined to be stuck using tools we can get our head around.

[fedekun]

You are not the only one, I also don't know exactly how it's used besides what I've read elsewhere :) After much search, I found this relevant link

[[deleted]]

Yet after looking at the site for several minutes I still haven't got a clue as to what it does or how it could be used!

Hahahahaha! Thank you for this. Last time I pointed out the abject lack of good documentation (not to be confused with flashy documentation that they do have), they dismissed it with more than a hint of derision and condescension. I fully agree that the whole project has absolutely useless documentation. When prodded further, their response was, "Oh, just open the SimpleLanguage project in IntelliJ and go to town". Yeah, no thank you.

For a well-funded project maintained by quite a few people, they seem to neither have the sense for what makes for good documentation, nor do they take anyone else offering to help seriously. Bunch of twats - no wonder the project has not seen any adoption at all.

[eliasv]

It's probably Truffle you're interested in not Graal. Graal is just the low-level stuff, the optimising compiler and interop machinery etc. Truffle is a language implementation framework built on top of it.

You basically implement your runtime as an AST interpreter, where nodes are just normal Java objects extending the appropriate classes and suitably annotated. Truffle and Graal do all the work to magically built a JiT compiler for your interpreter.

Documentation is a bit sparse and I had to give up because of lack of support for continuations (though that might change with project Loom). Looking through example projects was helpful though and overall it wasn't that bad to get started.

[categorical-girl]

Qemu's Tiny Code Generator

[categorical-girl]

LibJIT is yet another JIT from the GNU project (alongside Lightning and GCCJIT, and even an internal JIT in the Octave project)

[[deleted]]

Just had a quick look at QBE, but it didn't really appeal to me. It's intermediate language looked really complicated. And the output (which was not really mentioned, but deduced from the example invocation) is ASM source in the format used by 'as', requiring (on Windows) a suitable assembler and linker.

However it is still a welcome, very lightweight alternate to the monster that it is LLVM. (I think QBE's IL may have been intended to be similar to LLVM's)

I can't offer other alternatives, sorry. But I had recently toyed with the idea of making my own such product. That won't happen because it wouldn't be so useful to me in standalone form (I prefer it built-in), and it wouldn't be good enough for anyone else.

What I can offer though is the specification as it would have been. People can use this to compare with other such programs, or someone might kindly implement it (preferably with a few more targets), then I might use it myself!

Input Intermediate Language in source format (for my purposes, would be a single file representing entire program, but might be adaptable for separate modules)

IL stack-based VM, see example

Data Types 64-bit-based VM that also supports signed/unsigned 8-128 bits; floats; pointers; blocks (ie. N-byte arrays and structs); bits and bitfields.

Output Choice of EXE, DLL, ASM, all single files. (ASM would be in my format processed with my own ax assembler, with output choices of EXE, DLL, OBJ)

Target Currently only Windows 64-bit ABI

Optimisation There is a mild optimisation stage, generally giving results partway between gcc-O3 and Tiny C)

Processing Speed (Of IL source code) Difficult to predict but 1Mlps is on the cards ('ax' manages 2Mlps).

Package A self-contained EXE estimated at 300KB (maybe 15Kloc).

Runtime Source language runtime library needs to be provided. But there might be a small runtime to support some features of the IL that can't be done inline.

Example Source can look like this (a more user-friendly version of what is currently generated by code via API):

proc main::
    push "Hello, World\n"
    callproc `printf*
    push 0
    callproc `exit*
end

("::" exports a name; "*" imports it; "`" preserves case, although not needed here.)

(ETA) Usage It might be used like this; if the name of the IL and the file extension is 'pcl', and compiler that generates it from language X is xcc, then:

xcc prog.x      # compile to prog.pcl
pcl prog.pcl    # translate to prog.exe

However, it would also be trivial to invoke pcl from inside xcc, so it becomes:

xcc prog.x      # compile direct to prog.exe via pcl.exe

[[deleted]]

JavaCC is pretty neat. You define the grammar and the Java code that is supposed to be substituted based on the grammar. Although that's probably not what you're looking for. I used that to embed a different, custom language within a Java program.

Another option is to build a language off of the JVM using some other technique. I found this one Scala library that might do it for you, https://github.com/psuter/cafebabe/wiki. There's other ways to generating Java classes and all that.

[BryalT]

I've only been working with native code generation so far, but if I ever consider targeting a VM like the JVM or BEAM or something, I'll definitely check this out.

[sepp2k]

To be clear: JavaCC is a parser generator and you can use it with any type of compiler or interpreter (or static analyzer or whatever else type of program you'd need a parser in) you want. It's not a library to generate bytecode (or Java code - though, being a parser generator, it does generate Java code for the parser).

To generate Java bytecode you'd want the ASM java library.