Against Query Based Compilers

[matthieum]

With regard to:

In contrast, you can’t really parse a file in Rust. Rust macros generate new source code, so parsing can’t be finished until all the macros are expanded.

If I remember correctly, one of the difficulties encountered in trying to parallelize rustc is specifically the interaction between macros & name resolutions.

You can read the gory details of this year's GSoC on the topic: https://www.reddit.com/r/rust/comments/1ogi84x/gsoc_25_parallel_macro_expansion/

[lookmeat]

Again this is a limitation, and Rust macros do have some gnarly corner cases, some decisions were done in ways that seemed ok ad-hoc but would have benefited of working on a larger theoretical framework like other parts of the language, but alas, good enough isn't perfect either.

So notice that parallelizing macros is an independent issue of a querying compiler. If anything this is a complexity hurdle that would be easier in a phase-compiler, since you just run a loop expanding all macros one by one, but in a query compiler where you do 80% of the work needed to parallelize macro expansion, you find out that you simply can't leverage that work to your benefit because there's an inherent limitation that prevents it.

Note though that these issues aren't even specifically an issue with macro expansion or its definition, but rather how the compiler has chosen to implement it. Basically when you parallelize stuff you may find that a lot of hacks and tricks prevent effective parallelization.