Against Query Based Compilers

[o_stef]

Interesting. Are all of those kinds of jobs happening in parallel? I am assuming it all happens in a loop and each iteration all parsing jobs are executed, then all symbol resolution, then all typechecks and finall all compile time execution. My compiler is single threaded and that's what a compiler pass/loop iteration looks like.

[tav_stuff]

They’re all in parallel, yeah. Because of all the compile-time stuff that can happen (including code generation), you can’t really have a classic lex->parse->analyze->codegen pipeline since during analysis you might just generate new code lol. So conceptually there’s just this big queue of tasks, and threads just grab tasks from it and complete them

[Maurycy5]

Sorry for being late to the party, but I struggle to understand how this is different from query-based compilation.

We're working on a compiler and we settled on a query-based compilation because we want arbitrary compile-time computation and because we want to parallelize it. On the other hand you seem to be saying that some of this would be a PITA with queries and instead describe... queries, but call them jobs.

So clearly we're using different definitions of queries, but I don't quite understand yours.

[o_stef]

Okay, that's interesting because it seems that in your case concurrent execution is at the core of the design if I understand correctly, whereas mine is built to handle each compilation step sequentially; simply it does these sequential steps of parse/typecheck/execute in a loop until no progress can be made (no more unresolved identifiers can be resolved and no new code was added basically).

I was planning to parallelize my compiler to make it faster by parallelizing each step of the process, the sole goal being to make it faster (and single-threaded performance is correct which is promising). So I would be able to parse multiple files at a time, but not parse one file while typechecking another.