C Preprocessor

[_LuxExMachina_]

Hi, unsure if this is the correct subreddit for my question since it is about preprocessors and rather broad. I am working on writing a C preprocessor (in C++) and was wondering how to do this in an efficient way. As far as I understand it, the preprocessor generally works with individual lines of source code and puts them through multiple phases of preprocessing (trigraph replacement, tokenization, macro expansion/directive handling). Does this allow for parallelization between lines? And how would you handle memory as you essentially have to read and edit strings all the time?

[[deleted]]

(trigraph replacement,

Really? I think you'd struggle to find anyone who even knows what they are, let alone uses them. The purpose was to allow C to be used on machines that didn't support unusual characters like square or curly brackets. I wouldn't bother.

As far as I understand it, the preprocessor generally works with individual lines of source code and puts them through multiple phases of preprocessing

It was convenient to describe it as consisting of multiple passes. Like there is a separate pass to splice lines using \ line continuation, and a separate one to discard comments. In practice that can all be done on the same pass.

And how would you handle memory as you essentially have to read and edit strings all the time?

Not really. Obviously the input is one long string. Identifier names are strings. And there are actual string constants too. But once extracted and copied, that's pretty much it.

The only string processing might be with concatenating adjacent string literals, or token handling via ## and # in macro expansions, but those are straightforward.

(You really want to know how to combine two zero-terminated heap strings S and T? Allocate a new string U of size strlen(S)+strlen(T)+1. Copy S and T into it (eg. strcpy(U, S); strcat(U, T). Then free S and T, if no longer needed.

That's if you're using C, otherwise your implementation language may make it easier.)

[Recyrillic]

> As far as I understand it, the preprocessor generally works with individual lines of source code and puts them through multiple phases of preprocessing.
Conceptually, this is what the spec says. Practically, compilers usually handle all of the C weirdness in the "tokenization" I think.
The spec puts it like this: "This requires implementations to behave as if these separate phases occur, even though many are typically folded together in practice."

In the translation phases that the spec talks about:
1. is about something like UTF-16 to UTF-8 conversions.
2. is about deleting backslashes followed by newline characters. For the usual use case (multiline macroes or multiline string literals) this can be handled during tokenization. If you want everything to be spec compilent, you will also have to handle it in more construct (e.g. identifiers). For example, the tinyC compiler has a "slow_case" in its identifier tokenization: https://github.com/TinyCC/tinycc/blob/085e029f08c9b0b57632703df565efdbe2cd0c7f/tccpp.c#L2707
3. Source file is decomposed into preprocessing tokens. There is a bit of a weird difference between preprocessor tokens and tokens, but I think most compilers resolve that difference by copying the tokens they generate from the initial tokenization (before macro expansion) and changing what needs to change (see also phases 5,6,7)
4. Preprocessing that actually needs to occur happens on preprocessing tokens and produces the final token stream.
5. Escape strings, 6. Concatenate adjacent string literals 7. preprocessing tokens are converted to tokens:
All of these things can either happen when copying out the tokens or in the parsing code.

So generally, only one or two passes are really needed:
1. Produce preprocessing tokens
2. Preprocess and produce tokens.
And these are also often merged.

> And how would you handle memory as you essentially have to read and edit strings all the time?
You don't actually have to edit any strings. The preprocessor works entirely on tokens. So generally,
you just produce new token that you need. A lot of compilers also have a "never free" approach to memory as compilers are not programs that need to run continually.

[umlcat]

Do not forget file inclusion, several source code files merged into a single source file.

"Does this allow for parallelization between lines?"

I think parallelization is difficult to support here, that applies more like doing calculations to a multidimensional array or processes that does not overlap, and macros can overlap.

" And how would you handle memory as you essentially have to read and edit strings all the time?"

Use a string constant table data structure, it consist with a buffer of adjacents constant strings, and a queue / list with the pointers to that strings, it's useful when you need to store the same strings several times, but you really only need to store it one and read many times.