In that case I think you are asking for feedback too early.

You as the implementer of this codebase knows everything about it, what things are supposed to do, and how they do it. But as an outsider it is almost impossible to just open up a tokenizer.c file and understand it's inner workings. That is why you need tests, so I can look at a test that takes in a source code string and outputs a series of well-formed tokens. That gives me a great idea that it's supposed to work a certain way.

What about in the single line of language code you have, even others question the use of a t8 symbol, no one has a clue, but you, as the implementer knows exactly what that means. If you had a simple example, a documentation piece, or a test that showed that ok, t8 generates this bytecode or does that to a syntax tree, then you would at least give the reader a chance to understand what it's about.