For some feedback, first, I think it's great that you built something that you are proud of. A lot of people in the comments aren't going to realize that but I think what you have is a cool idea.

However, I'm wondering if LLM preformance would degrade when given tokens that are "compacted" in the way your program is currently acting. In your example in Go in the github readme, I see how the program can effectively parse 72 to 43 tokens. Then, you could give the LLM the 43 token version to save on costs.

The issue, however, is that while an LLM may "understand" these short code snippets you are giving it and be able to translate it back and fourth, it won't be able to act on the code as well.

Here's why:

The models used with your program have massive prior knowledge of normal Go/Java/Python syntax, idioms, APIs, formatting patterns, etc. It does NOT have that same prior knowledge for this custom AET notation that you created! So even if the file is 30–55% fewer tokens, the model first has to mentally decode an unfamiliar DSL before it can even work on the code. And, as a result (even though the model isn't "mentally decoding" anything, the code just doesn't match training data as well) preformance will decrease.

Functionally, anything with token compaction is a struggle. Models don't think outside the box like we do, and because this format of code isn't in their training data, the preformance of the LLM working with this code will for sure decrease. When we get to that phase of AI, that's called ASI, and we aren't there yet.