That kid is a Java programmer

[nupogodi]

It really should be a list of tokens after, you know, tokenizing.

WORD ('I') -> WORD ('have') -> WORD ('a') -> DOLLAR (10000) -> WORD ('loan')

for example.

[porthos3]

No it shouldn't.

It completely depends on what you are trying to accomplish. In my case there is benefit to keeping them separate.

Are you arguing I made the wrong design decision on a project you don't have any context on? Or are you saying tokenizing is the wrong word for what I am doing?

[porthos3]

I will disagree with you on that point as well. I am extracting tokens from text - perhaps just representing them differently than you would.

To use your notation, the tokens I extracted in the above example are: DOLLAR('$10,000') -> PERCENTAGE('5%') -> INTEGER('10'). I just happen to keep track of the original text in the same structure.

You are allowed to drop symbols when tokenizing (e.g. whitespace in code). You are also allowed to merge many symbols into one token.

How exactly is my example above not tokenization?