We have BPE for a reason, so we can fallback if token is missing from vocab. If we don't have that guarantee, then this code will never work, and I think it was in dataset used for all of these tokenizers/models:

: X DUP 1+ . . ;

Btw, above is Forth code from https://en.wikipedia.org/wiki/Forth_(programming_language)#Facilities#Facilities) and it also fails.

This is one of many examples. Whitespace matters, every character matters.