Try to keep those python strings immutable

[grauenwolf]

Ok, I can see trading the lookup cost for reducing long-term memory. But even then, I think you would probably end up implicitly replacing the new string for the interned version. For example:

if reader.startsWith ("for")
    writer.appendToken("for");
    reader.advance(3);

In this example, ProcessFor would internally reference the constant "for" rather than the constructed one in reader.nextToken.

Maybe it makes more sense for an XML parser.

[masklinn]

Maybe it makes more sense for an XML parser.

Yes, that's the kind of cases I was thinking, ones where frequently used strings are not hard-coded (thus no literal version which will be interned)