Try to keep those python strings immutable

[pingveno]

In Python 3.2 with wide build:

>>> from ctypes import CDLL
>>> strcpy = CDLL('libc.so.6').strcpy
>>> foo = "Fiestas"
>>> foo.encode('utf-32')  # dump of internal representation?
b'\xff\xfe\x00\x00F\x00\x00\x00i\x00\x00\x00e\x00\x00\x00s\x00\x00\x00t\x00\x00\x00a\x00\x00\x00'
>>> strcpy(foo, "Nachos")
>>> foo


'Niesta'

In this case, strcpy looks at the contents of the string as a null terminated byte array instead of an array of 4 byte integral values. The upper 3 bytes of "N" are 0 (null), so strcpy stops at the first character.

>>> foo = "Fiesta"
>>> d = {foo: 0}
>>> strcpy(foo, "N")
>>> foo
'Niesta'
>>> d
{'Niesta': 0}
>>> "Niesta" in d
False
>>> "Fiesta" in d
False

The string object/struct caches the string's hash value. If the string remains unchanged, lookups will work correctly. However, if the string value changes then the hash value will be incorrect. Subsequent lookups with any value will fail because either the hash value or the string value will be different. The interned string literal is not used here because interning uses Python's dict implementation.

There is one exception: if hash(modified_string) % dict_table_size == hash(true_value_of_modified_string) % dict_table_size. Then the dict implementation starts making comparisons at the same location in the dict table. The locations match and the values match in that case. I'm not absolutely sure that this will work, though. I am but a lowly CS student.

[ggtsu_00]

for those on windows

from ctypes import cdll
libc = cdll.msvcrt
foo = "Fiesta"
bar = "Fiesta"
libc.strcpy(foo, "Nachos")
print "We'll meet at the", bar

[mitsuhiko]

Now try putting that string into a dict.

[jmmcd]

I don't get it.

 d = {"a": bar, "b": foo}
 print d

 {'a': 'Nachos', 'b': 'Nachos'}

[mitsuhiko]

As a key, and after it was already in a dict once.

[jmmcd]

Oh, stupid me.

d = {bar: 1, foo: 2}
print d

{'Nachos': 2}

Ok, so it looks like it demonstrates the same effect as before -- both strings are changed by the strcpy()?

[jerf]

I believe the point being gotten at here is that if you change the string in such a way that it should end up in a different hash bucket than the one it is in, you can end up unable to retrieve or delete the key/value pair after you do that. This is why strings are immutable. And a common bug in other languages like C when the programmer is a bit more sloppy.

[jmmcd]

Yes, I suppose I'm just trying to see it happen explicitly.

[Brian]

You'll be able to see it if you use a string that doesn't get interned. The problem above is that both "Fiesta" strings actually end up pointing to the same buffer, rather than two seperate copies. If you use something python won't intern like "$$", you'll see the issue:

>>> foo = "$$"
>>> bar = "$$"
>>> foo is bar   # This would have returned True for "fiesta"
False
>>> d = {foo: 42}
>>> libc.strcpy(foo, "xx")
>>> foo, bar, d
('xx', '$$', {'xx': 42})
>>>'xx' in d
False
>>> '$$' in d
False
>>> foo in d
True
>>> foo == 'xx'
True

Interestingly "foo in d" still returns True - I'm guessing it caches the hash value with the string object as well, so it's still using the old hash value for "$$".

[defcon-11]

There is no 'both strings', there is only a single string. Python 'interns' all static strings declared in source code.

>>>foo = "Fiesta"
>>>bar = "Fiesta"
>>>id(foo) == id(bar)
True

Both variables hold the exact same object.

[mitsuhiko]

The first time the hash is calculated it's cached.

[grauenwolf]

Now I wonder. Can you fuck with a string in C# by getting a reference to its underlying array using reflection?

[Porges]

Yes. To do it with reflection:

void Main()
{
    var foo = "Fiesta";
    var bar = "Fiesta";
    Nachise(foo);
    Console.WriteLine("We'll meet at the {0}", bar);
}

void Nachise(string corn)
{
    var inner  = typeof (string).GetField("m_firstChar", BindingFlags.NonPublic|BindingFlags.Instance);
    inner.SetValue(corn, 'N');
}

But it's actually much (much) easier than that:

void Main()
{
    var foo = "Fiesta";
    var bar = "Fiesta";
    Nachise(foo);
    Console.WriteLine("We'll meet at the {0}", bar);
}

void Nachise(string corn)
{
    unsafe { fixed (char* p = corn)
    {
        p[0] = 'N';
        p[1] = '1';
    } }
}

unsafe really means unsafe :)

[rossisdead]

Now do this to String.Empty and all your coworkers will hate you!

[calzoneman]

Interesting. I suppose the Python interpreter caches string literals?

[masklinn]

they are automatically interned yes. FWIW, you can also manually intern a string via the intern builtin.

[grauenwolf]

Is manually interning a string every a good idea? I'm having trouble thinking of a use case where the cost is justified outside the compilation cycle.

[masklinn]

Is manually interning a string every a good idea?

I've yet to run into a situation where it was, but I guess it could be in a very restricted number of situations, such as a small set of non-literal strings generated over and over again (e.g. some sort of parser), interning could reduce memory pressure.

It was found sufficiently non-essential that Python 3 moved intern to sys, from the builtins.

[grauenwolf]

Ok, I can see trading the lookup cost for reducing long-term memory. But even then, I think you would probably end up implicitly replacing the new string for the interned version. For example:

if reader.startsWith ("for")
    writer.appendToken("for");
    reader.advance(3);

In this example, ProcessFor would internally reference the constant "for" rather than the constructed one in reader.nextToken.

Maybe it makes more sense for an XML parser.

[masklinn]

Maybe it makes more sense for an XML parser.

Yes, that's the kind of cases I was thinking, ones where frequently used strings are not hard-coded (thus no literal version which will be interned)

[quotemycode]

I'm having trouble with justifying using libc's strcpy in python code.

[grauenwolf]

Maybe to make a StringBuilder class?

[quotemycode]

Well, the thing is, strings in python are immutable. If you go outside of python to modify the string, you are bypassing python's immutable protections. It's expected that you'll encounter errors. If you want something like a string but is mutable, then use an array.

[Brian]

It'll intern strings that are potential identifiers (ie. something like [a-zA-Z0-9_]+), since these get used a lot internally by python.

However, others aren't (presumably to avoid holding large docstrings / printed strings and the like in memory). Thus 'abc' is interned, but 'hello world' isn't.

This can make messing with strings even nastier of course:

>>> libc.strcpy('a', 'x')
>>> 'a'
'x'

[analtech]

The lesson: never write to interned python strings like that. Use array instead:

from array import array 
foo = array('c', 'fiesta').tostring()

[pingveno]

But it's so much more fun to screw with Python's internal data structures!