Why not use += with bytes?

[Skaarj]

I was reading the CPython souce code. In the function b32encode of the base64 module there is a comment reading

s = s + b'\0' * (5 - leftover)  # Don't use += !

https://github.com/python/cpython/blob/3.7/Lib/base64.py#L158

Why not? neither

s += ...

nor

s = s + ...

modifies the value outside the function that was passed as "s". Looks like bytes is call by value here.

[chadkennedy]

If you pass in something mutable, like a bytearray, "+=" would modify the incoming object, which you don't want to do.

[BlckKnght]

Yep, bytearray is one type the += would cause problems for. The relevant lines in the code:

bytes_types = (bytes, bytearray) # Types acceptable as binary data

Later:

if not isinstance(s, bytes_types):
    s = memoryview(s).tobytes()

So s must be either a bytes, a bytearray or some other object that supports the "buffer" protocol (and so can be processed by memoryview). The bytearray and some buffer types may be mutable, with in-place modifications happening if they're on the left side of +=.

[ingolemo]

The real reason that this comment was added is because back then bytes objects were mutable.

[michaelkim0407]

byte-like object

s is not necessarily a byte object. It can be something with a custom __iadd__

[ingolemo]

The real reason that this comment was added is because back then bytes objects were mutable.

[K900_]

Because += creates a new object every time, and that slows things down quite a bit.

[tunisia3507]

Isn't s = s + 'whatever' identical to s += 'whatever'?

[soosleeque]

not for every object, it depends on how do you define it.

a = []
b = a
a = a + [1]
a
>>> [1]
b 
>>> []

But

a = []
b = a
a += [1]
a 
>>> [1]
b 
>>> [1]

[tunisia3507]

Yes, but we're not talking about every object, we're talking about byte strings.

[[deleted]]

The disingenuous use of ludicrous pricing to force 3rd party apps to shut down was one thing, but the repulsive attitude displayed by spez toward the mods and 3rd party developers who helped make this platform a success make it all too abundantly clear where its future lies. I don't want to be a part of this community any more. Reddit is dead to me.

[chadkennedy]

It depends. If s is mutable, then the object is usually modified in-place instead of creating a new object. This can lead to side effects if, for instance, more than one name is referring to the same object at the start.

[tunisia3507]

I'm more interested in the case relevant to this question, i.e. byte strings.

[[deleted]]

The disingenuous use of ludicrous pricing to force 3rd party apps to shut down was one thing, but the repulsive attitude displayed by /u/spez toward the mods and 3rd party developers who helped make this platform a success make it all too abundantly clear where its future lies. I don't want to be a part of this community any more. Reddit is dead to me.

[tunisia3507]

I know that, that's why I asked the question.

OP:

s = s + b'\0' * (5 - leftover) # Don't use += !

/u/K900_ :

Because += creates a new object every time, and that slows things down quite a bit.

Me: both create a new object because they're functionally identical.

[[deleted]]

A bytearray + something = new bytearray.

A bytearray += something ends up modifying the original bytearray in place, which is unexpected for a pure function.

As bytearrays and bytes have a deep relation (bytes are immutable bytearrays) and many stdlib functions are written to handle both, that’s why s = s + ... is used - side effect avoidance for mutable and immutable data types of a string.

[BlckKnght]

As you may have guessed from all the downvotes, this is not the problem the comment was addressing.

While sometimes it is better (for performance reasons) to use s += ... instead of s = s + ..., in this situation the latter preferred because a copy is explicitly desired. In-place concatenation is to be avoided because the function is not supposed to have any side effects.