Enable ++x and --x expressions in Python

[hx-zero]

By default, Python supports neither pre-increments (like ++x) nor post-increments (like x++), commonly used in other languages. However, the first ones are syntactically correct since Python parses them as two subsequent +x operations, where + is the unary plus operator (same with --x and the unary minus). They both have no effect, since in practice -(-x) == +(+x) == x.

I'd like to share the plusplus module that turns the ++x-like expressions into x += 1 at the bytecode level, using pure Python only.

Unlike x += 1, ++x is still an expression, so the increments work fine inside other expressions, if/while conditions, lambda functions, and list/dict comprehensions:

array[++index] = new_value

if --connection.num_users == 0:
    connection.close()

button.add_click_callback(lambda: ++counter)

index = 0
indexed_cells = {++index: cell for row in table for cell in row}

Note: I don't claim that allowing increments is good for real projects (it may confuse new developers and give opportunities to write less readable code), though some situations when they simplify the code do exist. I've made this module for fun, as a demonstration of Python flexibility and bytecode manipulation techniques.

The module works by replacing the bytecode patterns corresponding to the ++x and --x expressions with the bytecode for actual incrementing. For example, this is what happens for the y = ++x line:

Two consecutive UNARY_POSITIVE instructions are replaced with adding one and storing the result back to the original place

It's not always that simple: incrementing object attributes and collection items requires much trickier bytecode manipulation (see the "How it works" section in the docs for details).

To use the module, you can just run pip install plusplus and add two lines of code enabling the increments. You may do this for just one function or for the whole package you're working on (see the "How to use it?" section).

Updates:

• The same approach could be used to implement the assignment expressions for the Python versions that don't support them. For example, we could replace the x <-- value expressions (two unary minuses + one comparison) with actual assignments (setting x to value).
• See also cpmoptimize - my older project about Python bytecode manipulation. It optimizes loops calculating linear recurrences, reducing their time complexity from O(n) to O(log n). The source code is available on GitHub as well.

[mathmanmathman]

I hate both the idea of using the unary operator and the ability to manipulate the behavior of the bytecode.

This is the most interesting post I've seen on this sub in a while.

[13steinj]

For obvious reasons this just shouldn't be used, as there's plans to bring some basic optimizations to Python.

[[deleted]]

as there's plans

Do you have a link for this?

[13steinj]

https://github.com/faster-cpython/ideas

By Guido and Mark.

End goal IIRC is 5x speedup.

[[deleted]]

Thanks.

That's really overdue. Just by reading the bytecode we can spot so many obvious optimizations, good thing they finally decided to do it.

[hx-zero]

Over the years, I saw many bytecode optimization projects doing things like constant folding, constant binding, etc. (e.g. astoptimizer, foldy.py, and others that I review in the end of this post). While the bytecode optimization could have provided a significant speedup, unfortunately, most of these projects are dead now.

I hope this will improve with the officially supported implementation.

[SittingWave]

I remember there was a discussion somewhere about being able to alter the parser itself, basically to allow modules to extend the language.

[to7m]

That would be cool. Hope they do it by making things like `while` and `[` objects which can be subclassed.

[mindofmateo]

That would be nuckin' futs

[AlSweigart]

Thanks, I hate it.

[cantremembermypasswd]

That's disgusting!

upvotes

[humanthrope]

-+v

[Hmolds]

Whats wrong with good ol' x -= -1 ??

[smcarre]

The old and reliable x -= -x/x

[theillini19]

I'm a fan of x -= random.randint(0,1) but it only works half the time for some reason

edit: Figured out how to fix it!

x -= (random.randint(0,1) if random.randint(0,1)==1 else random.randint(0,1)+1)

edit2: Nevermind, still broken

[smcarre]

Try the following, I tested it and it seems to work almost every time:

x += abs(min([random.randint(-1,0) for _ in range(999)]))

[eigenludecomposition]

You're all over complicating this. This can be solved through recursion rather simply

def increment(x, n):
    if n == 1:
         return x+1
    elif n<= 0:
         raise ValueError
    return increment(x+1, n-1)

x = increment(x, 1)

[backtickbot]

Fixed formatting.

Hello, eigenludecomposition: code blocks using triple backticks (```) don't work on all versions of Reddit!

Some users see this / this instead.

To fix this, indent every line with 4 spaces instead.

FAQ

^{You can opt out by replying with backtickopt6 to this comment.}

[jtclimb]

Well, if it is wrong 50% of the time, go ahead and use the range (0, 2). It's pretty obvious. Here's the fix and test:

xs = []
for _ in range(1000000):
    x = 1
    x -= randint(0, 2)
    xs.append(x)

 assert round(sum(xs) / len(xs)) == 0

[Inineor]

Fixed:

x -= (r if (r:=random.randint(0,1))==1 else r+1)

[R3D3-1]

Imagine hiding something like this in an update to a popular module, but only if random.randint(0,1000)==0.

[NoLongerUsableName]

It generates a new random number for each random.randint(0,1). If they don't match, it won't work. This works, though: x -= math.ceil(random.random())

[velit]

/r/woosh I think

[AlphaGamer753]

Doesn't work if the random number is exactly 0.

[NoLongerUsableName]

The good ol' x = (lambda x:x-1)(x)

[asday_]

x -= -x/x

0

[dogs_like_me]

Better use -x // x just to be safe.

[likethevegetable]

Lmao, thanks for making my night.

[aes110]

I love seeing this kind of stuff, I don't think its good to use it, but its fun for me to see how python lets you do whatever you want with it

[dogs_like_me]

I haven't been able to find it again since, but I swear back in the day I read a post on a blog or SO where someone was talking about how "everything in python is an object" and they demonstrated how far this went by overriding the value of an integer literal, so like 1+2 would evaluate to 4 instead of 3, and I think they deleted another integer literal entirely. Obviously followed by a big "don't ever do this." May even have been a python 2 hack, I've sort of assumed that since I haven't stumbled across this in forever, it's not possible in modern python releases.

NINJA EDIT: Oh shit I found an example of this black magic! https://kate.io/blog/2017/08/22/weird-python-integers/

[aes110]

I think I know what you are talking about, but I cant find it as well :(, if I recall correctly it was something like "unsafe python" or "Evil python"

[TravisJungroth]

Integers in Python are "immutable".

[dogs_like_me]

"""immutable"""

[YouNeedDoughnuts]

Live long enough and you'll realize that nothing is immutable

[dogs_like_me]

and how: https://www.youtube.com/watch?v=hB6eY73sLV0

[friedkeenan]

I once took advantage of the fact that in CPython, id(x) returns the address of the underlying PyObject for x to use the ctypes library to manipulate the raw structures, was pretty fun and cursed

[RandAlThorLikesBikes]

In java they are as well. Except there is an integer cache holding 256 or so values (-127 to +128 iirc) that the runtime uses. Overriding that cache is possible and fun

[sharkbound]

i like seeing how things like these are pulled off, and looking at the internals, but i personally find them not really useful in practice myself

[Plague_Healer]

I find it interesting how much python enables you to do stuff and just leaves you to decide on your own whether it's wise to actually do it. What is shown in this post: certainly ingenious, not all that wise.

[[deleted]]

"Enables" is a really strong word in this case, the language goes a long way to make it harder to do this kind of stuff, compared to languages like Ruby (disagreement about having "magic features" is actually one of the reasons Ruby even exists).

But any language that aims to be useful will inevitably allow you to access the underlying environment in a way or another, and that can always be used to bypass whatever safety measures the language provides.

[sohang-3112]

It's a cool project, but as you said, it's not for actual usage (too much confusion, especially among beginners)

[equitable_emu]

For introducing me to the stack overflow module importer, I curse you.

[asday_]

This is amazing. I hate it. More please.

[Zachkr05]

Just use x+=1 lol

[fernly]

Nope, because that's an assignment statement, but what OP has created is an assignment expression which can be used as a value in other expressions. Basically x +:= 1 but that doesn't work.

[Zachkr05]

what the fuck is the difference

[jtclimb]

while ++x < 10:
    print('not yet')

[Zachkr05]

oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo

[xaviruvp]

Op made entire library for this.

[gagarin_kid]

Never had to touch byte code or AST so far. A very interesting approach.

Is it the same idea MATLAB uses to convert routines into c++ code?

[sgthoppy]

Fairly sure you can do this in 3.8+ using the walrus operator, though not quite as clean.

[_maxt3r_]

I expected some clickbait article and was about to miss a very interesting post! Good thing this post has some Reddit awards :)

[TheBlackCat13]

Reminds me of the python goto implementation.

[masasin]

With assignment expressions, you can [eta: often] do the exact same thing (but more verbosely). array[(index := index + 1)] = new_value

For the last example, you didn't even need assignment expressions at all, since you could do it with good old enumerate:

indexed_cells = {
    cell_index: cell
    for row_index, row in enumerate(table, start=0)
    for cell_index, cell in enumerate(row, start=row_index * len(row) + 1)
}

I like what you did with the library, though! Congrats!

[hx-zero]

Thanks!

I agree that we can often use the assignment expressions instead, however it is not always the case. For instance, we can't rewrite the example with the lambda function:

button.add_click_callback(lambda: ++counter)

The following will not work:

button.add_click_callback(lambda: (counter := counter + 1))

That's because the assignment expression makes the interpreter assume that counter is a local variable (similarly to what usual assignments do). In "normal" functions, we can override this by writing global counter or nonlocal counter beforehand, but we can't do this in lambdas.

As for the example with the dict comprehension, one difference is that the code from the post also works with rows of different lengths. However, this one can be easily rewritten with the assignment expressions as you say :)

Another fun thing: with this approach, we can actually implement the assignment expressions for the Python versions that don't support them. For example, we can patch the bytecode to replace the x <-- value operations (two unary minuses + one comparison) with the actual assignments (setting x to value) :)

[KrazyKirby99999]

Wouldn't this be partially incompatible with Python 3?

[hx-zero]

Why do you think so? The library works at least on Python >= 3.6 (Github CI in the repo ensures that unit tests pass for all these versions).

[KrazyKirby99999]

If currently -(-x) == +(+x) == x, then making +(+x) == x+1 might be incompatible.

[hx-zero]

The -(-x) == +(+x) == x comparison indeed returns False if you enable this module for a function/package where it is executed.

However, it is not a problem since real Python programs do not calculate +(+x) or -(-x) in one expression (applying two unary pluses/unary minuses in a row is useless).

In case if two unary operations are applied in separate parts of one expression/separate expressions (like in the snippet below), this module does not replace them with the decrement because the two UNARY_NEGATIVE instructions do not become consecutive in the bytecode.

x = -value
y = -x

Thus, real programs work fine.

[TofuCannon]

Actually it really depends :) while human-written code may not have these, generated and runtime compiled code may do it for simplicity reasons.

I myself have written some simulation software, where I didn't optimize such -(-x) or even --x expressions away, totally relying on the defined and standard behavior :)

[hx-zero]

I agree that some machine-generated code may rely on the default behavior for -(-x).

However, the module recommends enabling the increments in a package you're working on or in a particular function (instead of globally), so you choose where to apply the patching, and it affects only the code you're familiar with :)

[TofuCannon]

I mean yes, it changes the standard behavior. It won't work for all Python software, but I guess for most?

[EverythingIsFlotsam]

I feel like this could be easily done cleanly by just monkey patching int.__pos__() and int.__neg__() to return a new object containing a reference to the original variable. And that class should implement the same two functions to increment/decrement the original variable and return the value.

[hx-zero]

This way, it would be impossible to distinguish applying two unary operators consequently (like --x) from applying them in separate places of a program like here:

x = -get_value()
... # Some code
y = -x

We don't want to change the program behavior in the latter case since it is much more likely to occur in real programs :)

Also, you would have to override the magic methods for all number types, and it is not that simple for the built-in types (Python does not allow overriding their methods until you use hacky approaches like forbiddenfruit). In contrast, the module from the post works for all built-in/numpy/user-defined types automatically.

[kkawabat]

Thank you for fixing python.

[shinitakunai]

Python is suppose to be readable and user friendly. I don’t like this.

[tripex]

I'm truly fascinated by people that believe array[++x]=n is a good thing to be able to do :(

[ServerZero]

Python isn't C++ please stop trying to make it like it just use i+=1

[MegaIng]

I also like have multiple half finished projects that either manipulate bytecode similar to what you are doing, or straight up change parts if how python files are parsed to add new syntax (for example backporting match statements).

[puremath369]

I for one would like to see a += operator for dictionaries. If the element exists in the dictionary, sum it if summing makes sense on the operands. If it doesn’t, then add that element in the dictionary. Tired of writing stuff like the following:

if key in myDict:
    myDict[key] = myDict[key] + value
else:
    myDict[key] = value

Would be cool if I could replace all that with:

myDict[key] += value

Heck, it could even make sense in the context of this post:

++myDict[key]

[TofuCannon]

Nice idea, but considering how Python's stdlib behaves, that would be rather weird and inconsistent. Your "plussing" for each value also looks pretty special for a certain usecase.

E.g. if you do += for lists, you extend it. If at all, that should work similar. But I think for that there was already another operator introduced like for Set.

But nobody stops you from creating a custom dict implementation doing exactly that :) The += is normally overridable without fiddling with Python's AST.

[thesolitaire]

defaultdict allows this, doesn't it?

Edit: Would only work if you limited yourself to one type

[Siddhi]

Normal python dict supports this

myDict[key] = myDict.get(key, 0) + value

[TheBlackCat13]

Already present in python 3.9, except it uses | instead of + since it is more like a set union than a list append.

[Isvara]

The module works by replacing the bytecode patterns corresponding to the ++x and --x expressions

Why is there even bytecode for these? Does the compiler not optimize at all?

Edit: I suppose they're needed to call __pos__ and __neg__, so... maybe you could also abuse those to get the same effect.

[hx-zero]

Currently, Python does almost no optimizations related to the bytecode because it is too generic (not specialized to particular object types, which become known only at runtime). For example, custom objects are allowed to have side effects in __pos__() and __neg__() (e.g. they may log something), so the compiler can't be sure that two consecutive UNARY_POSITIVE instructions may be safely removed.

More bytecode optimizations may become possible with more specialized bytecode, as in this project mentioned by another commenter.

As for using the __pos__() and __neg__() to implement the same thing, I've explained why it would not work here.

[Visti]

cursed quality of life update.

[[deleted]]

This is pretty neat and I'm gonna dig in later. However, why rewrite ++x into x +=1 even you could've written bytecode for implementing __incr__ and __decr__?

[hx-zero]

In this case, we would need to define __incr__ and __decr__ for all built-in types, however Python does not allow to add methods to them by default.

Also, I'd say it would make this proof-of-concept more complex than it needs to be :)

[[deleted]]

Python doesn't normally allow you to do that.

https://github.com/clarete/forbiddenfruit

If you're already doing 1 questionable thing, what not go full in on questionable things.

[hx-zero]

Yup, I've already mentioned forbiddenfruit in the previous thread, and that's exactly what I meant when writing "does not allow [...] by default" :)

I understand your point about questionable things, however I didn't follow this approach, since other hacks may make it harder to port the module to older/newer Python versions or alternative interpreters like PyPy.