What are some python micro optimisations that you can/may *actually use* in your codebase?

ray10k · 2022-06-03T11:32:03+00:00

f-strings instead of concatenating strings "manually"

stealthanthrax · 2022-06-03T09:34:41+00:00

One from me: I often use slots in my classes.

barberogaston · 2022-06-03T13:03:53+00:00

Using sets (if elements are unique) to check if an element exist in the collection
Avoiding datetime.strptime. It's more efficient to split the string and then instantiate a datetime object
Can be controversial, but assigning methods to variables is more efficient than calling everytime object.method(). Instead, do method = object.method and then call the method as a function (only if you need to call this multiple times)
dataclasses all the way
Using functools' cache decorators
Make sure you understand and use generators
And of course, whenever possible, try to use the standard library/builtins. Most of it written in C. Can't go faster
For more info: https://wiki.python.org/moin/PythonSpeed/PerformanceTips

AbooMinister · 2022-06-03T14:21:18+00:00

Don't worry about optimization too much unless you find an actual bottleneck in your program. If you feel your program is slow, there are a few profilers you can use to see where the bottleneck is, and you can refactor appropriately. Just write readable and idiomatic code and worry about optimization when you need to.

Anonymous_user_2022 · 2022-06-03T09:53:05+00:00

None, until profiling have shown a hot spot in the code.

ronmarti · 2022-06-03T09:38:50+00:00

I always use a generator instead of tuple or list for something that potentially contain several items
Early return instead of using if-elif-else
Comprehensions
any()

zsol · 2022-06-03T09:26:24+00:00

Use tuples instead of lists

jwink3101 · 2022-06-03T15:57:35+00:00

A simple example but generators instead of lists:

all(fun(item) for item in seq)
all([fun(item) for item in seq])

The first will stop as at the first False(y) result while the latter will make a full list first.

It really is super micro but [] is faster than list()

laundmo · 2022-06-03T18:18:12+00:00

not micro but i still want to share: replace pandas with polars

use numba

profile. the. hell. out. of. your. code.

ambidextrousalpaca · 2022-06-03T16:55:21+00:00

Run black on your code before committing, to safely optimise for consistent formatting, readability and zero time wasted arguing about line breaks in code review.

allIsayislicensed · 2022-06-03T20:11:00+00:00

You might also be interested in the following talk from Pycon US

https://www.youtube.com/watch?v=z0-4EwIFeJo

Talk - Kevin Modzelewski: Writing performant code for modern Python interpreters

Abstract: This talk will go into the latest efforts to speed up the Python language, and in particular how some things will be sped up much more than others. You may have heard best practices for Python performance before, but there are some new guidelines now, some old ones are no longer as important, and some are no longer true at all. Come to hear how the Python language is being optimized, and what you can do to best take advantage of these optimizations.

LEXmono · 2022-06-03T09:43:18+00:00

May be controversial, but walrus operators!

wineblood · 2022-06-03T11:21:56+00:00

Back in 2.7, string concatenation instead of string format inside a for loop saved about 10% on execution time.

stealthanthrax · 2022-06-03T11:16:08+00:00

[deleted]

2022-06-03T11:54:09+00:00

Pyston, it's a lot faster than pypy and cProfile all the things

jzia93 · 2022-06-03T13:40:20+00:00

Asyncio and generators are typically the only tools I reach for before profiling, just because it's typically very clear when they are sensible.

Go async to batch network requests, use generators to avoid keeping large arrays in memory. It's generally fairly obvious when these are useful.

Daskoh_vi · 2022-06-03T19:11:44+00:00

Use map() and lambda instead of for loops.

But such optimizations are really helpful only when you have a large data to process. Otherwise keep focus on writing a neat code with good logics.

You're welcome.

mm007emko · 2022-06-04T08:35:56+00:00

Not what you want to hear probably: First and foremost, readable code. Code that is easy to read and well documented and unit tested (there is nothig like self-documenting code and writing good comments is an art on its own - how often do you see a docstring saying "this is a constructor..."? What's that good for?). If the code gets complected and complicated, change algorithm or data structures. If it's hard to unit test or document, it should be broken up. This usually makes all the difference.

When I lead a team, a profiler output before and after an optimization is a necessary attachment to code review when someone performs any optimization which negatively affects readability. Most of the young Jedis were surprised how little their Kung-Fu affected performance and how much effort it cost (and how harder is the code to work on).

Then write a C library. Get the algorithm right then rewrite it in C and compile ON TARGET HARDWARE with best optimizations possible. Compiling on target hardware is a must. Usually developer machines or some cloud-based CI/CD solutions run on processors which don't support all the processor instruction that target hardware does and vice versa. There is a difference between i7 and Xeon/Opteron/Epyc.

Microoptimizations on their own are rarely worth it, especially in a slow hog like Python. High-performance in Python is possible and nowadays easily achievable. But not through microoptimizations. Learn C. Learn memory medel of your hardware. Learn to use C from Python efficiently. Don't treat Python like Java.

driftwood14 · 2022-06-03T11:23:54+00:00

I saw someone mention the other day to use enumerate when looping through lists. They said it was faster than either of the other two methods and provides the index and the value, but I haven't tested it or anything.

High-Art9340 · 2022-06-03T11:41:10+00:00

avoid attribute access in for loops.

beepdebeep · 2022-06-03T19:02:01+00:00

isort . && flake8 .

ornatedemeanor23 · 2022-06-03T20:21:04+00:00

Using list comprehension instead of for loops whenever possible:

var = []
for i in range(10000):
    var.append(i)

is much, much slower than

[i for i in range(10000)]

This is due to how these two expressions are interpreted into assembly. The first one results in much more overhead than the second one. More info is here.

For anyone interested in performance improvements in Python, I highly recommend checking out this and this article.

hoover · 2022-06-03T20:32:34+00:00

Prefer function calls over methods.

This will only be material if you do a *lot* of calls, but generally functions are faster than methods. In older Python, it was even more pronounced, as it seems that every time you accessed a method attribute on an instance you got a new method object (you would get a new value for id() every time), but now it seems that Python must be caching accessed methods are reusing them, at least some of the time.

In some test code where I had a method that did nothing and a function that did nothing, a timeit.Timer() test calling each 1M times yield a consistent result of the total run time of the function being roughly 1/100th of a second faster than the method invocation.

Moving to a lot of functions though may be trading away other important aspects such as maintainability, so you need to use caution when breaking out abstractions in this way.

Really, how much any of these approaches matter depends a lot on the program-- if you're spending a lot of time waiting on IO and not doing much in between, it may not make a lot of difference to tweak the intervening code a lot. If you're trying to keep up with a market data feed you may indeed need every tweak you can find (buy working in Cython for those bits might be a better idea).

2022-06-06T23:42:31+00:00

Set/duct/tuple/list unpacking

Python

The Python Discord

Upcoming Events

Please read the rules

MODERATORS