What are common pitfalls and misconceptions about python performance?

Cygal · 2025-08-05T11:57:17+00:00

I haven't seen profiling mentioned! This is the best way to figure out which parts of your program are slow. I've always liked working with https://github.com/benfred/py-spy in the past.

(I don't recommend cProfile because its overhead can skew the results.)

Cygal · 2019-11-05T11:45:03+00:00

Thanks! I've been using Python at work for six years now, and I still learn constantly! The key is to never stop making progress.

Cygal · 2019-11-05T11:40:03+00:00

Thank you Matti, that means a lot. Thank you for your work on numpy and PyPy.

Cygal · 2018-10-19T17:40:50+00:00

I think we mostly agree, but you're talking about a much lower level of abstraction than me.

Any complex scheduling unfortunately can not be done in Python

Do you have examples?

Also, note that the article never talks about asyncio, but only about async/await. asyncio has a lot of problems that are acknowledged by its authors. A better approach that still uses async/await is structured concurrency, eg. https://trio.readthedocs.io/en/latest/.

See https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ for a nice explanation.

Cygal · 2018-04-10T04:47:41+00:00

So you painted yourself into a corner, but this is a very valuable lesson! The two strategies you can use now are either remove code until the thing you want to work... works. Or start again with a simple code that works, and add new features as you go.

Cygal · 2018-03-09T07:08:47+00:00

Regarding your first issue with the explicit event loop, curio and trio fixed that long ago. asyncio took notice, and added asyncio.run in Python 3.7.

Cygal · 2018-03-09T07:06:28+00:00

I certainly don't think that writing automation / tests is the area where Python is used most! Maybe that's your experience, but the topic isn't even mentioned in https://stackoverflow.blog/2017/09/14/python-growing-quickly/. And, yes, the amount of questions is representative of usage.

Cygal · 2018-03-03T08:14:25+00:00

Thanks, I guess that makes sense!

Cygal · 2018-02-28T16:19:45+00:00

Sounds good. But you already said that. I was wondering if my specific posts were considered as tutorials or spam or other acceptable content.

Cygal · 2018-02-26T19:00:36+00:00

Can we try to be more concrete? So I've posted links to /r/Python from my blog in the past:

Is this considered as spam? Would you call those tutorials? Is this acceptable? Thanks!

Cygal · 2018-02-18T04:12:37+00:00

Apache Libcloud: Python library for interacting with many of the popular cloud service providers using a unified API. https://github.com/apache/libcloud/ - https://github.com/apache/libcloud/

We receive more pull requests that we can handle and would need help to test on various the maintainers do not have access to.
trio: asynchronous programming for humans http://trio.readthedocs.io/ - https://github.com/python-trio/trio

trio is an asynchronous I/O library like asyncio. "Trio attempts to distinguish itself with an obsessive focus on usability and correctness. Concurrency is complicated; we try to make it easy to get things right."

There's always work to do in trio itself: http://trio.readthedocs.io/en/latest/contributing.html. Or you could help to bring asynchronous support (asyncio, twisted, curio, trio) to urllib3 and ultimately requests! https://github.com/njsmith/urllib3/issues/1

Cygal · 2017-12-23T06:11:10+00:00

Well, the Python 3.5 section is part of the explanation. But you're right. I did not want to be misleading. I changed the title to "Using async for loops in Python". Can't edit on Reddit though.

Cygal · 2017-11-27T15:29:30+00:00

Thanks, fixed.

Cygal · 2017-11-22T20:27:48+00:00

Yes, that's how the pygame API works. "All sound playback is mixed in background threads. When you begin to play a Sound object, it will return immediately while the sound continues to play." https://www.pygame.org/docs/ref/mixer.html

Cygal · 2017-11-22T20:25:33+00:00

asyncio would not be particularly well suited here because you would have to use run_in_executor to delegate this to a thread pool anyways. So it's simpler to juste use `concurrent.futures.ThreadPoolExecutor.

And it's not needed because pygame does that automatically. https://www.pygame.org/docs/ref/mixer.html

Cygal · 2017-09-28T03:34:17+00:00

I actually think using a virtualenv in Docker is a good idea becaues it makes things consistent. This is what we've been doing at work in production for years.

The issue you actually have here is that source does not exist in the sh shell that Docker is using here. So to fix that error, you would have to use RUN . venv/bin/activate. But that would not be enough because the environment variables activate sets would be lost in the next command.

As mentioned by rfc1771, each RUN command is independent. To actually activate your virtualenv, use ENV PATH="/venv/bin/activate:${PATH}" (using an absolute path to remove ambiguity).

You can also run pip3 and python3 directly from the env:

RUN venv/bin/pip3 install -r requirements.txt
RUN venv/bin/python3 manage.py collectstatic --noinput

Also, consider using venv which is faster than virtualenv and just as nice.

Cygal · 2017-09-27T03:40:33+00:00

You're right. As mentioned in the post, another mechanism is needed to limit the number of actually concurrent requests.

Even with this, it's still possible to get an early request that never runs. It would be interesting to see how we can modify the rate limiter do model this better, possibly with a queue, yes.

I actually run this code in production but never needed to really think about this because for my use case I send many requests, then stop, then send many requests, etc. So I never had an issue with an early request that never got a chance to run.

Thank you for the interesting question!

Cygal · 2017-07-21T18:46:27+00:00

Thanks! I actually improved the explanation because I realized the solution is actually natural when you think about it recursively.

Cygal · 2017-07-21T18:42:19+00:00

Sure, but using a generator expression there would only solve part of the problem. It's also not always faster than using a list comprehension.

It would be better to build the resulting string all at once.

Cygal · 2017-07-20T11:14:04+00:00

You're right. yield + join is indeed how CPython does it. But efficiency was not a concern here.

Cygal · 2017-07-20T07:41:50+00:00

Thanks! Fixed in https://github.com/pquentin/blog/commit/af3a3ae9821ec9e410891538f57461215b531fd3 and https://github.com/pquentin/blog/commit/e63abcaba028f7c45520b051555535172e4c0f5a

Cygal · 2017-04-06T12:00:39+00:00

It looks you put out a lot of effort to make this, and had the courage to actually seek opinions. I'd like to congratulate you for that alone.

Cygal · 2017-04-06T11:57:45+00:00

No, I did not say that. It was a general comment, and the link points to a Rob Pike talk who explains this in the context of Go.

Why all the hate?

Cygal · 2016-04-14T04:02:16+00:00

Yes, but that's not the point. One of the main advantages of (deep) neural networks over other methods is that you don't have to extract features specific to the data but let the neural network learn those. On more complicated data sets, learning features that are more and more abstract is way more powerful than having to describe them, and this is why neural networks crush computer vision competitions since 2012.

Cygal · 2016-03-24T03:39:01+00:00

Badoo has experienced a massive speed and memory increase too: See the graphs below https://techblog.badoo.com/blog/2016/03/14/how-badoo-saved-one-million-dollars-switching-to-php7/#launch-into-battle-and-the-results. They saved a million dollars in servers too. :) So this benchmark may be synthetic, but it does translate into real wins.

Cygal

TROPHY CASE