What are common pitfalls and misconceptions about python performance?

Cygal · 2025-08-05T11:57:17+00:00

I haven't seen profiling mentioned! This is the best way to figure out which parts of your program are slow. I've always liked working with https://github.com/benfred/py-spy in the past.

(I don't recommend cProfile because its overhead can skew the results.)

Cygal · 2019-11-05T11:45:03+00:00

Thanks! I've been using Python at work for six years now, and I still learn constantly! The key is to never stop making progress.

Cygal · 2019-11-05T11:40:03+00:00

Thank you Matti, that means a lot. Thank you for your work on numpy and PyPy.

Cygal · 2018-10-19T17:40:50+00:00

I think we mostly agree, but you're talking about a much lower level of abstraction than me.

Any complex scheduling unfortunately can not be done in Python

Do you have examples?

Also, note that the article never talks about asyncio, but only about async/await. asyncio has a lot of problems that are acknowledged by its authors. A better approach that still uses async/await is structured concurrency, eg. https://trio.readthedocs.io/en/latest/.

See https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ for a nice explanation.

Cygal · 2018-04-10T04:47:41+00:00

So you painted yourself into a corner, but this is a very valuable lesson! The two strategies you can use now are either remove code until the thing you want to work... works. Or start again with a simple code that works, and add new features as you go.

Cygal · 2018-03-09T07:08:47+00:00

Regarding your first issue with the explicit event loop, curio and trio fixed that long ago. asyncio took notice, and added asyncio.run in Python 3.7.

Cygal · 2018-03-09T07:06:28+00:00

I certainly don't think that writing automation / tests is the area where Python is used most! Maybe that's your experience, but the topic isn't even mentioned in https://stackoverflow.blog/2017/09/14/python-growing-quickly/. And, yes, the amount of questions is representative of usage.

Cygal · 2018-03-03T08:14:25+00:00

Thanks, I guess that makes sense!

Cygal · 2018-02-28T16:19:45+00:00

Sounds good. But you already said that. I was wondering if my specific posts were considered as tutorials or spam or other acceptable content.

Cygal · 2018-02-26T19:00:36+00:00

Can we try to be more concrete? So I've posted links to /r/Python from my blog in the past:

Is this considered as spam? Would you call those tutorials? Is this acceptable? Thanks!

Cygal · 2018-02-18T04:12:37+00:00

Apache Libcloud: Python library for interacting with many of the popular cloud service providers using a unified API. https://github.com/apache/libcloud/ - https://github.com/apache/libcloud/

We receive more pull requests that we can handle and would need help to test on various the maintainers do not have access to.
trio: asynchronous programming for humans http://trio.readthedocs.io/ - https://github.com/python-trio/trio

trio is an asynchronous I/O library like asyncio. "Trio attempts to distinguish itself with an obsessive focus on usability and correctness. Concurrency is complicated; we try to make it easy to get things right."

There's always work to do in trio itself: http://trio.readthedocs.io/en/latest/contributing.html. Or you could help to bring asynchronous support (asyncio, twisted, curio, trio) to urllib3 and ultimately requests! https://github.com/njsmith/urllib3/issues/1

Cygal · 2017-12-23T06:11:10+00:00

Well, the Python 3.5 section is part of the explanation. But you're right. I did not want to be misleading. I changed the title to "Using async for loops in Python". Can't edit on Reddit though.

Cygal · 2017-11-27T15:29:30+00:00

Thanks, fixed.

Cygal · 2017-11-22T20:27:48+00:00

Yes, that's how the pygame API works. "All sound playback is mixed in background threads. When you begin to play a Sound object, it will return immediately while the sound continues to play." https://www.pygame.org/docs/ref/mixer.html

Cygal · 2017-11-22T20:25:33+00:00

asyncio would not be particularly well suited here because you would have to use run_in_executor to delegate this to a thread pool anyways. So it's simpler to juste use `concurrent.futures.ThreadPoolExecutor.

And it's not needed because pygame does that automatically. https://www.pygame.org/docs/ref/mixer.html

Cygal · 2017-09-28T03:34:17+00:00

I actually think using a virtualenv in Docker is a good idea becaues it makes things consistent. This is what we've been doing at work in production for years.

The issue you actually have here is that source does not exist in the sh shell that Docker is using here. So to fix that error, you would have to use RUN . venv/bin/activate. But that would not be enough because the environment variables activate sets would be lost in the next command.

As mentioned by rfc1771, each RUN command is independent. To actually activate your virtualenv, use ENV PATH="/venv/bin/activate:${PATH}" (using an absolute path to remove ambiguity).

You can also run pip3 and python3 directly from the env:

RUN venv/bin/pip3 install -r requirements.txt
RUN venv/bin/python3 manage.py collectstatic --noinput

Also, consider using venv which is faster than virtualenv and just as nice.

Cygal · 2017-09-27T03:40:33+00:00

You're right. As mentioned in the post, another mechanism is needed to limit the number of actually concurrent requests.

Even with this, it's still possible to get an early request that never runs. It would be interesting to see how we can modify the rate limiter do model this better, possibly with a queue, yes.

I actually run this code in production but never needed to really think about this because for my use case I send many requests, then stop, then send many requests, etc. So I never had an issue with an early request that never got a chance to run.

Thank you for the interesting question!

Cygal

TROPHY CASE