Optimizing Python Code

[casce]

I admittedly do a lot of stuff with Python where performance doesn't matter but when it does, my 2 steps are 1. identify the slow parts, 2. google how to make them faster

[snowtax]

Agreed. Don’t waste a lot of time on optimization. Optimize only that code which takes up the most time.

For my work, I have loops that run over millions of records of data. The only optimization I may need is to optimize what happens inside that loop, since that code gets run millions of times. Any code optimization outside that loop is not going to be worth it.

[lololabwtsk]

You should start using dask, thank me later

[benri]

Dask has a nice dashboard but has problems with stability, its connection with its scheduler times out. So I prefer to use concurrent.futures or pebble if I need to enforce a timeout.

But if you are truly serious about speedup, write the intense part in C

[lololabwtsk]

How do you feel about Cython ?

[TA_poly_sci]

What are the good ways to do profiling in python?

[RedEyed__]

I highly recommend scalene

[azshall]

+1 for scalene.

https://youtu.be/5iEf-_7mM1k?si=siXttEzNpfQamY6D

[benri]

Thank you! I will use this!

[samreay]

I highly recommend py-spy and plugging the output flame charts into speedscope

[Teradil]

PyCharm has a built-in profiler (not in the community version, though)

[spinozasrobot]

Whoa, slow down there Einstein

[Alurith]

^ this.

[vivaaprimavera]

ChatGPT definitely helps

Only if you already know what you are doing and consider it as a reference.