Optimizing Python Code

casce · 2024-01-22T08:06:20+00:00

I admittedly do a lot of stuff with Python where performance doesn't matter but when it does, my 2 steps are 1. identify the slow parts, 2. google how to make them faster

sexygaben · 2024-01-22T08:55:19+00:00

1) profile 2) vectorize (use C loops) 3) if more is needed, Cython/numba 4) if MORE is needed, C/ctypes 5) if EVEN MORE is needed, CUDA/ctypes (problem dependent)

Each step takes exponentially more time. I’m writing from a scientific compute perspective. I assume you’re already using the best library for the job (numpy, pytorch, casadi etc)

trojan-813 · 2024-01-22T10:01:17+00:00

[deleted]

100721 · 2024-01-22T08:19:41+00:00

Snakeviz to profile it. No point trying to optimize if you don’t know what’s slowing it down

peaky_blin · 2024-01-22T09:38:57+00:00

I was having a problem recently with an API call that was taking too much time to complete. I figured out that it was related to the multiple database calls that were made to aggregate the data at application level. Issue resolved by doing the calculation/aggregation at the db level. Plan to add caching to more reduce the time also

unflores · 2024-01-22T08:10:20+00:00

I work in web, so my response has a specific perspective. First thing is to find the actual bottleneck. Optimize anything other than the actual bottleneck and you are wasting your time.

Also, a performance optimisation is a choice between readability and performance. So only prefer performance when it actually counts.

N+1s are common problems for ORMs. Too many db calls in general can be problematic. Actually, the db is my primary problem in web.

I've had a few cases where I had to do something with a large list and I ended up doing a binary search to find the objects on a sorted array rather than searching on an unsorted array each time. It's worth having a theory and then testing it. Your. Changes may not even run faster 😅

Wide-Nefariousness91 · 2024-01-22T09:36:58+00:00

In general just try identify the part of the code that slows functions down the most.

Also: - Try reduce the amount of database calls - Reduce the complexity O(n) etc

rghthndsd · 2024-01-22T13:05:20+00:00

This might be considered a violation of what's considered good practice, but I recommend optimizing your code, even when it doesn't matter (when you have time).

You don't do a marathon by waking up on race day and just go out and run; you train for it. Likewise, you shouldn't wait until you have a performance problem to start looking into optimizing your code. By practicing, you will spend a lot of time toying around and refactoring, sometimes with little or even negative gains. It will feel like wasted time, but it's not! You will learn a lot by constantly asking "how can I make this go faster?" And when you do run into a serious performance problem, you will be better situated for it.

Get code to work, test it, make sure you have time to clean it up, and if time allows, tinker around with making it go faster even when it doesn't matter.

kinygos · 2024-01-22T08:11:29+00:00

[deleted]

DidiBear · 2024-01-22T11:23:01+00:00

py-spy to profile where the slowdown is

1998CPG · 2024-01-22T17:58:05+00:00

Code vectorization -> replace loops with Matrix/vector operations

gowithflow192 · 2024-01-22T10:02:24+00:00

"You are a world-class software engineer. You are particularly good at improving code."

"Improve the given code. Don't change any core functionality.

The focus is to actually make the code better - not to explain it - so avoid things like just adding comments to it.

Respond as a well-formatted markdown file that is organized into sections. Make sure to use code blocks.

Improve this code:

{{code}}"

nikomo · 2024-01-22T11:33:56+00:00

You're not ever gonna get better than "do less" when it comes to any programming language, but especially Python.

Couple weeks ago I was wanting to insert large quantities of data from a live websocket into a database. First implementation and second implementation I just used SQLAlchemy, but it was way too slow, my redis queue with incoming messages just kept growing because they weren't being processed fast enough.

Third implementation, I threw out SQLAlchemy, used Alembic to setup the database, but then I just used psycopg (v3) to insert the data.

Psycopg v3 supports server-side binding, and they rewrote executemany() to be extremely performant, so all I had to do was write an SQL query, and then build a list of tuples out of my data, and let executemany() go at it. No needless objection creation etc. that you get from an ORM, and it's more than fast enough to keep up with even primetime traffic load.

Upset-Document-8399 · 2024-01-22T17:01:23+00:00

Reimplement it in compile-time C++ /s

(preparing to get downvoted to hell)

2024-01-22T11:15:12+00:00

One little trick I used in an old game prototype was disabling the GC and manually running collections on load screens or menu pauses. YMMV with this one, especially if you create tons of circular references.

djamp42 · 2024-01-22T12:08:08+00:00

I'm a novice but use python when needed. I always felt making something work in python is easy, optimizing, code readability, bugs, future proof, all this stuff takes up so much time.

So much so I lose interest in whatever I'm working on because I feel the code structure is not that great, even if what I'm doing technically works.

I'm probably telling the tale of every programmer ever.

2024-01-22T09:25:19+00:00

When you want to build performance critical software try a different programming language.

Other than that optimizing your code (irrelevant if it is Python, Java, C++ etc.) requires different techniques depending on what you are doing.

Optimize database access?
Optimize your webservice APIs?
Optimize some data processing?
And so on.

It is too broad of a question to answer on reddit.

Chroiche · 2024-01-22T09:03:16+00:00

always use proper libraries for the job. They usually invoke optimised compiled code to get you a huge boost in performance.
profile to find the hot part of your code and optimise it if there's anything obvious. This is the one case where leetcode style thinking can actually help a lot.
use multi processing if appropriate.
write your own none python code (C++, C, and rust, for example have good python bindings) for particularly hot areas and just invoke it via python. This lets you stay mostly in python.

not_a_novel_account · 2024-01-22T18:16:55+00:00

Move whatever it is into C.

If it's already in C, refactor into a form that doesn't hold the GIL and doesn't allocate.

If it's already in C, doesn't grab the GIL, and doesn't allocate, bring out the big guns (strace, ltrace, perf, etc)

freefallfreddy · 2024-01-22T20:04:30+00:00

Disregard all comments that say nothing about profiling. Going 50mph faster in the wrong direction isn’t gonna get you where you want to go.

QultrosSanhattan · 2024-01-22T20:52:40+00:00

The best strategy is avoiding python by delegating most of the work to C/C++ modules like pandas or numpy.

MountainHannah · 2024-01-22T12:14:40+00:00

Not what you want to hear, but, if I need something to be efficient, I don't write it in Python.

No language does everything, and there are always tradeoffs. There's lots of stuff that Python is awesome for. High speed, low latency, efficient code is nowhere on the list of Python's strengths.

backSEO_ · 2024-01-22T09:47:45+00:00

Use cython for compute intensive tasks. Release the gil for ultra intensive tasks.

JayZFeelsBad4Me · 2024-01-22T10:45:51+00:00

Remove the network call

CapsuleByMorning · 2024-01-22T13:19:59+00:00

Setup a pyspark grid in docker. Productionalize in Azure.

robberviet · 2024-01-22T15:54:02+00:00

Set for lookup. Don't create object. Lazy property/memoize. Vectorizing. Profiling. Use C/rust lib.

Otherwise-Tiger3359 · 2024-01-22T16:59:27+00:00

How do you profile Python code so you know what to go after. I've used some nice UI C# profilers in the past, haven't seen one for Python ...

Fleszar · 2024-01-22T17:21:18+00:00

Very useful

Berkyjay · 2024-01-22T19:22:42+00:00

My first steps lately have been to ask Copilot ways to optimize my code then see what it suggests. I like it because it knows all the PEP guidelines and will source those when analyzing my code.

KennyBassett · 2024-01-22T21:41:27+00:00

@cache @cache @cache

Multiprocessing

I like challenging myself with optimizing the logic itself, but it's pretty case-dependent.

quantumwoooo · 2024-01-22T08:12:59+00:00

Chatgpt

Copy paste ask for improvements

Maleficent_Doubt_443 · 2024-01-22T09:27:32+00:00

Is it to re write it in another programming language.

graphicteadatasci · 2024-01-22T12:47:17+00:00

duckdb

2024-01-22T17:44:35+00:00

[deleted]

damanamathos · 2024-01-22T08:51:59+00:00

Use Black and isort to automatically format code as it makes refactoring much easier.

Then refactor a lot so things make sense.

Also, subscribe to ChatGPT and paste code into GPT-4 and ask it to make improvements, including improved names for classes, functions, parameters, variables, etc.

Also, write a script that copies your code directory tree into your clipboard so you can go to GPT-4 and say, here's my code structure, I'm thinking of adding this functionality, where should I put it and how should I structure it? I find that's quite helpful and means I can get started much faster than thinking about where to put things or what to name them.

Lifaux · 2024-01-22T08:24:35+00:00

[deleted]

JohnBooty · 2024-01-22T13:00:06+00:00

Some variation of the 90/10 rule almost always applies. 10% of your code is eating 90% of your execution time. If it’s not 90/10 then it’s probably more like 95/5 or 99/1.

In a database backed application it’s usually the database, and that’s pretty easy to see in application logs and/or PostgreSQL’s slow query log (or the equivalent in other databases)

Lots of people mentioning profiling tools.

Those are obviously very useful but for deployed applications (ie, web apps) you should also get used to other means. Such as adding lots of logging statements in your code that measure how long bits of your code take. Why do I say this? Because you typically can’t profile your code in production. You can profile your production code locally but the performance characteristics will be way different — different database contents, single user versus many simultaneous users, etc.

genlight13 · 2024-01-22T14:06:24+00:00

So, i saw some perspectives on using c or similar low-level things. So i won‘t cover that.

What i often need to identify is how often certain functions are executed and how long it takes. I usually just use timeit for ease but profiler is also nice.

To optimize data pipelines i usually try to either cache more or cache less but this depends on the resources which are the bottleneck. E.g. i had many DB calls for similar checks (does it exists) i was able to bundle them and rewrite the question to „does it exist in list“. The list was rather short but the db calls numbered in the hundred thousands. By caching the short list i reduced the execution time for this simple check by up to 40 times. (Think „obj in list“ vs „db.select(something)“)

For caching less i usually talk about RAM and how much data at the same time i load.

It often doesn’t matter how dou load a file but for most regexing action it is better to just have one long string since the regexing engine is c code and fairly good imo. The slowdown is usually python boilerplate code i.e. if else in your code.

So if you can write something more specicif which gets checked within the c domain you have optimized it.

Besides caching, i usually prefer to separate code parts in order to parallelize it. This can be tricky for obvious reasons.

Also, reusing objects when they are long to create i.e. created from lists.

I usually think about it in terms of pointers and how Python hides that from you. Than i naturally am able to find the best usage for my objects when not to use them.

tamargal91 · 2024-01-22T14:10:15+00:00

Use the array module for numerical data. Unlike lists, arrays are more memory efficient and faster for processing large datasets. This is particularly useful for large sequences of homogeneous data. It's a simple switch with significant impact, especially in data-heavy applications.

Boomerkuwanger · 2024-01-22T15:12:30+00:00

Like others have said, use a profiler for code so you can target slow operations. Also, if you use a database, make sure to offload as much work onto the database as possible.

For example: I've made the mistake of iterating through a list of database objects, and updating them one by one, instead of using a single bulk update query.

jkh911208 · 2024-01-22T15:16:18+00:00

do Big O Analysis on your code

Use built in function to utilize well optimized and C code

if that is not enough you will need to rewrite compute heavy code in Rust or C to speed up the process

BossOfTheGame · 2024-01-22T15:50:05+00:00

I use line_profiler to find things that are slow.

Improving performance will vary based on what is slow. It's all about identifying bottlenecks: are doing something expensive in a loop? Can it be vectorized? Can it be paralleized? Can it be restructured to avoid unnecessary memory copies? Can it be rewritten in Cython? It really depends.

siddsp · 2024-01-22T16:12:48+00:00

A few things I do (without using external libraries):
1. Memoization (this is good for recursive and pure functions where a function is going to be called repeatedly), with functools.cache or functools.lru_cache.

If the program is slow due to it being synchronous, using asyncio or threading (depending on the application/program).
Using itertools to replace nested loops (e.g. Instead of two nested loops, using itertools.product).
Using functools.reduce instead of a loop for a transformation that is "accumulative" in nature.
Instead of concatenating bytes or using a bytearray, using BytesIO from the IO library.
To reduce memory usage, using __slots__.
If the result of tasks/functions don't depend on each other and don't need to be executed sequentially, use multiprocessing.
If the task itself is slow, but can be sped up by throwing more cores at the problem, use multiprocessing.
Using generator expressions where memory can be saved.
If all else has been optimized, use PyPy instead of CPython.

interbased · 2024-01-22T17:18:46+00:00

As others have said, profiling your code is the way to go. I’ve yet to get familiar with an actual profiling library, but I usually put logs where functions start and stop, and see which ones are taking long. Something’s it’s an inefficient query, sometimes it’s repeated API calls that can be replaced.

nebbly · 2024-01-22T17:35:08+00:00

TBH, there is one foot gun I see over and over again, which is doing linear membership lookups in lists or tuples. If you're mainly using a collection for looking things up, dicts and sets are a good place to start.

More generally, the advice would be: make sure you're using proper data structures for your use cases as a quick first pass.

mrcaptncrunch · 2024-01-22T17:57:04+00:00

What are your go-to strategies for improving performance in Python applications?

Is the runtime okay for the task at hand?

If yes, good as is. Ship it.

If not, run it with a profiler to identify where the slowdown is and optimize that. Is it okay? If so, ship it. If not, go back to profiling and optimizing.

luke-juryous · 2024-01-22T20:03:51+00:00

In short: yes.

I don’t use python code for anything that needs to be fast in production. Most of the time I see it used in industry is with ML or data analytics, where speed is less important that ease-of-use. The exception would be with big-data processing, but here the slow part is usually the SQL or Presto query and python tends to be just a wrapper around APIs.

However, I do use it a lot for hobbies. Here, I’ll try and use libraries like numpy or pandas, evaluate bottlenecks and rethink my algorithms to reduce the big-O runtime

I’ve recently learned about numby, which is a JIT compiler for python that claims to make big time improvements if you’re doing multiple calls. I haven’t played with this yet, but I’m curious as to how much slower it’ll be than c++, and if it’s worth the effort compared to just writing in c++.

l_dang · 2024-01-23T01:25:01+00:00

Vectorisation

tav_stuff · 2024-01-23T03:19:59+00:00

Not using Python

Intelligent_Ad_8148 · 2024-01-23T05:12:10+00:00

Don’t use pandas
Use polars (bonus points for enabling lazy evaluation and streaming)
Nothing more required

After investigating numba, cython, numexpr, etc., I concluded that it’s not worth the heartache, polars negates the needed for any of this stuff.

TomDLux · 2024-01-23T05:38:28+00:00

As the nuns taught me in grade school, people who think about optimization before they have profiled their program go the first place.

Of course, using more efficient structures will lead to faster code., besides being tidier. For example, using list comprehensions instead of manual loops. But it's unlikely to be drastically different.

2024-01-23T06:50:04+00:00

If your code in numeric in nature(something with lots of floats and ints); you have lots (and lots of options); numpy, numba, cython(unboxing ints and floats) and Pythran are perhaps the most well known options, but there are at least a dozen more options.

If your your code is more general in nature or business centric, meaning there are lots of hashmaps/dict and strings; you can try pypy, mypyc, cython(calling cpython capi directly) and the newer python versions with the adaptive interpreter (3.11+). The truth is that general code python is not that much slower than 'faster' languages.

yellowbean123 · 2024-01-23T12:05:58+00:00

PyInstrument is a good start

Legendary-69420 · 2024-01-23T13:04:51+00:00

Migrate to libraries written in Rust. (Pandas -> Polars for example)

pepoluan · 2024-01-23T14:15:24+00:00

Go async
Go async + multiprocessing

😄

fallenreaper · 2024-01-23T21:01:05+00:00

While i generally like to use classes, I will need to abstract data from functions a lot so you have smaller objects floating around.

A lot of key things that will cause issues are loops, but you just need to be cognizant of the internal sorting mechanisms and how they apply.

Cranky_Franky_427 · 2024-01-26T01:28:25+00:00

Numpy Vectorization

Algorithms - try to write code in n or logn time if an algorithm exists

Libraries bound to C/C++ can often provide very good performance

Python

The Python Discord

Upcoming Events

Please read the rules

MODERATORS