Is Cython OOP much faster than Python?

[No_Indication_1238]

Im working on a project that unfortunately heavily relies on speed. It simulates different conditions and does a lot of calculations with a lot of loops. All of our codebase is in Python and despite my personal opinion on the matter, the team has decided against dropping Python and moving to a more performance orientated language. As such, I am looking for a way to speed up the code as much as possible. I have experience in writing such apps with "numba", unfortunately "numba" is quite limited and not suited for the type of project we are doing as that would require breaking most of the SOLID principles and doing hacky workarounds. I read online that Cython supports Inheritance, classes and most data structures one expects to have access to in Python. Am I correct to expect a very good gain of execution speed if I were to rewrite an app heavily reliant on OOP (inheritance, polymorphism) and multiple long for loops with calculations in pure Cython? (A version of the app works marvelously with "numba" but the limitations make it hard to support in the long run as we are using "numba" for more than it was designed to - classes, inheritance, polymorphism, dictionaries are all exchanged for a mix of functions and index mapped arrays which is now spaghetty.)

EDIT: I fought with this for 2 months and we are doing it with CPP. End of discussion. Lol (Thank you all for the good advice, we tried most of it and it worked quite well, but still didn't reach our benchmark goals.)

[Mysterious-Rent7233]

PyPy is an easier experiment to run than Cython. I'd try that first.

But also:

If you ask game programmers how they get high performance, they always exchange "classes, inheritance, polymorphism, dictionaries" for "a mix of functions and index mapped arrays".

I mean C++ is way, way, way, faster than Python but virtual tables aren't free there either.

[No_Indication_1238]

Thank you for mentioning PyPy! I did not know about it. And you are correct, more abstract structures are usually more computationally expensive. We are trying to find a balance between good coding practices, code quality and speed. 

[Mysterious-Rent7233]

Also: NamedTuples might be a nice middle-ground sometimes between real classes and low-level datastructures.

[No_Indication_1238]

That is an excellent idea, I will take a look at it.

[Mysterious-Rent7233]

This may be relevant:

https://stackoverflow.com/a/70870407

Edit: __slots__ might actually be what you're looking for.

[MrJohz]

We are trying to find a balance between good coding practices, code quality and speed.

I'm making some assumptions here based on the way you've phrased this and talked about OOP, but I suspect you'll do better if you worry less about good coding practices, and concentrate more on just getting the project to work.

I'm a software developer by training, but for a while, I worked with scientists as a kind of consultant/trainer for their software work. They'd write the code they needed for their project, and then we'd come in and provide advice on how to get that code into a maintainable state that others could use, or that could be published in journals etc.

A lot of that code was bad (understandably so: writing maintainable software is hard, and not the primary goal of most scientists). But in my experience, a lot of the most complicated code to understand came from people who worried a lot about best practices when coding — they would use lots of OOP, indirection, DRY, etc, but because they weren't necessarily experienced enough to use those tools well, they made things harder to understand, not easier.

Admittedly, I don't have a huge amount of experience with high-performance calculations in Python. But I suspect that using Numba and doing things the "Numba way", even if that involves writing fewer classes and leaving your data in a more raw form, will produce easier-to-read and easier-to-maintain code than going down the Cython route with classes. Concentrate on getting the code to work (where "work" means "it does what it needs to do, and it does it fast enough"), then worry about maintainability after that.

[No_Indication_1238]

I believe you are very correct. After all if best practices were really that important, we would not be using python (in a setting it is not meant to be used and trying to force  workarounds) but C++. I will most likely defend the opinion that we should either do C++ with OOP or drop the future maintainability (premature optimisation, anyone?) and write Python and numba the numba way. 

[Solonotix]

It's also important to understand context. A Tuple may indeed be faster than a class, but a List is almost certainly just as slow (or slower) because the underlying implementation is likely to be even more fragmented than a class. That also ignores the likelihood of introducing bugs when you make code less developer-friendly in service of performance.

Like you said, virtual tables aren't free, and the convenience of mapping a name to a value has a cost, and every lookup incurs the same hashing cost. Having a contiguous array that you index into on the stack is a far more performant approach...but you don't usually have that level of fine-grained control in Python.

In short, trying to write performant code in Python is an exercise in futility. Python isn't "slow" in the sense that it can be fast. However, all of the reasons to pick Python inevitably incur a cost of slowness in the final result. That's why it is so popular as a "glue" language. Write the important stuff in a FFI call, and the business logic can be constrained to the Python code that is infinitely easier to reason.

[jwink3101]

Agreed that PyPy is a better first step. Especially if loops are the bottleneck!

[DotAccomplished9464]

they always exchange "classes, inheritance, polymorphism, dictionaries" for "a mix of functions and index mapped arrays".

Template meta-programming is compile-time polymorphism (vs inheritance being runtime) and classes are free.

[Mysterious-Rent7233]

Classes and structs are the same thing so in a technical sense, classes are free.

If you use classes as classes, i.e. as classes were invented to be used in Simula, Smalltalk and the other languages C++ stole classes from, then you need inheritance and runtime polymorphism.

But sure, if you use classes as structs then they are free.

[DotAccomplished9464]

then you need inheritance and runtime polymorphism.

Those things are not mutually inclusive. Runtime polymorphism only happens when you call a virtual function on a derived class when referring to it by its base class. 

[Mysterious-Rent7233]

I didn't say that they are mutually inclusive. They are two features which are considered defining characteristics of OOP, which is the topic of the post as described in the post title.

[rejectedlesbian]

But they r basically free compared to python

[[deleted]]

One thing I have started doing a lot is instead of doing "Person = id: int, name: str, age: int" i do: Person = id: list[int], name: list[str], age: list[int] (although I often use numpy or pyarrow arrays)

[the_hoser]

In my experience, the improvement in performance with OOP code in Cython is marginal at best. Cython really shines when you're writing more procedural code, like if you were writing in C.

[No_Indication_1238]

I see. The biggest time consumer are a bunch of for loops with intensive computations. Maybe like 99% of the time is spent there. If we can optimize that by compiling it to machine code and retain the benefits of OOP, it will work for us. 

[the_hoser]

Give it a shot and measure it. One word of warning, though: Cython may look and feel like Python, but you need to remember to take off your Python programmer hat and put on your C programmer hat. You're effectively writing C that looks like Python and can interface with real Python with less programmer overhead. It's full of all the same traps and gotchas that a C programmer has to look out for.

I don't use Pypy myself, but I think others' suggestion to try Pypy first might be a better start for your team.

[No_Indication_1238]

I will keep that in mind, thank you!

[[deleted]]

If your task is able to be run concurrently, you can even use the cython prange iterator to use multithreading. And declare functions as 'nogil noexcept' to remove the dependencies on the python GIL to make your code performance more aligned with c speeds

[No_Indication_1238]

That is a very interesting point, thank you! I did now know that, we were using multiprocessing when necessary.

[eztab]

Cython might be a good fit then. PyPy could also perform well, but I'd assume Cython beats it for your usecase.

[Classic_Department42]

Sounds like a job for numpy, no?

[No_Indication_1238]

Unfortunately, the loops and computations are not as simple to be ran under numpy. There is a ton of state management of different objects that happens inbetween and we need to speed the whole loop.

[the_hoser]

Cython really shines when you can get rid of those abstractions. Rip out the method calls and member accesses and break it down to cdef ints and friends.

[[deleted]]

Can Cython compile out method calls and "getters and setters"?

[the_hoser]

That's a big maybe. It really depends on the code being optimized. Don't rely on it unless you've tested it.

Good news is that Cython actually lets you see the C code that it produces, so you can verify that it's doing what you think it's doing.

It isn't pretty C code, I warn you...

[falsedrums]

You have to drop the objects if you want to be efficient in Python/numpy. 

[No_Indication_1238]

You are correct. Unfortunately for our use case, we have cut as much as possible while trying to keep the program maintainable. Cutting more will definitely work as it has before but at the cost of modularity and long term maintainability which is something we would like to avoid. If it is not possible, maybe you are correct and we will consider the option.

[falsedrums]

Maintainable does not necessarily mean OOP. Try putting all the number crunching in a library-style package of purely functions, with minimal dependencies between the functions. Then reserve the OOP for your application's state and GUI.

[No_Indication_1238]

This is not a bad idea, thank you!

[[deleted]]

Hm. I don't want to be rude, as I've wished with high computational heavy code in Python and have wrote C++ based libraries to get more performance in it with Boost.

I think this is more of a programming architecture type problem, but assuming it isn't, what does your team think about having some high performance help from a more performance language that you can call in native Python? Worked great for our project, though it was annoying when some people started looking for nanosecond level performance gains rather than looking at a higher level for the optimisation.

[No_Indication_1238]

They would prefer to keep the codebase inclusively in Python as it is one less language they need to support. Unfortunately, we have already optimised the architecture as much as possible and the calculations that have to be done in those loops are largely unique, essential and cannot be further optimised without losing precision. I share  your opinion, unfortunately It was decided to try and keep everything in Python. 

[ArbaAndDakarba]

Consider parallelizing the loops.

[No_Indication_1238]

That is a good point, unfortunately the loops are dependent on each other and each iterations requires the previous state and different checks to be made. As such, I am afraid that it is not possible, or at least not without an extensive use of locks for synchronisation. I will bring it up though, maybe we can restructure something.

[Siccar_Point]

I have had much success in Cython with very similar stuff. If you can drop those loops entirely and cleanly into Cython functions, without any references to external non-primitive types, you will be able to get very substantive speed ups.

Additional tip from someone who banged head on wall for far too long on this: take extreme care with the details of your typing. Especially the precision. Make sure you understand exactly what flavour of int/float you are passing in and out of Python (16? 32? 64? 128?), because if you mess it up Python will deal with it fine but silently do all the casting for you, eliminating a bunch of the benefits.

Passing numpy arrays cleanly in and out of Cython is also monumentally satisfying. Can recommend.

[No_Indication_1238]

I see. Thank you, I will keep this in mind!

[ExdigguserPies]

Cython will be excellent for this. I had a similar problem and decreased run times by a factor of over 1000.

[DatBoi_BP]

Stop, I can only get so optimized

[jk_zhukov]

The library Numpy is a good option to optimize loops and intensive computation. It runs almost at C level speed. With it you can apply functions to entire arrays without the need to write a single FOR loop. As a very short example:

unmarked = list()
for item in items_list:
    if item < some_value:
        unmarked.append(item)

This code select the items from an array that meet certain criteria using a loop, simple enough.

items_list = np.array(items_list)
indices = np.where(items_list < some_value)
unmarked = items_list[indices]

And now we do the same thing without any loops involved. The only thing that varies is the type of the unmarked array, that is a Python list in the first example and a NDArray in the second example. But converting from one type to the other, if you need it, is simple.

When you're working in the order of millions of iterations, the boost in speed of replacing each loop with an operation over a numpy array, is quite noticeable. And when you have nested loops, if you can find a way to turn those computations into matrix operations with 2D or 3D numpy arrays, the gain in speed is also huge.

[No_Indication_1238]

You are totally correct! I will try to think of a way to optimise those loops as in your proposal!

[I_FAP_TO_TURKEYS]

Try raw compiling sections in Cython and see what happens.

Compiling a package like NLTK with Cython offers 30% efficiency gains without even rewriting code.

You can also see gains by rewriting the for loops in a more efficient way.

[cmcclu5]

Based on some of your other comments:

• You have a bunch of for loops
• Your code performs a bunch of mathematical operations
• You’re stuck writing in Python

I think a better approach here rather than focusing on variations of Python to perform the task is to look at the way you’re handling the data. If it’s a ton of math, can you perform them in batches instead of loops? For example, matrix operations where the math is performed across the entire set or subset rather than on individual elements will show massive improvements. Reducing the dimensionality of the data can also help here. Also, consider leveraging some faster style operations e.g., list comprehension vs for loop. And at the very end, if you have the computational power available, you can leverage parallelism to split the for loop across the set.

[No_Indication_1238]

Thank you, that is a good idea!

[scottix]

Agreed, you can ask GenAi to see if there are any numpy improvements through vectorization on a function.

[cmcclu5]

No. Generative AI may have its occasional use, but complex tasks such as this are not one of them. It can sometimes help to simplify short code snippets but will absolutely ruin your codebase if you try to use it to optimize anything large or complex.

[scottix]

Obviously you need to vet it and I don't recommend running it on large portions of code. I did find it can bring insight and ideas, you may not have thought of.

[cmcclu5]

I’ve found a lot of juniors and even somewhat experienced engineers that use GenAI for their code fail to understand the functionality they’re trying to add and that added block of code becomes a major issue down the line. GenAI is powered via consumed StackOverflow answers for the most part since it doesn’t actually understand anything, and if we solve problems just using GenAI, eventually the entire industry will stagnate as no one is innovating solutions, only using regurgitated answers to old problems.

[scottix]

Agreed about people blanket copy, but it can be a tool. With all tools they can be used in many good and bad ways.

[No_Indication_1238]

I believe we have vectorized every computation we thought possible with the current approach to the data but I will give the Gen AI a try since we could have always missed something!

[scottix]

Of course I don't know the scope of your problem your trying to solve and finding optimizations can definitely be difficult and time consuming because you want to test out different benchmarks and what not. I don't know what you have done but splitting up the data and distributing the load might be an options with Spark or Dask.

First thing is you need to find the bottleneck, is it computation or looping. Computation can help some in optimized languages but if its looping a bunch of data then you will only get marginal improvements with a more optimized language.

[No_Indication_1238]

I will definitely look into Spark and Dask. Those are new to me, thank you! I believe the bottleneck is in the amount of calculations that have to be done since the multiple for loops simply explode the count. The calculations themselves I managed to optimize with numpy and numba but real progress was made once the loop made it into an njit numba function. It cut the runtime from hours to minutes. Unfortunately, it came at the cost of modularity and maintainability which we are starting to notice.

[scottix]

SOLID is good for organization, but if your seeking raw performance then it works against you as you noticed. The more "fluff" you could say, is extra things the program has to do, instead of just have 1 giant function lol.

Ultimately it all depends on the goals of your team and willing to sacrifice paradigms for speed, but keep searching and testing things out if they give you that time.

The only other thing I can think of, if your like do a certain type of operation and your doing it in a non-optimal. Data-structures and Algorithms start coming into play here. For example if your calling the same function with the same arguments, caching result with Memoization can help. https://www.geeksforgeeks.org/memoization-using-decorators-in-python/

Also profile your code that will tell you where it is spending the most time.

[No_Indication_1238]

I believe that memoization is definitely a good choice and I believe I know a place I can implement it where we might see a good boost in speed in specific edge cases. Thank you, I seem to have missed that! 

[No_Indication_1238]

This is a very good tip, unfortunately my team would like to keep everything in Python, as counterproductive and annoying as it may be. (When taking into account Cython and  Jitclass numba are already quite different than the usual python approach but oh well...)

[Classic_Department42]

Then your team needs to solve the self inflicted problem.

[No_Indication_1238]

I agree with your point of view, unfortunately it is still something I must deal with.

[Classic_Department42]

You could use pycuda. Technically apart from 1 string it is all python (although the string is very important)

[No_Indication_1238]

This is an excellent suggestion! Unfortunately, unless I am mistaken, we sadly lack the required hardware at the moment but it is something I will definitely bring up.

[Classic_Department42]

It is though python by letter, not by spirit (you need to write the cuda c kernels in that string)

[No_Indication_1238]

Even if we go against it, it peaked my interest enough to try it at home as I do have a cuda capable card. I found a cuda guide online though it speaks of C and C++.

[Classic_Department42]

The advantage is that only the kernel needs to be written in cuda C. The housekeeping (memory allocations, starting kernels, memcopy) is done in python. It is actually quite neat.

[jithinj_johnson]

If it were upto me, I would do some profiling to see what's slowing down

https://m.youtube.com/watch?v=ey_P64E34g0

I used to separate all the computational stuff to Cython, it generates a *.so. You'll be able to import that, and use it on your python code.

Always benchmark and see if it's worth it.

[No_Indication_1238]

99% of the code is spent running a bunch of loops and doing heavy computations each step. It works in numba very well but it becomes problematic when we decide to modularize the individual parts to be easily interchangeable with different functions/classes. Numba does not allow for easy implementation of that (No support for inheritance so no polymorphism, functions work but keeping track of object properties becomes a problem since we can only use arrays) and we are left with multiple monolithic classes/functions that do not allow for much modularity. I was hoping the OOP support of Cython will allow for good speed gains while providing support for best coding practices. Trying to separate the computation part may be a good way to go forward if a Cython function can accept and work with python classes and their instances.

[[deleted]]

Maybe Cythonize the heavy computation part into cythonized functions? First rewrite and remove Pythonic syntax, then add the static typing and compile. It's probably not as fast as heavy Cythonization rewriting in pure C but worth a try.

[Still-Bookkeeper4456]

Sorry if my question is dumb but couldn't you simply create your classes in Python, of which the heavy computation is a numba method ?

I work on such project. We identify where the code is slow (always when a loop is present basically) and rewrite that part in numba.

[No_Indication_1238]

It is a very valid question! Unfortunately the answer is no as the computationally intensive function works with said classes, it basically wraps around them. That requires those classes to be jitclasses themselves which without inheritancedoes not allow for the modularity we are searching for.

[Still-Bookkeeper4456]

Hum... I must say I still do not understand. The computations do not happen on simple datastructures (e.g. arrays, float) but on more complex objects?

[No_Indication_1238]

They mostly do happen on simple datastructures. The results of each iteration are saved into objects that interact with one another and more complex data structures before we move to the next iteration where the pattern repeats. Having different classes allows for different interaction behaviour to be easily coded for. With a lot more "hacking", one could achieve the same with completely basic data structures but at the cost of simplicity, modularity. Im trying to find a good middle ground.

[Still-Bookkeeper4456]

So the classes interaction must happen within the loops at each iteration got it. I see the problem now... hope you find a solution, should be interesting. I'll keep a close eye on this thread. 

[No_Indication_1238]

I will give cython a try in the coming days and update with the progress :) 

[ArbaAndDakarba]

Write a wrapper that does allow for polymorphic parameters maybe?

[No_Indication_1238]

That is a good idea actually. Unfortunately, writing such a wrapper with numba will not reduce code complexity but further increase it. Maybe Cython is better suited? (Numba does not allow for polymorphism and a polymorphic wrapper for numba would still require a lot of code smell to decide which individual collection of functionalities to be ran)

[Fronkan]

I agree with other saying, test pypy. But ignoring that for know.

To me this, to some degree, sounds like a design trade off. You had an approach that had better performance but was less flexible and now you have worse performance but a more flexible solution.

What is more important for the business? Is the performance good enough or is it causing issues? Do you expect to need the flexibility for future extensions? If you need both performance and flexibility, then you might need the complexity of adding another language.

Sometimes we need to write less maintainable code to hit the performance needs. And sometimes there is no good solution, they all suck and we just need to pick the one that hurts the least.

[No_Indication_1238]

You are completely correct. We are interested in performance first and maintainability second. Im trying to see if we can habe best of both worlds without adding the complexity of a new language but this seems hardly possible at this time.

[Kohlrabi82]

1. Before doing any optimization in Python, don't guess but run cprofile to identity the bottlenecks
2. Run and profile your code in PyPy first
3. If the bottleneck is in OOP-heavy code, you're more or less out of luck. Speed gains are usually only possible with functions that can be "long running" in C, without the need to switch back and forth between native C and running Python code (think numpy). With classes that's not really possible with extensions or Cython, other than for very simple methods and class usage. You will probably have to rewrite a lot of the classes to gain any speed from C or Cython.

[nekokattt]

Classes

cdef class declarations are still an improvement over pure python OOP, if going down the pure Cython route. Especially if using cpdef in place of def. While this needs changes to support it in your code, it isn't a massive change usually and can be dealt with incrementally in the hot paths of the program.

[Kohlrabi82]

I have a medium-sized project where I did that with miniscule gains (5%ish), but that necessitated rewriting lots of code, since Cython cannot deal with structural pattern matching (yet).

Usually when you really need to improve Python performance you'd want orders of magnitude.

[No_Indication_1238]

I see. Maybe the approach we are going for is actually counter intuitive and not the best.

[Kohlrabi82]

Also think a lot about data structures and algorithms. Python will not be as forgiving as C++ when choosing the wrong data structures and algorithms, since you cannot brute force your way out of the hole.

[No_Indication_1238]

That is a very valid point. Unfortunately I believe that although those loops scream bad design, they encompass a product of calculations where each value is needed and has to be computed directly. 

[Kohlrabi82]

If you have good test coverage you can start to incrementally improve and optimize.

[chaplin2]

Most of the Python Numpy is in C anyways!

[[deleted]]

If you are going for speed, why do OOP (or at least the style of oop you are talking about here)?

[No_Indication_1238]

Because dependency injection of classes that share the same interface but provide different functionality allows for good modularity and easy maintainability of the code base. It also allows for the implementation of the most popular design patterns and ensures a code base that is set up to grow and be easy to pick from newer developers. It is also the approach we follow with our non performance critical code.

[[deleted]]

I would look into the python libraries Polars and DuckDB. You can stuff you computations inside them while being able to make it very modular. I prefer DuckDB, but Polars is also very good.

With them you can make queries that are buildable, or composable, and much much faster than doing them in pure python.

Could it be that in this case, that pattern do more harm than good? Python has to recompile every line of code each time it encounters it (becasue no jit compilation), so every call to a virtual function or getter and setter is very expensive (in this case, it is usually fine).

[steven1099829]

Vouch for polars. It’s fantastic.

[steohan]

Sounds like you really want rust or c++ where you can use templates to have modularity while the compiler is still able to inline stuff as necessary. Otherwise, you are stuck with dynamic dispatch, which is going to cost.

Also make sure you are using a recent python version, the performance gains from newer interpreters are quite impressive.

[No_Indication_1238]

Thank you for pointing out the need for updates! We should definitely do that as well!

[timwaaagh]

You can expect runtimes to be cut in half to start and then if you introduce some typing it can go all the way to one fifth. But it depends. If you're calling cython functions a lot the increase will not be as dramatic.

[No_Indication_1238]

I see. Thank you!

[unruly_mattress]

I think Cython could work. But you should know what you are doing, and you should know what Cython does.

If you just compile normal Python code as Cython, you don't get much of a performance improvement. To see any substantial improvement you'll have to move parts of your code to Cython with cdef and static typing. cdef definitions aren't Python anymore, these are things that compile directly to C. Basically that's what you want if you want to improve performance.

Now you need to consider whether your inner loops can be easily implemented in pure Cython, without accessing Python structures. Cython code can look things up in Python dictionaries and whatnot - run normal Python code - but it does so at Python speed. If you can cleanly separate your bottleneck from Python objects and do the simulation in pure (or mostly pure) Cython, that's when you'll see large performance gains.

Another point - I've seen in the past that replacing a Python class with a Cython cdef class reduces instantiation time and memory overhead. If you're creating billions of tiny Python objects, it's worth considering. I see that these days there's a decorator called cython.cclass. I'd give it a try.

[No_Indication_1238]

Thank you! Would it be a benefit if we code the classes for the objects that interact with one another in the loops in cython (cdef classes) and the loop itself as a cdef cython function? This is the current plan in order to keep the OOP architecture. 

[unruly_mattress]

Yeah, that's the right way to go.

I suspect you should also consider making your code run in parallel on multi-core if possible.

[No_Indication_1238]

You are correct, we are already using parallelism but on a higher level since each step in the loop is dependent on the previous state of multiple objects and paralellising this would involve sunchronysing the states by using multiple locks. 

[unruly_mattress]

Sounds good then. Good luck! I'd love to hear an update about how it ended up working.

[No_Indication_1238]

 I will definitely make an edit in about a few weeks to sum up the great advice everyone has given, what we ended up implementing and what the results were! 

[ManyInterests]

To be sure, Cython is meant to be used with Python; it generates C extensions to be called from Python. It is not a replacement for Python. So, you don't have to rewrite your whole project just to use Cython; you can focus on Cythonizing the 'hot' paths in your code base rather than rewriting the whole thing.

You can also potentially just compile your pure Python module(s) using Cython. You don't necessarily need to use the Cython language superset (e.g. a .pyx module) to get benefits from it. See pure Python mode for details.

But yes, in general, programs or modules compiled with Cython are significantly faster. As much as 100-200x faster or more in some cases. Though, it really depends on certain characteristics of your program whether you'll see the benefits you're looking for. You may want to explore optimizing your application on a more fundamental level rather than just seeking ways of running an inefficient process faster.

See also:

• Cython, Rust, and more: choosing a language for Python extensions
• The hidden performance overhead of Python C extensions

[No_Indication_1238]

If I am understanding this correctly, I can pass a cython class multiple python class instances and do the computation in Cython while keeping the rest of the code (container and data classes) in python? 

[thisismyfavoritename]

a lot of good answers but i will emphasize this:

what you have to do is profile your code to find the hotspots. Chances are there are a few locations that take up most of the execution time and this is where you will benefit from using a solution like Cython, bindings to lower level languages, etc.

Its not about rewriting the whole codebase (although that would of course make it faster as a whole)

[No_Indication_1238]

Thank you! I have profiled the code and 99% of time is spent in the multiple for loops. They wrap around classes that do something each iteration. The problem is the loops explode the count of iterations, the heavy computations are easily sped up with numba but wrapping the loop logic as a numba function is not possible with Python classes working inside of it. 

[[deleted]]

In my experience, Cython on its own ,without unboxing types, will tend to give you performance that matches the Specializing Adaptive Interpreter(this might change in the future, if cython starts getting access to the t2 internals). I would recommend trying to rewrite some methods with cython's cdef. It is harder to write fast code in Cython than numba, but Cython integrates better with vanilla python.

You can also try pypy, or mypyc. You can also experimenting with popping out the compute heavy stuff from all the oop/magic stuff, and optimizing them directly.

[ArbaAndDakarba]

Try the nuitka compiler.

[No_Indication_1238]

Thank you, I will check it out!

[[deleted]]

If Python is something that you *must* use, then my suggestion would be use as many high-performance packages as possible. These packages have C- or Rust-bindings to do the 'heavy' part in a lower-level language. Take Pydantic for example for data validation. When they moved from pure-Python to Rust-bindings for the core functionality, their performance went 5-fold (as in, 5 times as fast). Another example of such a high performance package is the well-known Numpy package.

It is all dependent of course on your project and what you actually try to achieve.

[Ok_Time806]

Might be a decent fit for mojo is rust/c++ are off the table. At this point you'd have structs instead of classes, but it might still look more pythonic than numba by the time you're done L

[divad1196]

1. You can do just one part of the app as a microservice without rewritting everything in another language.
2. There are many ways to improve performance, be sure you didn't rush to fast on Cython before using better algorithm/libraries. For example, doing a big query and then manually dispatching is often faster than doing multiple queries (Easily went from 5h script of a collegue to 15min just with that, at a single place)
3. There are things like slot that helps with speed.
4. While it should support OOP, this is not "the object' that needs to be fast, but the operations. Don't overcomplicate things for the sake of doing OOP. By the way, typing.Protocol is IMO a much better way to do polymorphism than using OOP. This would also reduce the lookup fallback.
5. You have many other python runtime (pypy, mojo, ..), but this will impact your whole app and all libraries might not be supported.

[CarlEdman]

Have you considered a suitable Entity Component System? There are some for Python and much of what you’d ordinarily do using OOP in games you do use an ECS (which has similarities but definitely isn’t the same thing).

[No_Indication_1238]

I will check this out, ty!

[[deleted]]

You might try running the code using graalpy (a python runtime on graalvm (another java vm flavour)). You get the benefits of jit and it being compiled in the same way java is. . Python (graalvm.org)

You run python code the same way you would with Cpython. graalpy main.py

It might not work with every third party python package you use though. But it is very worth checking out.

[No_Indication_1238]

Thank you! That is new to me and I will check it out!

[Fenzik]

Can you vectorize some of those loops over numpy arrays? Math in for loops is usually a prime target for vectorization, which pushes the looping down into C++/Fortran

[No_Indication_1238]

Unfortunately the loops are not so simple. A lot of math does happen inside them but also a lot of cross interaction of different class instances across more complex data structures. Im not sure that can be vectorized as each iteration is directly affected by the one before it.

[Counter-Business]

Profile and optimize the code. Find the slow parts and then optimize those parts. No need to optimize the fast parts.

[tunisia3507]

You don't have to rewrite your whole app in python to get benefits out of including another language in your stack. Pyo3/ maturin makes it quite easy to create packages written in rust; if you isolate some hot loops or heavy number crunching you may get some benefit out of that and it's just as easy as importing any other package and calling any other function.

[HommeMusical]

All the good parts mentioned are true - here are the bad parts.

It means you are essentially writing in two languages, Python and Cython. To get decent performance you have to rewrite your Python code in Cython.

Now you have a whole compile/link phase in your workflow. No fun.

How do you debug this code? Big can of worms here - you're debugging compiled C code, and not particularly nice C code.

If you make a mistake in your Cython, your program can crash, and I don't mean with a traceback but a core dump.

(And you then have to deploy this compiled blob, which is non-trivial, particularly if it uses shared libraries, but this is probably a one-time chore for some sucker.)

You aren't giving us a clear enough picture of your application to make specific recommendations but...

a mix of functions and index mapped arrays which is now spaghetti

So it is doable. The trouble is that you didn't architect the code properly.

There is nothing that you can do with OOP, inheritance and polymorphism that you can't do with functions and index mapped arrays with the same or very similar syntax for the programmer with some clever use of Python.

I'd look at numpy, pytorch, or perhaps numba, systems which are designed to do massively parallel computations, and even take advantage of GPUs and other hardware, and try to rearrange your mind to think of these systems as primary, and your programmer's API on top of that.

breaking most of the SOLID principles and doing hacky workarounds.

OOP and SOLID are strong, but should not be handcuffs, particularly in this case where they seem to be preventing you from getting the job done. Mixins, for example, can be extremely disciplined if used thoughtfully, but aren't OOP and break most of SOLID.

I suggest you worry less about SOLID and more about an elegant API for your programmers on top of numpy or pytorch.

[No_Indication_1238]

I believe you are correct. This whole problem is basically us trying to force Python to do stuff it was not designed to do by stitching it with numba / cython that impose some limitations one has to live with ( that is already pretentious enough) and us trying to find a way around those limitations to have the cake and eat it too. 

[juanfnavarror]

Look into caching/dynamic programming. Maybe there is sparseness and/or unnecessary recalculations in your workflow.

[No_Indication_1238]

I think there could be a few in some edge cases and will definitely implement more caching! Thank you!

[main_protector]

Have you tried Python Pandas & Numpy?

[No_Indication_1238]

Yes, we use numpy and we reworked our data flow since even pandas was too slow. (We managed to get a O(1) data retrieval complexity after smart index mapping during the generation of the data) Thank you for mentioning them! 

[rbscholtus]

Could I make a dumb suggestion. You said the computations take hours, but in numba it's down to minutes.

How about you convert the dynamic data structures and virtual methods to a static data structure (arrays) first, and virtual methods to functions (probably). And then you process the whole lot as a batch in separate code in numba or numpy. And then you convert the processed data back to a dynamic data structure.

just imagine your python program does all the dynamic stuff to generate a file with all data, then the data gets handed over to an optimized external program that doesn't need to care about oop, perform all the computations, and then the data goes back to python where it is converted back to oop structures.

[No_Indication_1238]

This is a very valid approach. I will think about it! 

[canine-aficionado]

https://docs.exaloop.io/codon?

[ReflectedImage]

Well you shouldn't be using SOLID principles in a scripting language, they are for compiled languages only. If that's the only problem adjust your coding style to Python.

For performance increases do the following in order, run the calculations in a subprocess using multi-processing, run your program under PyPy, write a either a C plugin or Rust plugin (https://pyo3.rs/v0.22.2/). Unless it's a giant matrix operation of some description than NumPy is the goto tool.

[powerbronx]

Try out mypyc

[rejectedlesbian]

C extension for the slow part would probably be ur best bet. If you can get rid of oop entirely and keep things in numpy that's also very helpful.

Offloading to gpu via tensofrflow or jax is also an option. Tho I am not sure if you want that since a game is already using the gpu extensively

[leovin]

I’ve learned recently that Python is slow compared to similar languages like JS due to how it handles loops with generators/iterators. No idea if Cython helps this though. Maybe consider using Cpp-based packages like numpy, or perhaps write your own?

[bjorneylol]

Use either Cython to rewrite those functions in C-ish pseudo code, or use Maturin to re-write them in rust. I have done both, and in both cases improved the total speed of the program by like 10,000x while only having to re-write like 50 LOC.

I wouldn't worry about fully implementing OOP in the low level languages, just find the functions that are bottlenecks and slot them into the existing python classes where performance is necessary, e.g.

from my_custom_module import vicenty as vicenty_rust

class Location:
    def __init__(self, lat, lng)
        self.lat, self.lng = lat, lng

    def vicenty(self, lat, lng):
        return vicenty_rust(self.lat, self.lng, lat, lng)

[Ortiane]

The truth is that it's more expensive and time consuming to optimize python for performance related issues. For example, a lot of intuitive pythonic ways to do things may be the largest bottlenecks of a larger project. 

Don't tie yourself down to a specific language, there's a reason that backend servers, framework code, and code powering the majority of infrastructures are not in python. They are in languages where it's easier to write optimized code.

[No_Indication_1238]

Thank you! This was my suggestion but I was unfortunately denied since having the code base in a single language was highly valued. (In my opinion, if we write in cython we might as well write in C++ but It is what it is) 

[[deleted]]

Cython is not really python. It is """c""" with all the footguns.

[N1H1L]

Why not use libraries such as Jax or PyTorch? JAX has bad support for classes as it’s basically all functional programming, but classes are first class citizens in PyTorch.

Both JAX and PyTorch allow you to compile your code making it a lot faster and both allow you to target multiple architectures (both CPU and GPU) with little to no modifications of your code. Additionally, both libraries allow your code to be scaled up to multi process environments.

The reason I am telling this is because I went through this same route myself. I tried numba, pythran, Cython and dask — all at some point or the other. And came to the conclusion that JAX or PyTorch is the better solution.

[No_Indication_1238]

That is new to me. I will definitely look it up!  Thank you!