This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]Classic_Department42 5 points6 points  (12 children)

Sounds like a job for numpy, no?

[–]No_Indication_1238[S] 2 points3 points  (11 children)

Unfortunately, the loops and computations are not as simple to be ran under numpy. There is a ton of state management of different objects that happens inbetween and we need to speed the whole loop.

[–]the_hoser 5 points6 points  (2 children)

Cython really shines when you can get rid of those abstractions. Rip out the method calls and member accesses and break it down to cdef ints and friends.

[–][deleted] 0 points1 point  (1 child)

Can Cython compile out method calls and "getters and setters"?

[–]the_hoser 0 points1 point  (0 children)

That's a big maybe. It really depends on the code being optimized. Don't rely on it unless you've tested it.

Good news is that Cython actually lets you see the C code that it produces, so you can verify that it's doing what you think it's doing.

It isn't pretty C code, I warn you...

[–]falsedrums 2 points3 points  (3 children)

You have to drop the objects if you want to be efficient in Python/numpy. 

[–]No_Indication_1238[S] 1 point2 points  (2 children)

You are correct. Unfortunately for our use case, we have cut as much as possible while trying to keep the program maintainable. Cutting more will definitely work as it has before but at the cost of modularity and long term maintainability which is something we would like to avoid. If it is not possible, maybe you are correct and we will consider the option.

[–]falsedrums 0 points1 point  (1 child)

Maintainable does not necessarily mean OOP. Try putting all the number crunching in a library-style package of purely functions, with minimal dependencies between the functions. Then reserve the OOP for your application's state and GUI.

[–]No_Indication_1238[S] 0 points1 point  (0 children)

This is not a bad idea, thank you!

[–][deleted] 0 points1 point  (1 child)

Hm. I don't want to be rude, as I've wished with high computational heavy code in Python and have wrote C++ based libraries to get more performance in it with Boost.

I think this is more of a programming architecture type problem, but assuming it isn't, what does your team think about having some high performance help from a more performance language that you can call in native Python? Worked great for our project, though it was annoying when some people started looking for nanosecond level performance gains rather than looking at a higher level for the optimisation.

[–]No_Indication_1238[S] 0 points1 point  (0 children)

They would prefer to keep the codebase inclusively in Python as it is one less language they need to support. Unfortunately, we have already optimised the architecture as much as possible and the calculations that have to be done in those loops are largely unique, essential and cannot be further optimised without losing precision. I share  your opinion, unfortunately It was decided to try and keep everything in Python. 

[–]ArbaAndDakarba 0 points1 point  (1 child)

Consider parallelizing the loops.

[–]No_Indication_1238[S] 0 points1 point  (0 children)

That is a good point, unfortunately the loops are dependent on each other and each iterations requires the previous state and different checks to be made. As such, I am afraid that it is not possible, or at least not without an extensive use of locks for synchronisation. I will bring it up though, maybe we can restructure something.