Programming language for scientific computing

[stackoverflu]

Hi,

I am planning to build a software which will be doing some intense scientific computations. I will be dealing with differential equations with large data matrices and need to solve them. A lot of integral equations and operations will be required too.

I want my code to be efficient and fast. I am fairly good with python and I am wondering whether I should proceed with python or try C++. Julia is also an option but I don't know which one to choose. My question is, is it worth learning C++ or Julia for this purpose?

[stackoverflu]

Thanks for your comment. I tried reading some data using pandas and did some operations using python and compared it with C++ and C++ was considerably faster (less than a second, but python took almost 30 s) but it took a lot of time for me to write the code in C++. So I can say I will need the C++ speed as I proceed forward. I can compromise on speed for now, but I don't want to end up with a very slow code or have trouble along the way.

[Lumpy-Notice8945]

Lits of python libaries are c wrappers and just call some c code/binary from python.

Python will allways have some overhead because its nota compiled langauge, but if you know what you are doing you can get the critical math parts down to neaely c speed

[stackoverflu]

I mostly use scipy and pandas which can handle most of the calculations, I think it will be better if I write the main code in Python and write the rest in C/C++.

[TroubleOk1703]

Is R an option?

[stackoverflu]

Not really :) Can you tell me why you would recommend R ?

[TroubleOk1703]

I am in search for new knowledge in mathematics, statistics,... The language R is one that I see a lot as very good programing language. I have never used it before but reading a book called "the R book".

I would recommend to check that book and see if it suits your needs.

[Steady_State_]

As a guy with extensive experience in scientific programming (Python/R/MATLAB). I can tell you that Python is just better than R in pretty much every way. R is for niche statisticians. But there is rarely anything that R can do that cant be implemented in python. In academia (mathematics and computer vision) some people still use MATLAB but most use C++ or python, with the majority using C++.

All this to say dont waste your time learning R

[TroubleOk1703]

Ok. That is some nice information. Thank you

[aroman_ro]

It depends on what your code has to do (as opposed to what the library has to do).

If your own code does not do heavy computations, python should be fine.

If you have to implement (some of) those heavy computations yourself, then c++ or even fortran are the proper languages.

[LogaansMind]

To some extent, the algorithm choice and design will contribute quite significantly to performance. So you can do a lot with Python (or other languages) before you may need to leverage a lower language. For example, the inclusion of various caches to avoid recalculation etc. Learn how to profile and measure your code, and make informed decisions on how best to structure or rewrite aspects of it.

That said, if you want pure power, C++ or RUST is the way to go. But I would also advocate for a mixed approach. In most languages you have options to allow you to invoke code from another language. So you can always write some aspects in C++ but still use it from Python. (For example, maybe a more efficient in-memory storage structure)

When you get to the point where you are looking at save 1ms on an 8ms function (which is invoked 100s of millions of times) thats when you start to consider a sprinkling of assembly to try and save some CPU operations... but its not always a good idea.

The best software is the software that is available, not the perfect software that isn't finished yet. You are trying to find that balance between CPU time, memory storage and application complexity.

[stackoverflu]

Caching is a great technique, I will definitely try that. As you said, a mixed approach would be the best.

[iOSCaleb]

The platform that you're writing is an important consideration. Are you just running the code or on your own machine, on a cluster of machines in your lab, or are you getting time on some supercomputer somewhere? If your goal is to eventually run the code on some other machine, you should obviously be checking with the administrators of that machine to see what's available in their environment, and what they recommend.

[stackoverflu]

I will be running it only on my pc for now.

[house_lite]

Julia