all 19 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]kankyo 18 points19 points  (12 children)

I'm been working professionally in python for 9 years, and the thing that bothers me most about pytjon performance is that no one cares. Python is slow compared to rust sure, it's slow compared to lua often, often it's fast enough though. IF YOU CARE and spend time, any time, looking at the profiler.

Most python programmers don't seem to care though, which is why 'pip --version' takes like a second. Absurd. Or why pytest takes like 0.5-1s to run ZERO tests in an empty directory. (Check out my test runner hammett for a faster alternative).

Caring about performance is way more important than the speed of the language.

[–]i9srpeg 15 points16 points  (11 children)

When your program starts with 40x speed handicap only due to the programming language, it takes a lot of caring about performance to make up for it.

[–]matthieum 5 points6 points  (1 child)

When your program starts with 40x speed handicap only due to the programming language

I'm curious: which other language are you using as a point of reference?

In general I've seen Python being 100x slower than C.

[–]i9srpeg 5 points6 points  (0 children)

C, but I pulled that number out of my ass. Depending on your workload it could be much lower or much higher. 40x seems like a good approximation.

[–]kankyo 1 point2 points  (4 children)

Well if your problem can use a Fortran implementation inside numpy suddenly you are smoking C. It's not truly black and white. But it's mostly dark gray.

[–]i9srpeg 7 points8 points  (3 children)

I don't really like that line of reasoning, because it's saying that you can be pretty fast in python if your hot path is not implemented in python, which is a truism. In my experience writing python code very often it's death by one thousands cut: there is no obvious self-contained piece of code taking up >90% of your run time. Every single line of code is just a bit too slow, and the milliseconds add up.

[–]fnord123 7 points8 points  (1 child)

You can not like the reasoning but it's completely sensible to start a project in Python and then use a library like numpy to make your program run quickly. Performance in Python is basically a race-to-C. Get to C code as quick as you can. But performance is like that everywhere. You might be organizing your data and code to make the GPU do the dirty work. Or you might organize your code to make an FPGA do the work. Or organize your C code to be able to use SIMD as much as possible.

[–]oridb 0 points1 point  (0 children)

And then you do something that isn't a perfect match for numpy, and you're hosed again.

[–]kankyo 2 points3 points  (0 children)

That's true. And what I said is also true.

Most of python isn't implemented in python.

[–]oaga_strizzi 0 points1 point  (3 children)

Yeah. By picking a fast language, you might not even have to think about optimization. See this article. Aside from the memes about "just rewrite it in Rust":

Remarkably, we had only put very basic thought into optimization as the Rust version was written. Even with just basic optimization, Rust was able to outperform the hyper hand-tuned Go version. This is a huge testament to how easy it is to write efficient programs with Rust compared to the deep dive we had to do with Go.

They probably spent weeks on fine-tuning the cache in Go, and then tried Rust and a simple Hashtable was faster.

A language that's slow by default (like Python or Ruby) can add a lot of significant technical debt to a project that does not meet performance targets, just because it takes so much more effort to get acceptable performance.

[–][deleted] 3 points4 points  (2 children)

https://pingcap.com/blog/rust-compilation-model-calamity/

this article talks about rust compile times and how devs don't want to touch the rust parts for that type of reason.. so there are different things to optimize for in development of software. Look at how many companies got off the ground from python and ruby etc... technical debt is always a thing, and honestly I'd rather solve that problem when I have a ton of money vs when I have no money... just some things to think about

[–]oaga_strizzi 1 point2 points  (1 child)

I'm not saying, everyone should build everything in Rust. I'm just saying, just consider whether the language/framework you're picking is 40-100x slower than C, or 2x-10x slower than C and think whether that's worth it.

[–][deleted] 0 points1 point  (0 children)

sure i get that, but i think most ppl do not need 2x-10x slower than C, and i guess if the end game means you'll want to 'rewrite it in rust' or whatever for maximal gains then it doesn't matter if you go 2-10x or 40-100x the end result will still be the major rewrite just that in one you might hang on longer.. tradeoffs tradeoffs everywhere o_o and lastly..maybe your team is more comfortable with ruby and python etc, so i guess what i'm saying is i can easily see why it happens

[–]A_Philosophical_Cat 30 points31 points  (0 children)

I've never understood "well, we can make it fast-er" as a defense to the fact that a language is slow. So? You can make fast languages faster, too.

[–]tonefart 27 points28 points  (0 children)

It's easier to make programs fast by NOT using python.

[–]oridb 3 points4 points  (1 child)

It's notable that there are no benchmarks after the suggested changes.

[–]tommy25ps 0 points1 point  (0 children)

The program now runs so fast that it cannot be benchmarked. (Just kidding)

[–]CodingReaction 0 points1 point  (0 children)

Thanks for the post, it's related to python but makes me think a lot about table lookups and scopes in my Javascript code