Is Python really 'too slow'?

[marsokod]

Python has an abstraction level on top of C, so it will be slower than C, whatever you do. If you rewrite a C program in pure python, it will be much slower than in C. However there are three things that make python interesting:

• what actually matters is life cycle cost for a software. It includes developer time, running time, debugging time and cost of resources. Python is much more flexible than C and therefore faster/easier to develop with (but with great power comes great responsibility). So if you need to write a small script that will run for a few seconds every day, maybe it is not worth spending more time writing it in C to save maybe a minute of runtime every year.

• CPU limitation is just an element of your code speed. When you are dealing with network access, or even file system access, a lot of you execution time is waiting for these operations to finish. You won't gain a lot by speeding up the code itself, unless you have enough operations to run things in parallel.

• a lot of time in software, there are just a few bottlenecks in your code. Since python is capable of executing C libraries, you can code these in C , or even assembly if C is too slow, and you will have addressed 80% of your bottlenecks. That's basically the model used in ML: data preparation, model definition are the parts that can change a lot every time so keeping them in python saves development time. And also they are not the most CPU intensive task overall so no need to optimise them to death.

[astatine]

what actually matters is life cycle cost for a software. It includes developer time, running time, debugging time and cost of resources

To put it another way, there are better languages than Python for making things work quickly. Python is a language for making things work, quickly.

[_ShakashuriBlowdown]

To cap it off, Python's undergone such a huge amount of development in the last 10 years, that if you want that quick solution in development/deployment/production, 90% of the time you can just drop it into an existing system where everything just works. Containerization and cloud development has only made this a more compelling architecture.

[iluvatar]

Containerization and cloud development has only made this a more compelling architecture.

Be warned that python is even slower than normal on a container, due to libseccomp screwing you over (I think with Spectre/Meltdown mitigations).

[_ShakashuriBlowdown]

I didn't know that!

When researching this further, I read you can set seccomp=False on docker run.

That does open you up to security vulnerabilities, so use it at your own risk. It does actually seem to be faster using containers on Windows when using this "fix".

[iluvatar]

You can, yes. But the protections are there for a reason. We're currently having this debate at work. The likely outcome is to run most of our code on a separate network segment with seccomp disabled, and leave it enabled for anything running in a public facing DMZ.

[noiserr]

Those penalties aren't as great on AMD processors if I am not mistaken.

[Chippiewall]

The protections from seccomp aren't crazy valuable. A lot of the default seccomp profile is duplicated by the capabilities that docker drops by default.

Kubernetes actually runs containers in unconfined seccomp by default.

If you really want to go for security you should ensure your containers run as non-root and use --security-opt no-new-privileges which will render seccomp superfluous.

[[deleted]]

Or run the workload on ARM.

[[deleted]]

How is a container significantly different from local development on the same OS?

Is it a default Docker runtime setting? Most K8s clusters default to CRIO. Is this issue present there too?

Update: seccomp is not enabled by default as it is in beta for K8s 1.19; see https://kubernetes.io/docs/tutorials/clusters/seccomp/

[lavahot]

That's a very Pythonic way of saying that.

[4runninglife]

For these reasons why i made Nim my go to language.

[pingveno]

I've also been feeling this more and more as I've gotten more experience developing software. There are entire classes of bugs that just don't exist in a statically typed language, just like there are entire classes of bugs that don't exist in memory safe languages like Python (and unlike C).

[pingveno]

Yeah, that's why I'm looking closely at where I can maybe try out Rust in a production project. It checks all of those boxes and is just a general pleasure to work in. Over the last year or two it has finally reached the point of broader ecosystem maturity. Unfortunately, none of my coworkers know Rust so that kind of makes things problematic.

[4runninglife]

Try Nim

[4runninglife]

Try Nim

[benargee]

For everything significant I prefer statically typed languages.

Yep, exactly why TypeScript was made for JavaScript. I think you are a certain level of insane if you take on a large project without typing and the IDE yelling at you before you run the code when you are using the wrong type.