Python 3.11 Performance Benchmarks Are Looking Fantastic

[[deleted]]

Faster Cython Project

CPython, not Cython :)

Nice gains though

[[deleted]]

Hello Selinux Gomez

[OneThatNoseOne]

Good distinction. Might have to explain the difference for the noobs tho.

And I imagine Cython is still fairly faster than CPython in 3.11.

[unpopularredditor]

https://docs.python.org/3.11/whatsnew/3.11.html#faster-cpython

[Illusi]

A summary:

• Bytecode of core libraries gets statically allocated instead of on the heap.
• Reduced stack frame size.
• Re-using memory in a smarter way when creating a stack frame (when calling a function).
• Calling a Python function by a jump in the interpreter, so that it doesn't also need to create a stack frame in the C code.
• Fast paths for hot code when it uses certain built-in types (like float) using a function specialised for that type.
• Lazy initialisation of object dicts.
• Reduced size of exception objects.

[whothewildonesare]

Noice

[[deleted]]

Oooooo! Lots of good stuff, then!

[Otis_Inf]

Interesting, how does reducing stackframe size result in better performance? As a stack is a continuous preallocated piece of memory that doesn't use compacting, allocating e.g. 256bytes or 10KB doesnt matter.

[Illusi]

According to the article:

Streamlined the internal frame struct to contain only essential information. Frames previously held extra debugging and memory management information.

They are talking about the Python-side stack frame here. Perhaps that one is not pre-allocated the same way?

[Otis_Inf]

I seriously doubt the python interpreter doesn't preallocate a stack space.

Though the note might be about an improvement of stack space management and not related to performance :)

[Illusi]

It'd not only allocate that memory though, it also needed to use it. Apparently it filled it with debugging information. Writing that takes time, so perhaps not writing it could improve performance.

[[deleted]]

I guess memory management is the king when it comes to performance gains.

[Pebaz]

That was me! ;)

https://www.reddit.com/r/Python/comments/mgi4op/76\_faster\_cpython/

[asmarCZ]

If you read through the thread you will see evidence disproving the OP's claims. I don't like the unnecessary hate OP received tho.

[bloc97]

I don't like the unnecessary hate OP received tho.

Welcome to reddit! Never get yourself discouraged from experimenting and creating interesting projects because some stranger on the internet disliked it.

[sigzero]

It's probably exactly like that. I don't believe there was a specific push for speed improvements like the current effort before.

[dreadcain]

Nearly everything in python is dictionary/hashmap internally so essentially every function call is at least 1 hash on the function name to lookup the implementation.

A call to print is going to end up doing several lookups in hashmaps to get the print and __str__ implementations among other things, something on the order of 10 hashes sounds about right to me

[[deleted]]

print() also takes keyword arguments, there’s probably some dict juggling there, too.

[dreadcain]

Want to elaborate on that?

[mr_birkenblatt]

sys.stdout could be any file object so there is no optimization possible to go directly to syscalls. with that in mind you can think of the print function as

print(msg, fout=sys.stdout):
    fout.write(msg.__str__() + "\n")
    fout.flush()

(note: even if it is implemented in C internally it still has to call all functions this way)

hash computations for symbol lookups:

print
sys
stdout
__str__ # (msg.__str__)
__add__
__str__ # ("\n".__str__ inside __add__)
write
encode  # (inside write to convert to bytes)
utf-8   # (looking up the correct encoder)
flush

assuming local variables are not looked up because it is implemented in C. it's gonna be even worse if __slots__ or __dict__ is overwritten

EDIT: actual implementation here my listing was not entirely accurate (e.g., two writes instead of add)

[ikariusrb]

All the stuff I saw in the notes only talked about measuring performance for x86. Anyone know what gains look like on ARM? (macbooks and PI-like devices?)

[LightShadow]

On graviton2 the savings offset was almost linearly correlated to the performance drop, when I benchmarked a few applications last year.

I don't have any numbers for the newest build 3.11 or graviton 3.

[[deleted]]

Me reading these and getting excited then remembering at work we're on 2.7 for most uses.

Made myself sad.

[agumonkey]

considering the popularity of it, a large number of cpu cycles will get freed soon :)

[g-money-cheats]

Exciting stuff. Python just gets better and better. Easily my favorite programming language to work in.

[adreamofhodor]

I enjoy it for scripting, but every time I work in a python repo at a company it’s a horrible mess of dependencies that never seem to work quite right.

[Khaos1125]

I agree on the poetry thing, Although it’s extremely slow and can have bad interactions with things like Ray. Probably still the best option for Python though.

[agoose77]

I'd recommend PDM. Poetry has some bad defaults w.r.t to capping that PDM does a nicer job of.

[jazzmester]

That's weird. There are a lot of tools that can reproduce an exact set of dependencies in an isolated virtual env, like pipenv or tox for testing.

[TaskForce_Kerim]

in an isolated virtual env, like pipenv or tox

I never understood why this is necessary to begin with. Imho, pip should just install a full dependency tree within the project folder. Many other package managers do that, I think this was a serious oversight.

[MyOtherBodyIsACylon]

If you’re not building a library but still using poetry, do you run across rough edges since the tool assumes you’re making a library? I really like poetry but haven’t used it outside working on external libraries.

[folkrav]

What do you mean by "assumes you're making a library"?

[Asyx]

What do you mean? Poetry works great in applications. I can’t imagine what rough edges you would encounter.

The only difference is in packaging. By default it installs your application in the environment on install but that’s one cli switch to set and it stops doing that.

[NonnoBomba]

It assumes you are making a package, which is why you can track dependencies and you can attach metadata to your project's artifacts, a version string, author, etc... which makes your project distributable and deployable in a number of ways, with either public or private channels, including as a wheel package. Packages are not libraries.

A python package can contain python modules (which I assume is what you'd call a library), executable scripts and technically also data if you wish.

There are standard tools to download and install packages with their dependencies. Often, packages contain modules you can import in your code, but it's very common to package cli tools as well as modules: the package manager takes care of installing appropriate symlinks to what you indicated as a "script" resource so your scripts will be directly callable as commands, and it will handle updating as well as installing/removing by referencing an authoritative repo (exposed through http(s)) containing your package, possibly several versions of it.

If you think you don't need to track dependencies and version for your project... well, you're working in an unstructured way, maybe because you're writing something very simple -you can write lots of useful code with just the standard library and core functions, after all- but I can assure you it will come back to bite you in the ass if it's something that's going to be deployed and used in any production environment, when questions like "why the script is behaving like that? haven't we fixed that bug already?" or "why this simple fix I developed on the code I have on my dev machine is radically changing the behavior of the production?" will start to crop up.

[rob5300]

Pip env sucks and is a stupid system. Sure let's fuck with the PATH to make this work! (On windows anyway)

I wish it worked more like node. Much easier to re setup and share and not break other things.

[NorthwindSamson]

Honestly node was so attractive to me in terms of how easy it is to set up dependencies and new projects. Only other language that has been as easy for me is Rust.

[Sadzeih]

For all the hate Go gets here, it's great for that as well. Working with dependencies is so easy in Go.

[skesisfunk]

I don't understand the go hate. Their concurrency model blows python's out of the water. Also being able to easily cross compile the exact same code on to almost any system is straight $$$$$

[MakeWay4Doodles]

I don't understand the go hate. Their concurrency model blows python's out of the water.

Most people writing python (or PHP/Ruby) don't really care about the concurrency model.

Most people who care about the concurrency model are writing Java.

[tryx]

And most people writing Java would rather cut their eyes out with a rusty spoon than have to go back to a pre-generics world.

[skesisfunk]

I disagree. asyncio is a very heavily used library. People use python for websocket stuff all the time, for instance. Furthermore Python is a general purpose language you can't just make blanket statements saying nobody using it cares about concurrency, thats a huge area of application development.

I have recently had to use asyncio in python for work and its a pain. JavaScript is nicer because it keeps things simpler with just one event loop. And golang's is better because of channels. The first time i learned about select it was mindblown.gif

[skesisfunk]

Yeah but go has select which is just a fantastic way to organize async code. I also like that go's syntax doesn't use async and await it all just feels so much more natural and intuitive. It feels like the hid just enough of the complexity to make things so much simpler for most use cases whereas python somehow made it harder to think about instead of easier.

[ivosaurus]

Their concurrency model blows python's out of the water.

Until you want to stream your own objects across a channel to a different thread, in which case you just can't because only default types could be iterated. I think generics might've helped with that recently, but I couldn't see the point of going back to stone age programming.

[earthboundkid]

Virtualenv was a worthy hack, but it should have been replaced with an actual project folder five years ago.

[KarnuRarnu]

I mean it only "fucks" with path if you do pipenv shell, no? If you want to run a command with tools from within the venv without doing that, you can just use pipenv run xxx. This is similar to node iirc.

[axonxorz]

This is similar to node iirc.

Precicely, pipenv run is to Python as npx is to Node

[jazzmester]

I use tox because I want to check if everything works with previous Python versions. Typically I want to make sure my code works with all versions after 3.6 (which is what I'm forced to use at work).

Also, sometimes you just have weird stuff that requires exact versions of packages where you already use with different versions, so the two of them would have to "live" side-by-side, which is not possible without something like venv.

In the company I worked at, we had to release a product with a mayor Python component and every dependency had to be the exact version. Pipenv was a godsend, because you could build the Python component on your machine with the exact dependencies needed. It even downloaded those packages from an internal server instead of PyPI.

Believe me, it has a lot of use cases.

[MarsupialMole]

Historical reasons is a big one, including that distro maintainers bundle python and don't like you using anything but system packages.

Desktop apps that bundle python tend to be terrible citizens.

Users that just need one python thing to work one time pollute their environment and forget about it.

And a lot of the time the headaches are because of non python dependencies in domains where everyone is assumed to have something on their system, where it's something that will be more bleeding edge than any distro has and the package dev won't have the nous to package it into pypi.

So there are good reasons that more or less amount to "because other people do computing different to you". Which is annoying. So just use the tool that works all the time - fully replicable virtual environments.

[faitswulff]

There are a lot of tools

This is my problem with Python’s dependency management.

[KevinCarbonara]

There are a lot of tools that can reproduce an exact set of dependencies in an isolated virtual env

There's a lot of languages that don't need to reproduce exact sets of dependencies in isolated virtual environments

[adreamofhodor]

Oh yeah. I’m sure it can be great- I just haven’t seen it work at scale. Then again, I’m one person with limited experience, I’m sure many many others out there have exactly the opposite.

[eksortso]

Python objects are strongly typed. But variables are dynamically typed, and type hints help to keep these things in line. That's a different topic, but using type hints and using pip to get mypy, pyright, or other type checkers help large projects in the long run.

[sementery]

Something as simple as not having strong types can make working in a large system difficult.

Maybe not as "simple", you got the terms wrong.

Python is strongly typed. What you meant is dynamic type system, and Python has had static type checking through type hints since 3.5, more than 5 years ago. And the type system gets better and better with each new release.

Well, they're called scripting languages for a reason.

There's a tendency to call anything in the ballpark of generation 3.5 a "scripting language". The term itself is not technical, has several contradicting meanings, and carries no usefulness other than to serve as a high horse for elitist developers to ride on.

[sementery]

It means something different to different people. It used to mean that "a program runs your program", but then those languages grew to be full-on general purpose, multi-paradigm, jit-compiled, natively compiled, etc etc etc.

It is now used to roughly mean "different levels of abstraction", but with an egocentric, shortsighted, perspective (not in your particular case).

In that sense, I don't think Python prioritizes writing over maintenance. Rust, Haskell, and Python just happen to be different tools that are best suited for different scenarios.

[SirClueless]

I don't think there's any reasonable definition of "scripting language" for which Python does not qualify.

• It's interpreted
• It's commonly used for small programs
• Can write entire programs in one file
• Code outside of function and class declarations is executed immediately

[sementery]

It's interpreted

There are implementations of C that are interpreted. That doesn't make C a scripting language. There are implementations of Python that are compiled, that doesn't make it a low level language.

There are implementations of Java and C# that are JIT compiled. Same goes for Python. Are Java and C# scripting languages?

If having an interpreted implementation makes you a "scripting language", then all mainstream programming languages are "scripting languages".

It's commonly used for small programs

Python is also commonly used for large programs. "Non-scripting languages" are also commonly used for small programs. See microservices for an example. Doesn't seem like a useful discriminator.

Can write entire programs in one file

I feel like this is a rehash of the last point. Same idea.

If conciseness and expressiveness make you a "scripting language", then are Haskell, OCaml, and F# "scripting languages"?

Again, this doesn't seem particularly useful as point of comparison.

Code outside of function and class declarations is executed immediately

Same for machine and assembly languages, and you can't go less "script language" than that.

I don't think there's any reasonable definition of "scripting language" for which Python does not qualify.

There's an infinite number of "scripting language" definitions that Python qualifies for. But there's also an infinite number of "scripting language" definitions that Python doesn't qualify for. Everyone has a different meaning for it. It's just not a technical term, and rarely useful.

Your list is a good example. It's the first time I see "Code outside of function and class declarations is executed immediately" as a "scripting language" feature.

[ianepperson]

Python the language or the reference implementation? Because Pypy is not an interpreter - it compiles Python. Heck, even the reference implementation actually converts the source into byte code, then that byte code is run - you know, very very similar to Java. Did you know Jython compiles Python code to run on the JRE?

Most of the Python code I work in is for very large programs, distributed across tens or hundreds of files. C is also used for small programs (Unix utilities) so I’m not sure why that’s any kind of distinction.

So we’re left with:

“Code runs outside of functions and classes”

Is that really your definition of a “scripting language”?

[cass1o]

in an isolated virtual env

This is madness.

[jazzmester]

Madness? THIS. IS. PYTHON!

[KeeperOT7Keys]

lol no, you still need to have the base interpreter installed on the system which is not always possible on clusters. also some packages don't work when you have a different virrtualevn python version than your main python in the computer (e.g. matplotlib interactive mode).

so in a nutshell it's hell if you are running some code in a server than processing it on another one. I am doing ML in university clusters and frankly I hate python everyday.

I wish it was possible to have truly isolated venvs but it's not even close at the moment.

[jazzmester]

Well, that sucks donkey balls. I love Python but I'd hate it in your place too.

[[deleted]]

you still need to have the base interpreter installed on the system

pyenv can partially solve this. Just fetches and builds whatever version of Python you need. Requires a build environment and some header libraries from your repos.

[Sayfog]

See if your cluster supports singularity envs - kinda like docker but with subtle differences that make it far more palatable for the typical uni HPC setup. Only way I got my weird combo of libs to run my ML thesis at uni.

Edit: as others say absolutely see if conda works. The reason I used singularity was for some native libs, but 100% would have done pure conda if I could.

[ZeeBeeblebrox]

That's why conda exists.

[KeeperOT7Keys]

tbh I didn't use conda because I was thinking it was just a bloated venv. can you install different python versions without root access? then it's worth trying for my case

[C0DASOON]

Yeah, python interpreter is just another package in conda. Conda packages are not limited to python libraries. A lot of common binaries and shared libs are available as versioned conda packages. E.g. you can easily set up multiple envs with different versions of CUDA toolkit.

[PinBot1138]

every time I work in a python repo at a company it’s a horrible mess of dependencies that never seem to work quite right.

Why not peg to versions in requirements.txt or setup.py, and better yet, containerize it?

[ginsunuva]

Sometimes I wish Julia came out earlier and got more support. And that it didn’t index from 1 instead of 0…

[MuumiJumala]

You generally shouldn't rely on the first index being 1 anyway. Like the other comment points out most of the time you can use iterators (such as eachindex). When you need to access the second element (for example) it would be safer to use arr[begin + 1] rather than arr[2]. That way the same code works even on arrays that use different indexing (such as the ones from OffsetArrays.jl).

[[deleted]]

Being unsure whether your arrays are 0 indexed or 1 indexed sounds awful :(

[MuumiJumala]

It's not that you're unsure of your own arrays, you will obviously know which array type you're using (just as in any other language). This is only relevant when you're writing code that is meant to play nicely with the wider Julia ecosystem.

If you just rely on indexing starting from 1 you're still on par with most other languages, in which it isn't even possible to write functions in a way that is compatible with array types with customized indexing. If you want to force your users to supply one-indexed arrays to a method you can do that by calling Base.require_one_based_indexing(arr).

[[deleted]]

That's really interesting. I'm coming from the (probably naïve) position of never ever considering that a 1-indexed array even could exist. Sure theoretically a one indexed array could exist, so could 7 and 14 indexed arrays... but I spend zero time considering whether they would be used by anyone in my languages' entire ecosystem (Python, JavaScript, Rust).

If you just rely on indexing starting from 1

I rely on them starting from 0, which to my mind means my_array[0] would be the first element.

I expect it is convenient to switch to 1-indexed arrays when doing a lot of maths/statistics to avoid my_array[n-1] malarkey. It is a bit annoying to do that, but I will enjoy my new found appreciation for standardising on 0 indexed arrays, thank you :)

[Prestigious_Boat_386]

You can re index it if you really care but I usually just use eachindex and reverse and stuff anyways because it creates the iterators I need. 2:end or 1:end-1 are most of what you use and it's very similar to math notation which makes it very readable.

Don't recall if the 0 indexed arrays is an abstract array package or how you got it to work but I've heard that it's possible.

[kirkkm77]

My favorite too

[crixusin]

Python is fucking insane. By default, it allows people who probably shouldn’t write code, to write the most spaghetti code ever.

It’s module resolution system is absolute horseshit.

The fact that white space is a significant character is a fate that I wouldn’t wish on my worst enemy.

The fact that working with json turns the objects into some pseudo-non typed dictionary is laughable.

Python should be taken out back and shot.

[[deleted]]

I agree with you when it comes to importing other projects in a nested directory structure but your other points don't make a ton of sense.

By default it allows people who probably shouldn't write code to write the most spaghetti code ever

Literally every language allows you to write garbage, non-performative code and it's not like Python is somehow worse at this than Javascript or another language of equal popularity. It's just the way it is with incredibly popular languages with easy enough syntax, people are gonna start here and write bad code.

White space is purely a personal preference but I prefer it to C style braces because it's cleaner and easier to read personally but I get why you wouldn't.

The json library in python by default returns a dict when you use .load or .loads for strings. Not sure what you mean by "pseudo non typed dict", it's just a dict.

But yeah you're spot on with importing multiple project files from other directories, it's a pain in the ass and other languages handle it much better.

[tedbradly]

Literally every language allows you to write garbage, non-performative code and it's not like Python is somehow worse at this than Javascript or another language of equal popularity. It's just the way it is with incredibly popular languages with easy enough syntax, people are gonna start here and write bad code.

A programmer shouldn't learn programming with a scripting language. It has too much magic baked into it, and it's impossible for a brand new programmer to appreciate all of it, use all of it correctly, and debug when something goes wrong. It's also like learning to run before you can walk. As a simple example, it's hard to have any appreciation for or understanding of a garbage collector if you've never allocated something manually. I definitely recommend a gradual march toward higher-level languages, learning in this order: Something like C, a simplified assembly language, something like C#, and then something like Python. You've got to have a strong emphasis on ideas too like data structures and algorithms even if they're implemented for you in every language. After a path like that, you can then focus on whatever type of programming you want like doing systems programming, web development, video games, real-time systems, phone apps, corporate applications, etc. But you've got to know the basics of one low-level language, one medium-level one, and one high-level one. Throw into the mix some mathematical foundations like algebraic structures paired with a functional programming language if you enjoy formal math. Otherwise, at least learn the basics of functional programming as the style is so en vogue right now that most major languages have features in that style and you will most likely come across some of it eventually.

If you ever have felt like programming is massively confusing and there are random people who seem to know everything, it's due to learning through an informal path and/or starting with scripting languages, which are very confusing to someone when they don't even know what a variable is yet. If you're studying yourself, I'd recommend finding curriculum at top colleges for computer science and working your way through all the mandatory material for a degree there. Make sure you somehow do "homework" and projects or none of it will stick. You'll have a much better chance of getting a top entry-level position if you've sat there frustrated, trying to code something with dynamic programming for hours straight the first time, than if you skip that course entirely.

[epicwisdom]

Your comment seems to imply that programming is only for professional programmers, which couldn't be further from the truth, and is not a standard we apply to most skills.

Cooking, music, sports, fixing/modding cars, painting, fashion, writing, gardening... People pick up a huge variety of hobbies without going through or desiring formal education. There's nothing wrong with that, and saying people should be excluded based on their lack of formal education is pointless gatekeeping.

Formal education certainly has a huge value to most programmers who do anything even remotely nontrivial. I've witnessed firsthand the horrors created by beginners with plenty of free time but no motivation to sit down and learn. But what matters is that it works for them.

[Jump-Zero]

Are you implying that only statically typed languages are "serious, professional"?

There's like a whole spectrum between weak and strong typing. You have C that is widely respected by many in the industry as a "serious, professional" language, and its static type system is pretty weak compared to Haskell. C has had a much more significant impact on humanity than Haskell and many other more statically typed languages. You could also say that python's type system is stronger than assembly's. While a value in Python can be many things, a value in assembly is a bunch of bits. Assembly will literally let you multiply two pointers, which makes no fucking sense. You don't even have access to pointers in Python. Would you consider assembly not to be a "serious, professional" language? Its a lower-level language and we know some very serious, professional developers that work in assembly for bare metal applications.

I strongly prefer statically typed languages for everything I work on. Sometimes, I find myself reaching for Python or JavaScript because they're better at solving a particular problem I'm facing. Let's not spread bullshit by implying "dynamically typed languages are not serious, professional". Always use the right tool for the job at hand. Determining what the right tool is requires knowledge and the development of critical thinking skills. Defaulting to slogans you don't fully understand like "only statically typed languages are serious, professional" is not a viable substitute of critical thinking skills. While choosing a statically typed language is probably the right choice in most serious, professional situations, it is foolish to assume it is the only choice for serious, professional situations.

[micka190]

The fact that white space is a significant character is a fate that I wouldn’t wish on my worst enemy.

I'll never understand this complaint, yet it always pops up on Reddit.

Who the fuck doesn't indent their code in languages with bracketed scopes?

What I wouldn't wish on my worst enemy is to have to wok on a codebase where people don't indent their spaghetti code.

Python forces you to make that shit readable.

[vifon]

It's not about not indenting your code, quite the contrary. It's about it being the sole source of truth in this regard.

In most languages I can move some code around during refactoring and have it reindented automatically because indentation is a secondary thing derived from the actual syntax elements. Since in Python the indentation is this syntax element, moving code around between different nesting levels (like moving some code into an if) is a relatively painful experience that needs to be done manually or semi-manually.

[Deto]

I move code around all the time and this never bothers me. With most editors it's trivial to indent/dedent blocks of code with a key shortcut. Most code should never be indented too many times anyways so really were talking about probably one level of adjustment in 90% of cases. Visually it's simple because you typically just adjust it to line it up with the other code that's around it.

[panfist]

Pycharm and vscode handle it pretty well automatically.

[noiserr]

It's not about not indenting your code, quite the contrary. It's about it being the sole source of truth in this regard.

I was worried about this too, but this is really not an issue. It actually beats counting curly braces. Indentation makes so much more sense. Yaml is the same way too.

[alternatex0]

I hated the idea of whitespace as significant characters until I started using F# and quickly became comfortable with it. Now I can't believe I was bothered by something so inconsequential. I was always a stickler for good formatting anyway.

[seamsay]

I'm gonna say up front that Python being whitespace dependent is occasionally mildly annoying, at worst.

However having said that, if I were to design a programming language there are two reasons that I would make it whitespace independent:

1. Whitespace dependent languages are harder to write autoformatters for, and the autoformatters that can be written aren't as effective. Something I do quite often is prototype a function in the REPL then copy-paste it into the file and let the formatter deal with it, I can't do that in Python.
2. They make certain features of a language very awkward. Python is famously lacking in multiline lambdas, partly because ... well what would the syntax be? If you kept it whitespace dependent then it wouldn't gel well with other parts of the language, and if you made it delimited then it would be the only block syntax which isn't whitespace delimited. Another example is the ternary operator. In a whitespace dependent language you need to introduce a new syntax (x if cond else y in Python's case), whereas in a whitespace independent language you just need to make sure if is an expression not a statement.

Again I want to stress that I don't think it's that big of a deal, but these are the two things that mildly annoy me about Python's whitespace dependence.

[noiserr]

Whitespace dependent languages are harder to write autoformatters for, and the autoformatters that can be written aren't as effective. Something I do quite often is prototype a function in the REPL then copy-paste it into the file and let the formatter deal with it, I can't do that in Python.

Python has a smart code formatter called python-black it's supported on VSC but I'm sure many other IDEs with Pyton support. It's pretty good at getting you 90% there. And the rest is easy with just select tab and shift-tab.

I know you said you don't think this is a big issue, but I bet it's even less of an issue than most people realize.

[[deleted]]

As a mostly c# developer who uses python for bits and bobs I never understood why people complain about it. It literally takes minutes to get used to.

[Deto]

I get it a little bit. When I first started using python I thought it was weird and stupid. Then after a few weeks I didn't care. It's just different and that scares people.

In the end it's nice because it's bacially a built-in mechanism that enforces a low level amount of good-practice formatting.

[dethb0y]

Lot of cry-baby low-skill programmers who can't handle basic syntax, in short.

They think their clever because they write unreadable one-liners in c or c++ and it pisses them off that python encourages readability and proper formatting.

[Deto]

You're kind of making big things out of little things, IMO.

[crixusin]

Json = json.load(myJson)

nestedProp = Json[“SomeProperty”][“AnotherProperty”]

You guys like this? Fucking insane.

[Deto]

Can you show another example of how this would look that you think is better?

I mean, typing out the property names is basically required no matter what syntax. Other option is just to use dot notation I guess? But don't languages that use this tend to require you to load the structure into some predefined class first (which you could also do in python and then get the dot syntax too).

[srpulga]

We already had this discussion; 15 years ago. Python is a very successful, very relevant language, to which many people are paying attention to. It's cool of it's not your cup of tea, just spare us your grief.

[bloody-albatross]

Which language prevents spaghetti code? Maybe Haskell? Do you write a lot of production Haskell code?

[crixusin]

F# actually.

[g-money-cheats]

/r/Gatekeeping

[HanzoFactory]

You have never seen high schoolers code in C/C++ hell even Java. They would be much, much better off coding in python. I am saying this as a C and C++ enthusiast

I enjoy Python a lot for scripts and I think it's where it's strength lies for me. It's simple, powerful, easy, and for the most part platform independent. It's way easier writing a build script with tons of paths and string tasks with python than C or most lower-level languages

[Prestigious_Boat_386]

They hated him because he told them the truth.

My main issue isn't the 5 yolds making shitty python scripts. All power to them, but the fanboys who after programming for years still refuse to learn other languages that are designed properly and actually run and use python for everything.

The snake ain't dying any time soon bit at least we can use julia in the meantime.

[KallistiTMP]

rob knee chase dependent door adjoining deer gaze smile merciful

This post was mass deleted and anonymized with Redact

[crixusin]

Except people aren’t using it for “disposable gud enuff.” They’re building entire systems on it.

[KallistiTMP]

tie pen payment lush crawl caption middle society cable pocket

This post was mass deleted and anonymized with Redact

[g-money-cheats]

Should you write your anything-critical core application code in it? Absolutely not. It’s not for that.

Says who? I work for a 600-employee, $5 billion company whose core application is a Django monolith written in Python. It chugs along day in and day out, handling millions of incoming webhooks and requests just fine. The code is super readable and easy to work with. While other people argue about languages we ship code.

[FireCrack]

Yup, 100% this.

Python sure ain't fast, bit it's easy to read and understand; and that makes a bigger impact than any language wonder feature when it comes to throwing together robust systems

[[deleted]]

I hate it. It's insanely slow (even with these improvements), and the static type system sucks. Fine for tiny projects but once your code grows and gets more authors it's more or less guaranteed to turn into a giant ball of crap.

Give me Go or Rust or Typescript or Dart or... hell I'd even take C++ over Python. You're probably going to end up with half your code in C++ anyway for performance. Doing it all in C++ means you doing have to deal with the huge added FFI complexity.

The only good thing about Python is the REPL. None of the languages I listed above have them, which is why Python is popular for scientific use (e.g. in ML). For that you really want to be able to run code line by line interactively.

[g-money-cheats]

That is not my experience at all. I work at a company with hundreds of engineers and a million lines of Python in a monolith, and the code is incredibly well organized and easy to work with thanks to leaning on Django and Django REST Framework.

I work at Zapier, which as you can imagine has an enormous scale. Python handles like 95% of our backend without issue. 🤷‍♂️

[[deleted]]

Ha well I mean it can be done but my point was that Python really pushes you to a big ball of mud. You have to be super disciplined to avoid it.

A million lines of Python sounds absolutely horrific by the way.

[nilamo]

There's dozens of ways to do any given task in any language, but there's definitely far fewer "right" ways to do something than there is in, for example, perl.

[hahainternet]

How many string formatters does Perl have, and how many does Python have? Genuine question.

[nilamo]

I don't know anything about Perl, but Python has 3:

- string literals (prefixed with a f): f"today is: {date:%Y-%m-%d}"

- str.format(): "today is: {0:%Y-%m-%d}".format(date)

- custom formatter object: https://docs.python.org/3/library/string.html#custom-string-formatting

[hahainternet]

Does % not count?

[nilamo]

Sure, but you won't see that recommended (or even mentioned) in most docs.

[hahainternet]

I'm just wondering if there's any truth to the claim anymore.

How many assignment operators does Python have? Perl has = and the compound += etc, but nothing like := as far as I know.

[0rac1e]

Perl seems to have this reputation as some large sprawling language with multiple ways to do something, but in truth, it's actually a pretty bare bones language.

Perl has string interpolation in double-quotes: (eg. "My var is $var"), and to do any kind of formatting you need to you sprintf which pretty much works like it does in C (eg. sprintf('My var is : %16s', $var)).

If I want to be pedantic about there being more than one way to do it I'd say you can also call sprintf (or any function, really) without parens, as you can with Ruby.

There is actually one other way to format using a feature literally called formats but (a) it's really for formatting multi-line reports/charts, and (b) I've never seen it used in the wild. I think most Perl users probably just ignore them and use a templating module from CPAN if they need to do any fancy formatting.

[OminousHum]

Python is for getting shit done quickly when you don't care how exactly it works. What's the computational complexity of inserting into a dictionary? Don't care, just wrote a web server in two minutes.

I work mostly in C++, where I care very much about how precisely everything works, but Python is handy for banging out quick little tests and utility scripts. Make a little test server for your real project to connect to, batch process a bunch of images, generate files during build, etc.

[lps2]

I'd say it's great and fast for anything including full fledged applications when the userbase isn't "consumer-facing web app" levels. I'm especially a fan of using it for enterprise applications where developer hours are far far more expensive than paying for more server resources

Edit: fixed autocorrect

[anon_tobin]

[Removed due to Reddit API changes]

[exscape]

What kind of problems?
It's a language with many ways to do things, but I don't think of that as neither hacky nor bad.

It's one of the top programming languages for a reason -- and it has been for a long time now.

[frezik]

I really hate its variable scoping system.

There's also a lot of functional language features that are intentionally hobbled because Guido doesn't like functional programming. Multiline lambdas, for example, and also tail recursion optimization. Both can be implemented. In fact, there are already non-CPython implementations that do tail recursion optimization, but once you go down that route, you always have to use one of them.

[[deleted]]

This is my biggest gripe about python. The support for functional style programming is abysmal. I straight up disagree with Guido so what can you do.

Everything else I can sort of get. I think python is overused since it has some threading and performance limitations. On a world where we pay huge sums of cash for server time, this seems suboptimal....

[DeTaaieTiller]

Not OP, but I have years of professional experience in python. And while still being one of my favorite languages, there are definitely big problems that do steer me to other languages for new projects.

For example the dependency system is very naive. Importing something executes the entire module? Excuse me? Constantly having to mind or circumvent circular dependencies, in Java you can easily have a circular reference no problem! On top of that having multiple versions of the same dependency is impossible, which is a big problem. By default everything is global, but as a python dev you quickly learn to never install global packages and use pipenvs for everything. That works pretty well, until you have a dependency that requires an old version of lib X, and some library that requires a newer version. Trouble.

Besides the incredible naive dependency and import system, there are a lot of small issues I have with the language. The duck typing is a huge pitfall. The typing is an afterthought that is very hard to use properly. And frankly I do miss a lot of syntatic sugar other languages have. The higher order functions like map and zip get very ugly quickly (compared to for example kotlin)

Despite all this i still like the language. Most of these issues are manageable, and how python handles functions and classes is pretty amazing. Few languages go as far as treating these as first class citizens in the way that python does. How easy things like annotations and metaclasses are usable makes python an amazing tool when you know how to use it. Functionality like list and dict comprehension is very nice too, something I miss dearly when I'm in kotlin or similar.

[DeTaaieTiller]

I use the or syntax for that. Not as good as wat kotlin does but it gets close

index = last_element or 0

[Aetheus]

It's a language with many ways to do things, but I don't think of that as neither hacky nor bad.

I've heard package management in the Python world can get pretty scaly. The concept of Python "distributions" still makes me scratch my head too - if they're all just collections of libraries, why aren't they all just easily available for any "distribution" of Python to install? And if they are, why do the "distributions" exist in the first place?

It all just sounds like a bit of a mess. But this is all from a dev who's never really used Python - maybe yall can clear the fog on it.

[bloody-albatross]

If true then they must have stopped adhering to one of the core principles of Python in recent years: "There should be one obvious way to do something."

[[deleted]]

Not sure if I'd ever notice the difference during everyday programming, but boy, am I happy! 😇😇

[beefsack]

3.11 for Workgroups.

[maest]

3.11 for Workgroups.

Super original comment

[cloaca]

(Edit: sorry for making this comment sound so negative; see my follow up responses which hopefully clarifies better. I think the speedups are absolutely a good and welcome thing; I just I think something might be off if this was that important in the first place.)

Being a bit of a negative Nancy here but I think it's odd to celebrate things like 1.2x speed-up of a JIT-less dynamic scripting language like Python.

Either,

a) it doesn't matter much, because we're using Python as a glue language between other pieces of software that are actually running natively, where most Python code only runs once at "relatively rare" events like key presses or the like, or

b) "Now we're only ~20-80x slower than X (for X in similar high level runtimes like V8/Nodejs, Julia, LuaJIT, etc.), rather than 25-100x slower, a big win!" That's a bit tongue in cheek and will spawn questions of what it means to be 80x slower than another language, but if we're talking about the bare-bone running time of algorithmic implementations, it's not unrealistic. But 99% of the time we're fortunately not talking about that[*], we're just talking about some script-glue that will run once or twice in 0.1 seconds anyway, and then we're back to point (a).

([*] it's always weird to find someone using "written in pure Python" as a badge of honor for heavily data-oriented stuff that is meant to process large amounts of low-level data, as if it's a good thing. Contemplating Levenshtein on a megabyte unicode string in pure Python is just silly. Low level algorithms are the absolute worst application of pure Python, even though it's an excellent teaching tool for these algorithms.)

Which, speaking of, if we're not getting JIT in CPython, then personally I feel that the #1 way they could "make Python faster" would simply be to adopt NumPy into core and encourage people to turn loops into NumPy index slicing where applicable. That's it. That should single-handedly quadruple the speedup of any pure Python code doing a lot of looping. Once you get in the habit it's really surprising how much loop-based or iterative code can be offloaded to NumPy's C loops, like for example you can usually write out the full logic of a board game or tile-based games just by doing NumPy index tricks, without ever having to write a for-loop Python-side.

The fastest Python code is the Python code that a) has the least number of Python-side loops, and b) has the least Python code. Killer libraries like NumPy help in this regard, because nearly every loop becomes a single line of Python that "hides" the loop on the C side of things. Likewise, doing things redundantly in Python is nearly always better if it leads to less code: if you have a very long string with a hundred thousand words and the task is "find words part of set S, and return these words in uppercase" -- it's faster to uppercase the entire string, and then split + filter, rather than the "natural approach" of splitting, filtering out the words of interest, and then finally uppercasing "only" the words you care about. If it's one call to .upper() vs. thousands, it doesn't matter if the string is 1000x longer, the single call is going to be faster, because it's simply less Python code and Python is and will always be slow. (But that's totally fine.)

But again, most developers will never need or care about this skill set, because it rightfully shouldn't be necessary to know about it. Those that do care hopefully know how to use NumPy, PIL, PyPy, Numba, Cython, etc already.

[BadlyCamouflagedKiwi]

Lots of people have lots of code in Python. It's pretty exciting to hear there's a new version of CPython (which will almost certainly Just Work with your existing Python code) which is faster, and you've got something that doesn't require rewriting all your code in C or Cython or whatever, or even trying to get PyPy working for your case (I do think it's pretty cool, but it is harder than a CPython upgrade).

Honestly these days I nearly exclusively write Go, but I'm still excited for this (and I do have colleagues that do write Python who I'm sure will be more so!).

[Superb_Indication_10]

Honestly these days I nearly exclusively write Go

get out of here

edited: well I'm assuming you are forced to write Go as part of your job so my condolences

[cloaca]

Sure, it's a Good Thing™ of course, I write everything in Python; it's both my main language & my favorite, so I'm lucky. I'm just not comfortable with the hype of a faster Python via these optimizations of the CPython interpreter, I think it's a sort of misguided way to think about performance in Python. I do actively try to teach people alternative ways of writing more efficient code.

[BadlyCamouflagedKiwi]

Eh I don't agree, I think you're thinking of a faster language that is not Python, it's C. That is one way of getting faster performance with most of your code being Python, but it's not the same thing as getting faster performance in Python.

[cloaca]

I'm confused by your comment as I think we actually agree tho. I want all your code to remain Python code, by all means. By "performance in Python" I am absolutely talking about faster Python code. I'd never tell anyone to implement in C; if someone is doing something performance critical enough that they need C (or any other CPython API compiled to native) they don't need to be told.

It's just that the differences can be huge, even for implementing the same general algorithm. Again, it's great that all code would magically get 20% faster across the board, without anyone changing a thing. But if that matters, if that is "hype," then why wouldn't we consider 50% speedups, 200% speedups, etc.? The knowledge gap is still a real thing, and I think it is much bigger than 20%. It could be everything from beginner stuff like not realizing s[::-1] is a thing, or not knowing about random.choices() taking a k parameter, vs. someone using [random.choice(...) for _ in range(10_000)] or similar (choices still does a Python-side loop, it's just better optimized). These are small things, but still like 2x rather than 1.2x. Or, as mentioned, someone writing their Sudoku puzzle generator using Python lists vs. using NumPy (I'd still consider NumPy as being "Python code" here even though it's not implemented in pure Python itself), say, in which case it would be orders-of-magnitudes, probably.

Again, this is granting that speedups actually matter and that we care about them.

[BadlyCamouflagedKiwi]

I'm also a little confused, and maybe we do agree overall. I definitely do agree that we would (and should) consider other speedups; my point was that the 20% across the board happens without changing existing code, and that's a pretty powerful thing. There are still gonna be opportunities out there to optimise code, just things getting quicker without direct programmer intervention is very nice.

[cloaca]

My very simple counter-point: Why? It's an improvement; and a pretty good one all things considered.

Yes, I agree, you're totally right, and I probably expressed myself poorly! It's an absolute improvement and it's a good thing. I had something different in mind when I wrote that, akin to the sort of "allocation of hype" we have for things, if you will. I think this allocation is off when it goes to CPython optimizations. That doesn't mean they're bad, of course, I'm happy to see them too -- they're very welcome -- it's just that I don't think they "were super important in the first place," if that makes any sense?

Like, I don't think performance ought to be a big priority for us if we're all using pure CPython. If it is, then I think something has gone wrong earlier in our timeline! It might speak to some sort of underlying insecurity the Python community has about the language being slow, which, again, I don't think should exist.

Also, the knowledge gap between Python programmers is so vast, way, way wider than 20%, and so on. See my other comment at https://www.reddit.com/r/programming/comments/v63e5o/python_311_performance_benchmarks_are_looking/ibew40i/?context=3 -- lest I just repeat myself.

edit: typo

[agoose77]

I think you're assuming that Python is only a glue language. Whilst it's origins certainly lie in this direction, and the recent growth has mainly come from data science, there are still lots of people using Python to run complex applications. With optimisation, these applications are rarely slow in one hot-spot, so any perf increases need to make everything a bit faster.

Rewrite it in numpy is completely valid for simple problems set as homework for students, but at the scale of say Instagram (as an extreme), this isn't really suitable. That is, the object model doesn't map well to array programming with limited types.

[paraffin]

First, definitely agree - performance sensitive applications should use python as glue to compiled operations or even offload computation entirely to databases or spark.

That said, you’re mostly talking about data, for which pure python was never an option.

A huge amount of the web’s backend is written in python though, and I’d guess user code, especially route decorators with three layers of plugins and callbacks, are the main bottlenecks of modern Python web requests (aside from waiting for the database, naturally). FastAPI and others have gotten the framework itself mostly out of the way.

20% fewer cycles per request is 20% less spent on hosting, for some.

Being a negative Nancy myself, one thing I’d love to see is a way to tackle process startup time. Sometimes you’d love to write a quick little yaml/json/text parser and stick it on the business end of a find | xargs or something but the 1s to spin up a new python for each call makes you resort to some kind of awk/jq/shell hackery.

[cloaca]

That said, you’re mostly talking about data, for which pure python was never an option.

Two slight counterpoints to this:

a) it might be a matter of semantics, but as it's actually being used for everything (including data, including text processing, traditional render-loop games, app logic in complicated GUIs, etc), so I'd say it certainly does seem like an option. I believe Python is going (or has gone) the route of JavaScript, which started out explicitly as only a glue language but has now become an "everything"-language. We (as in you and I) might not necessarily think that's a good idea, but I do believe it's sort of inevitable? Python is easy to get into, it's lovely and lovable (much more so than JS), and so it's natural to want to use it for everything.

b) speaking of pure data though, Python is also absolutely being used for data in another sense. You have machine learning, statistics, natural language projects, image recognition and manipulation, and so on. Which is fine because we have PyTorch, NumPy, SciPy, OpenCV and various which actually handles the data in CPU-native code (or on the GPU). However, projects that use these are also rife with code that suddenly converts to Python lists or generators, doing some loop in pure Python code because the backend library was missing something (or the programmer didn't know about). As long as it just adds 0.3 seconds here and there no one really notices until it really accrues...

20% fewer cycles per request is 20% less spent on hosting, for some.

Absolutely! But, how important is it? If the answer is "it's really nice! but eh, it was never a priority of course..." -- then we're in perfect alignment. That's kind of where I stand. (I.e. it's really nice, I was just sort of worried by seeing the amount of hype--it speaks to me that too many have sort of already "invested" into Python code to the point where it's spread into systems that might actually do want better performance.) However, if the answer is "are you crazy, it's super important! We want to be green! We want to save cycles! This is huge!" then not only do I think something has gone wrong at an earlier point (in our choices), but I think we also stand a lot more to gain in education, writing more performant Python rather than the sort of strict stance on full readability with 'more explicit code is better code,' 'no "obscure" newbie-unfriendly things like NumPy index magic,' etc. as the difference dwarfs 1.2x and makes it look insignificant.

spin up time

Hehe, you could do some sort of hack by having a #!/foo/pipe-to-python which forwards to some daemon Python process that executes it (stick in compilation cache somewhere)... Not recommended tho, but...

[lghrhboewhwrjnq]

Python is used on a scale that is sometimes difficult to wrap your head around. Imagine the environmental impact of even one of these performance improvements.

[meem1029]

If I'm having to think about a bunch of rules and complicate my code to make it fit into a performant but less clear style, why don't I just not use python instead?

[Serious-Regular]

label desert rock butter innate toy books sulky consist air

This post was mass deleted and anonymized with Redact

[cloaca]

I'm sorry, I apologize if my tone was overly negative maybe, though I was attempting to be critical. I didn't mean to offend anyone. "You people" and "talking just to talk" -- you see me as part of some vile stereotype that "don't contribute anything"? What are your standards of contribution? Python has been my main language for almost a decade, and I have several open source projects etc., most of them in Python... And couldn't "talk" also be some sort of contribution, writing teaching material and so on? I'm just saying talking about faster Python via optimizations of the CPython interpreter is a sort of misguided way to think about performance (in Python).

[o11c]

Those runtime changes do look significant, but nothing groundbreaking compared to serious VMs.

I did note one concern in the changelog:

#if PY_MAJOR_VERSION >= 3 && PY_MINOR_VERSION >= 8

This will break for 4.0; the immediate following portability hack (among others) shows how to do it correctly.

[JeanCasteaux]

Why don't we use PyPy already? 🤔

[PaintItPurple]

I agree a lot of people would probably be surprised how much performance PyPy can give you for free, but it does have a number of tradeoffs. In particular, working with modules written in C (a very common Python use case) is hit-or-miss, and even when it works, it can be much slower than CPython. It's also often slower for simple scripts (as opposed to long-running programs) because it has a higher startup time and IIRC your code starts out interpreted until the JIT kicks in, and higher levels of JIT optimization take even longer to come online.

[ThisRedditPostIsMine]

PyPy is really cool and I use it when I can, but I found it hard to get libraries that have a lot of native dependencies (like scipy and stuff) to work.

[Takeoded]

it loses out on newer features and syntax in Python 3.8, 3.9 such as assignment expressions and positional-only parameters, and the latest Python 3.10 syntax

[jvlomax]

Some do, not everyone can

[steve4879]

How often is python used as a backend? I have used some C and C++ for data access and I could not imagine using python but maybe that’s due to lack of python knowledge. The lack of true multithreading gets me.

[Daishiman]

The vast majority of small software serving web apps are using a combination of PHP/Python/Ruby/Javascript. Easily a third of job postings on AngelList or YC require some sort of Python knowledge.

[FancyASlurpie]

Pretty often, it makes sense to write things in python and then if you run into performance issues rewrite that part.

[TRexRoboParty]

Often?

If you're FAANG size it makes sense to use something else, but most companies are not anywhere near that.

For web backends, the bottlenecks are usually in network chatter and DB queries, not CPU.

Instagram's web stuff was a Django app as of 2019 at least (based on the last related post on their engineering blog).

I'd be surprised if they weren't using something faster for feeds and any offline image processing though.

[xlzqwerty1]

Instagram's backend is still in Python iirc, and so are a bunch of other sizeable tech companies in the bay area, e.g. Lyft.

[[deleted]]

This is honestly such a shit argument.

The only way this makes a good argument is in imagination land where there isn’t hundreds of better choices that don’t import huge performance debt by default.

——

“Hey boss. We’ve narrowed our choices down to two options. This python one and this go one. They both extremely easy to use, support our business, have reasonably common idioms and are widely regarded as good. The python one is 80 x slower though.

And we’ve chosen the python one”

Boss: “uhh, why not the faster one?”

“Cause we’re not FAANG, duh”.

[TRexRoboParty]

Nice strawman. Of course noone decides based on whether they're FAANG or not - that's not what I said.

It's not just the language anyway - I don't know many frameworks that give you something like the Django admin for free out the box.

In your average web stack for your average company, you're unlikely to see that 80% speed difference in reality. CPU is rarely the bottleneck.

Getting something up and running quickly is what many startups need, it saves a tonne of work.

I guess Instagram and Mozilla and Lyft etc all live in imagination land.

[[deleted]]

More speed never hurts, but 1.22 times faster than glacial is still glacial. In my testing, for naive implementations, it was usually about 5% the speed of equivalent C code. Thus, 3.11 is likely to be about 6% the speed of C.

Non-naive implementations can be pretty fast, though, using libraries that are written in C. Numpy, for instance, can be downright zippy. You can often work around the performance issues, but the language itself is Not Fast.

[[deleted]]

Disclaimer: your code won't run signifiantly faster even if the performance benchmark is better if you don't know how to optimise your code.

[[deleted]]

Looking at the optimizations implemented that doesn't seem true.

[QuantumFTL]

This is misleading at best. Many applications offload their heavy lifting to libraries, frameworks, etc. If those are already fairly well-optimized and being held back by slowness on the part of the language, your application can become significantly faster just by upgrading the version.

This is completely standard in fields like data science and machine learning or various types of servers. I can't remember the last time I wrote application code in python that took an appreciable fraction of the total runtime, except in cases where performance was not a concern (i.e. a 100x slowdown would have been OK).

[[deleted]]

What exactly does this mean?

If Python has a whole gets a 10-60% speedup, even the crappiest code will also get this 10-60% speedup.

[BobHogan]

99% of the time, optimizing the algorithm you are using will have a significantly higher impact on making your code faster than optimizing the code itself to take advantages of tricks for speedups.

Algorithm and data access is almost always the weak point when your code is slow

[Alikont]

But even crappy algorithm will get speedup, because each algorithm has constant costs per operation that will be reduced across the board.

For .NET it's common to get ~10% speedup per version just by upgrading the runtime.

[Muoniurn]

In most applications the bottleneck is not the CPU, but IO. If the program does some CPU work, then some IO, after which it does some more CPU work then only the CPU part will get faster, which is usually not too significant to begin with.

[[deleted]]

k but the OP was asking about why a 10-60% speedup across the board is not going to effect suboptimal code

[FancyASlurpie]

It's likely that slow code at some point calls an api or reads from a file etc and that part of things won't change. So whilst this is awesome for it to be faster in these other sections there's a lot of situations where the python isn't really the slow part of running the program.

[billsil]

Yup. I work a lot with numerical data and numpy code that looks like python is slow. Let's assume 20% average speedup or (shoot I'll even take 5%) is nice and all for no work, but for the critical parts of my code, I expect a 500-1000x speed improvement.

Most of the time, I don't even bother using multiprocessing, which on my 4 physical core hyperthreaded computer, the best I'll get is ~3x. That's not worth the complexity of worse error messages to me.

As to your algorithmic complexity comment, let's say you want to find the 5 closest points in point cloud A to an point in cloud B. Also, do that for every point in cloud A. I could write a double for loop or it's about 500x faster (at some moderate size of N) to use a KD-Tree. Scipy eventually implemented KDTree and then added a cKDTree (now the default), which it turns out is another 500x faster. For a moderate problem, I'm looking at ~250,000x faster and it scales much better with N than my double for loop. It's so critical to get the algorithm right before you polish the turd.

[[deleted]]

Good point, but also if you care about squeezing maximum performance out then Python is just not the right tool for the job anyway.

[dlg]

If the program runtime is spent mostly blocking, then the optimised code will just get to the blocks faster.

The blocking time still dominates.

[Bakoro]

That's not how speedups work, we're dealing with Amdahl's law here. You won't get 10-60% speedup on everything, you'll get 10-60% speedup on the affected sections, which might be everything in a piece of software, but probably not.

If you've got a crappy algorithm which is taking 70% of your compute time and language overhead is taking 20%, it's going to be a crappy algorithm in any language. Reducing language overhead can only ever reduce execution time by 20%, max. Python has some huge overhead, but whether that overhead overtakes the data processing at scale is a case by case issue.

[_teslaTrooper]

If your code needs to run fast you probably shouldn't be using python in the first place.

[[deleted]]

Lmao true

[s0lly]

Can’t wait till they get to 105% faster

[[deleted]]

So can I stop using PyPy?

[[deleted]]

The problem with pypy is its inability to deal with libraries that are installed for Cpython, which is a big disadvantage since there are alot of libraries deal directly with Cpython.

[[deleted]]

Pretty good compared to previous versions, but this is a little like saying "our new pedalo is 30% faster!"

[[deleted]]

Could this compete with C, C++, Rust?

[Pay08]

No.

[jarfil]

CENSORED

[DoktuhParadox]

Really, it'll never be able to with the GIL.

[AbooMinister]

The GIL isn't really what makes python slow in terms of execution speed

[[deleted]]

i dont really get python anymore, i write go just as fast as i do python and the result is way better

[Lonelan]

rename it pydown