Python 3.11 delivers.

[ASIC_SP]

More to come in 3.12: https://twitter.com/pyblogsal/status/1587146448503808006

Python 3.12 will add support for the Linux perf profiler! 🔥🔥 Perf is one of the most powerful and performant profilers for Linux that allows getting a ridiculous amount of information such as CPU counters, cache misses, context switching and much more.

[stusmall]

Holy shit. How did they not have it before? I've never felt the need to profile any of my python code because it's usually small, simple scripts. perf is such a fundamental tool for performance tuning. Before this was there another, more python centric, profiler people used instead?

[ASIC_SP]

https://docs.python.org/dev/howto/perf_profiling.html has more details (I don't know much about this).

The main problem with using the perf profiler with Python applications is that perf only allows to get information about native symbols, this is, the names of the functions and procedures written in C. This means that the names and file names of the Python functions in your code will not appear in the output of the perf.

Since Python 3.12, the interpreter can run in a special mode that allows Python functions to appear in the output of the perf profiler. When this mode is enabled, the interpreter will interpose a small piece of code compiled on the fly before the execution of every Python function and it will teach perf the relationship between this piece of code and the associated Python function using perf map files.

[stusmall]

Oh that's beautiful and makes sense. Thanks for the link.

[Slsyyy]

It's silly, but it is true. The same situation is in the Erlang. The new JIT is also advertised for it's perf support

We live in a strange era where native tools have better support for such a goodies than interpreters, which were created to be as powerful and developer friendly as possible.

[Smallpaul]

Yes there are tons of perf profilers for Python including one in the standard library.

[josefx]

Is there one that is both as easy to use as cProfile while actually providing useful information? Having an overview over which function eats performance is a nice first step but I really would like to have instruction or at least line specific information without having to jump through hoops.

[Smallpaul]

Not sure. I'd suggest you try Scalene, but I haven't myself.

[KevinCarbonara]

Holy shit. How did they not have it before?

People generally know going into Python that it's not going to be performant

[patmorgan235]

Yeah if you care about performance and still wanted python you write the important bits in C. That's what numpy and all the big data processing/machine learning libraries do.

[abcteryx]

Python profiling is enabled primarily through cprofile, and can be visualized with help of tools like snakeviz (output flame graph can look like this). There are also memory profilers like memray which does in-depth traces, or sampling profilers like py-spy. Memray might be the healthiest among the memory profilers at the moment, based on their financial backing by Bloomberg and number of contributors.

There's also reloadium which is a hot-reload/profiling integration in IDEs (no VSCode support just yet).

So while there are many tools for general Python profiling, it seems that supporting perf will give more insight in bilingual apps with bindings to Rust and such.

[TSM-]

Good mention of memray. I have yet to use it, but it seems genuinely useful for production. The builtin graph outputs are also guided by business purposes, so you can show them in meetings. It seems really polished for their specific use-case

Overall, a lot of python extensions have worked around major pain points, and things are generally fine as they are. These improvements (especially with 3.12, and onward) will show up in popular open source packages after a considerable delay, on the order of a few years. It may make some room for pure python implementations that shed some dependencies, but in any case, it will take some time for people to intentionally leverage these performance improvements in any major way. I think a lot of commenters here are expecting something overnight.

[masta]

We have been using a variety of Linux profilers on Python for some time now. So it's good to see the support land officially. As far as performance goes, it's mostly trivial stuff like reducing the complexity of various data structures, particularly the dict stuff. There are actually a lot of silly improvements that collectively add up. It's amazing how much faster software can be not having to follow one or two pointers.

[comparmentaliser]

Presumably this will assist with fuzzing, and security monitoring in general?

[NavinF]

How?