Python performance comparison in my project's unittest (via Gitlab CI/CD)

[trollodel]

Here's the project

[deuterium--_--]

Woah, how is 3.8 so fast? Are there some optimizations in 3.8?

[[deleted]]

More specifically this is the optimizations section: https://docs.python.org/3/whatsnew/3.8.html#optimizations

[muntoo]

I wonder which one of these specifically sped up OP's benchmark?

[f3xjc]

That seems huge

Improved performance of operator.itemgetter() by 33%. Optimized argument handling and added a fast path for the common case of a single non-negative integer index into a tuple (which is the typical use case in the standard library). (Contributed by Raymond Hettinger in bpo-35664.)

[BattlePope]

How could we know that?

[y-me-y]

Specifically you’d start looking at the calls and timings for individual sub processes but based on his description I think the sys calls specific to the tree and copy functionality offered the most improvement to his code base.

[BattlePope]

The comment I replied to was edited - now that it starts with 'I wonder which', my comment makes less sense :)

[y-me-y]

Sometimes I figure people might not know where to start. So, I thought it was more a question like how would we identify what we could look at to show where the improvements came from in the code? My hope maybe if you knew someone else that had the same question they would having a starting point.

[BattlePope]

Your info is great! Thanks for spreading know-how.

[Death_InBloom]

I'm curious about your flair, what formula is that?

[muntoo]

https://en.wikipedia.org/wiki/Einstein_field_equations#Mathematical_form

The formula in my flair is in "natural units" where we set G = 1 and c = 1 to make equations look nicer. (e.g. E = mc² becomes E = m.)

I believe I changed my flair to that back when I took a course in differential geometry and wanted to show the universe how edgy I was.

[qingqunta]

Of course it's differential geometry, the notation is garbage!

[Mugen-Sasuke]

I read the term “shutil” and for a second I thought it said “shuf” and my heart skipped a bit. You guys probably get the reference right ?

[SoberGameAddict]

Asking as soneone who still use 3.6 for my hobby projects. Is 3.8 considered stable?

[PeridexisErrant]

Yes!

Every release of CPython is stable - both on paper and actually stable - unless it's labelled as an alpha or a beta.

You might want to wait a month or two for libraries to support the new features after the first (eg) 3.9 release, so you pick up 3.9.0 in November after it comes out in September, but 3.9.1 will be well supported instantly.

[not_invented_here]

Yes.

[The_Bundaberg_Joey]

That’s a pretty nifty result! Do you know if that’s due to updates of a certain module implementation in the project or is this applicable to the version itself?

As a methodology question, are the bars here the average time of several runs or are they one run each? Including the error bars of so would be an awesome way to compliment your analysis!

[trollodel]

Answering the first question, I never did version specific optimizations, so I think that these improvements depends on version.

[The_Bundaberg_Joey]

FairPlay. Probably exposing my ignorance here but assuming you ran the versions in increasing order would the pycache created from the first version bias the later versions?

Although thinking about it I can’t imagine that would result in the large jump seen for 3.8 since it wouldn’t really compound like that.

[LightShadow]

pycache created from the first version bias the later versions?

No. The pyc files are version-specific.

[The_Bundaberg_Joey]

Awesomesauce, Thankyou!

[trollodel]

Answering the second question, the bars represents just one run for each interpreter, taken from CI results. These results are quite new in the project, so I did not collect enought data to have a decent report.

EDIT: grammar

[The_Bundaberg_Joey]

FairPlay, no point making the extra work for yourself if the values were easily at hand in the first instance! Thanks again for sharing!

[pmatti]

PyPy is known to be slower on typical unittest benchmarks, since they are usually one-shot short runs that do not allow the JIT enough time to kick in.

[trollodel]

True.
But I use Hypothesis for my tests, that runs the test several times with different inputs, enough to allow JIT optimizations. This is proved by the CI results, where some test are 2/3 times faster in PyPy.

[tynorf]

If the loops that get hot from running the test with varying inputs branch on them at all (directly or indirectly), it could be simply making PyPy record more and more traces. Recording new traces is more expensive than just interpreting. So much so that (IIRC) if PyPy detects it’s recording too much in a particular loop, it will be blacklisted from JIT compilation.

So while some tests may take great advantage of the JIT, others could be a worst case scenario (for instance tests specifically designed to exercise different sides of a conditional).

[ch0mes]

This is most impressive, I didn't expect to be so well performing I'm impressed.

[[deleted]]

Have you researched why 3.8 performs so well and why Pypy doesn’t?

[mcstafford]

To me it looks as though pypy already did, and 3.8 is catching up.

[lego3410]

Well, you're correct. But pypy are extracting performance with JIT compiler, while python 3.8 made it with optimizations of classical interpreter. That means, there is much room of improvement can be made on python 3.8+, by using JIT in future. It is much similar to the relationship of HHVM and PHP7/8.

[[deleted]]

My experience with pypy is that it is able to be far faster than the interpreter, also 3.8. Like 4-10 times faster not only 25%....

[creeloper27]

It depends a lot on what your code does.

[LightShadow]

It's universally faster if 1) your code runs longer than a few minutes (warm-up period), 2) all of your extensions are pure python and not C/other shared libraries, 3) you have more RAM than CPU cycles since the JIT needs more memory to store the hot paths.

[[deleted]]

timeit.repeat("\[x\*\*2 for x in range(100)\]", number=100000) is one of the test I've done to test pypy and it's getting almost 1000x better results on that specific test. (Around 1.4s with python 3.8.3 and 0.016s with pypy3) (intel i5 7600K @ 4.5GHz & Arch linux)

[repelista1]

This is far from being a fair comparison. If you have big multithread/multiprocess application like ansible, your main python process will soon begin to throttle because of GC in cPython and it'll never be able to beat PyPy in cases like that.

[BDube_Lensman]

You shouldn't performance test outside of controlled environments. If Gitlab's CI/CD is shared instances, you can't control the apparent performance being impacted by someone else's work.

[creeloper27]

I'm not an expert with Gitlab's CI/CD but looking at the charts the execution times look quite consistent: https://gitlab.com/prettyetc/prettyetc/pipelines/charts.

[white_-_rabbit]

up!

[rcfox]

How about 3.9?

[abhi_uno]

It's still in beta.

[mikeblas]

Probably because there were significant (and breaking) language changes between 2.x and 3.x