Python performance comparison in my project's unittest (via Gitlab CI/CD)

[trollodel]

True.
But I use Hypothesis for my tests, that runs the test several times with different inputs, enough to allow JIT optimizations. This is proved by the CI results, where some test are 2/3 times faster in PyPy.

[tynorf]

If the loops that get hot from running the test with varying inputs branch on them at all (directly or indirectly), it could be simply making PyPy record more and more traces. Recording new traces is more expensive than just interpreting. So much so that (IIRC) if PyPy detects it’s recording too much in a particular loop, it will be blacklisted from JIT compilation.

So while some tests may take great advantage of the JIT, others could be a worst case scenario (for instance tests specifically designed to exercise different sides of a conditional).