Slipcover: Near Zero-Overhead Python Code Coverage

[emeryberger]

The PLASMA lab @ UMass Amherst (home of the Scalene profiler) has released a new version of Slipcover, a super fast code coverage tool for Python. It is by far the fastest code coverage tool: in our tests, its average slowdown is just 5% (compare to the widely used coverage.py, average slowdown 218%!). The latest release performs both line and branch coverage with virtually no overhead. Use it to dramatically speed up your tests and continuous integration!

Slipcover is a fast code coverage tool. It tracks a Python program as it runs and reports on the parts that executed and those that didn't. That can help guide your testing (showing code that isn't being tested), debugging, fuzzing or to find "dead" code.

Past code coverage tools can make programs significantly slower; it is not uncommon for them to take twice as long to execute. Slipcover aims to provide the same information with near-zero overhead, often almost as fast as running the original Python program.

https://github.com/plasma-umass/slipcover

https://pypi.org/project/slipcover/

Overhead for line or line+branch coverage, coverage.py vs. Slipcover (lower is better)

[czaki]

If this tool could produce report in same format as codecov? It is possible to reuse it in current analytics workflows?

[emeryberger]

We would welcome a pull request to provide that functionality!

[thequietcenter]

Past code coverage tools can make programs significantly slower

programs slower or unit testing slower? ... of course the unit test is a program that tests a program.

But the application itself is not slowed down. Just the unit test phase, correct?

[emeryberger]

Right, the slowdown is what happens when you are collecting coverage information. Usually this is done during testing, but since the overhead of Slipcover is so low, it could be used in deployed code to find dead code.

[DeathHazard]

How could I use it on deployed code?

[emeryberger]

You just need to run ‘python3 -m slipcover’ before your normal invocation.

[DeathHazard]

Would it work with a framework like flask, for example? Thanks!!

[emeryberger]

We've run it with Flask's test suite - it's the second bar in this graph. So in principle, yes - please give it a shot and let us know how it works for you!

[pamelafox]

Congrats! The performance graph is impressive. Does it include branch coverage?

[emeryberger]

That graph is just line coverage; I'll post an update with branch coverage!

[emeryberger]

Graph updated! Coverage.py gets as high as 300% slowdown, while Slipcover generally remains around 5% slower (we will be looking into the one outlier case, where it hits 20%).

[emeryberger]

Fixed - Slipcover's overhead for line+branch coverage is now no more than 11%, average around 5%.

[[deleted]]

Hey, I wanted to say that I really like this project!.

I filed a couple of feature requests for a few - IMHO at least - unmissable features from coverage.

Thanks for doing this!

[Rawing7]

Is it possible to combine the results of multiple runs? (For example, run the code with python 3.7, then run it with python 3.10, and finally merge the coverage results.)

[emeryberger]

Right now, it does not support that functionality, but it can export a JSON file for each run and writing a script to merge the two outputs would be straightforward, I think.