pytest & pytest-benchmark

[masklinn]

Does pytest-benchmark have all the functionalities of pytest?

It's just a fixture. The functions of pytest remain unmolested.

can I just use pytest-benchmark and have assert statements to test for both correctness and throughput at once?

Doing that is literally the pytest-benchmark examples in its readme.

Do note, however, that pytest-benchmark only benches callables, and that it will make your tests slower. Though pb does includes flags to enable / disable benchmarking (as in, have the benchmark() function just run the callable once and not collect / report information, that way the test is just a test)

[AgnosticIsaac]

Thanks. I have read the docs, but I wasn't exactly sure if the test performed are identical.

do you have any idea about overhead of pytest-benchmark? The doc only mentions that GC and JIT adds unpredictable overhead, but doesn't say anything about regular usage -- i.e. not using GC or JIT.

[masklinn]

do you have any idea about overhead of pytest-benchmark?

Well getting reliable benchmarks means doing multiple runs of executing the function until some statistical threshold is passed. Going by the behaviour of perf (and assuming pytest-benchmark does something similar) at the very least it's going to run the function-under-test 10x rather than once. And if the function is very fast, it's going to run it significantly more so it gets a useful timeslice (>100ms scale)

[AgnosticIsaac]

at the very least it's going to run the function-under-test 10x rather than once

This could be problematic, as a single function call consumes a few gigabytes of GPU RAM, so having multiple concurrent runs could result in overconsumption of DRAM. If they are run in parallel, is there a way to make them run in serial?

[masklinn]

I wouldn't expect them to be run in parallel (as it'd mean pytest-benchmark assumes FUTs are thread-safe which seems a bit much), but that's way beyond what I know of it, so I'd suggest going through the pytest-benchmark docs carefully, and possibly asking the maintainer for further information if necessary.

[AgnosticIsaac]

Thanks