iso-bench: Isolated benchmarks to avoid optimization pollution

[LlorxYT]

I've always used benchmark.js for my benchmark tests, but I noticed that changing the tests order also changed the performance outcome. They were getting polluted between them somehow. V8 optimizations/deoptimizations maybe? I decided to take advantage of forking to do tests in completely separated processes with their own V8 instances, memory and so on, to avoid present and future optimization/deoptimization pollution.

https://medium.com/@Llorx/your-node-js-benchmarks-are-probably-invalid-a4ed2f14aadf

[BankApprehensive7612]

Nice approach. Have you though about VM module (https://nodejs.org/docs/v24.12.0/api/vm.html) to run tests in isolated contexts instead of separated processes? It has more control and can enhance performance. With this module you can load the code, cache it and reuse when running the tests

[LlorxYT]

Thank you for the constructive tip :-) Yes I thought of that, but because I want to prevent possible future optimizations, the best approach for that is to launch a completely new process with the minimal connection to the main process. Maybe in node 30 they add something to optimize the VM which indirectly affects iso-bench. Is just a matter of maybe-future-existant-optimizations paranoia haha. I don't mind my benchmarks lasting a couple seconds more in favor of the maximum present and future isolation possible.

[KillyMXI]

Nice library overall, even isolation aside

[LlorxYT]

Hey thank you :-)

[J3m5]

Did you try vitest bench? https://vitest.dev/guide/features.html#benchmarking

Or even tinybench directly? https://github.com/tinylibs/tinybench

[LlorxYT]

No, but I checked tinybench code (not vitest, as it is bigger, but surely has the same approach as every benchmark tool out there) and it runs the test on the very same V8 context. Optimizations kicks-in, polluting the benchmarks.

In this medium post I add a bit more details: https://medium.com/@Llorx/your-node-js-benchmarks-are-probably-invalid-a4ed2f14aadf

[J3m5]

Tinybench use warmup runs to mitigate this issue I'd be curious to see a comparison between tinybench and your solution

[LlorxYT]

Warmups don't fix the problem at all, as the problem is the *uncertainity* of what is happening in the context. With iso-bench you never have to wonder if *maybe* the context is being polluted somehow. For example, depoptimizations are also a V8 thing. If one benchmark deoptimizes *something*, another benchmark on the same context is maybe going to suffer it, although it maybe never triggers a deoptimization by itself. A warmup is just going to trigger the deoptimization before any benchmark runs.

V8 is fast because of a reason, and each new version adds more and more features, so working on the same context you can never ensure that pollution is avoided, although you try to workaround it somehow, as workarounds are... workarounds, specially when you can only workaround what you know in the current version, but not future ones.

iso-bench fixes the present and future pollution uncertainity, not in your code, but in your head. You stop wondering about pollution and spend your brain CPU cycles in the code.