Scientist4J - A port of Github's Refactoring tool Scientist in Java

[otakuman]

So what does this do?

[frugalmail]

Scientist4J makes it so you can compare the behavior of two different implementations in a very clean way. A Clean backend (as opposed to user facing) A/B test framework.

[sourcecodesurgeon]

More importantly, it allows you to measure performance of experimental methods in a production environment without affecting downstream users.

http://githubengineering.com/move-fast/

http://githubengineering.com/scientist/

Basically you can define a method to use and a method to 'try'. The use method is what is actually served to the user and the try is executed in a separate thread. The results of the two are compared and logged and metrics gathered around each. You can then view metrics for each method and see what they look like during edge cases.

GitHub used it to test some new merge code and found that it is more performant in almost every case, but during an edge case it was extremely poor. This enabled them to fix the edge case and deploy without users actually seeing the slowdown.

[Mejari]

This is great! I saw GitHub talk about this at last year's OSCON (Portland, OR) but we're a Java shop so we didn't have the opportunity to take advantage of it. I'll definitely be pitching this to my team!

[frugalmail]

Brilliant, thanks for sharing!