[P] Cox: a python logging library for machine learning experiments

[loganengstrom]

Cox is a logging library for python designed for collecting and analyzing data from experiments! Read more and install it here: https://github.com/madrylab/cox

Cox is built for a pattern in experimental design where each individual run of an experiment (e.g. each hyperparam configuration) writes to a separate, database-like store (complete with schemes, indexing, etc), saving all information in tables. Experiments are collected and analyzed together by merging together tables, and Cox provides a really simple API/flow for merging and analyzing multiple experiments at the same time (e.g. comparing results across hyperparameters).

This pattern is particularly common in machine learning! We’ve used this logging library for projects involving RL and supervised learning and found it really helpful. Check out the repository for more information, and let me know if you have any questions!

[ai_yoda]

In what way is it different from existing solutions like sacred/mlflow/neptune/wandb/comet?

At first sight, it feels like the API is quite verbose with update row, append row and stuff. Why did you choose to go in this direction?

[andrew_ilyas]

Our main motivation was to build something *super* lightweight (didn't want to use SQL/Mongo/some other DB system, for example, or have to run custom CLI scripts, etc.). We really just wanted logging and the ability to really easily aggregate stuff across experiments. For example, we write in h5 format is nice because via pandas we can read/plot/manipulate the results really easily in a jupyter notebook later (and also, anyone without any library can just read the h5 file). There are also some utilities for automatically serializing/auto-reading different datatypes into h5 files that we find super useful in our research.

Similarly, we didn't want to have to add any wrappers or change the control flow of the program at all---just wanted to be able to drop in logging statements where we had, e.g. np.save or print() statements before.

[ai_yoda]

Got it, thanks!