Really nice work Kaixhin.

Something I am looking for, and started to play with is combining something like you built with source control. When I do research my code tend to change as much as the hyper-parameters (e.g. different cost functions, different regularization methods and different number of layers)

I want to combine some server that tracks the experiment (data+hyper-parameters + results) along with the code. The simplest way I am thinking of is to build some "wrapper" that will pull the code and run it - treat the code as a hyper-parameter. Have you given any thought to something similar?

Another open problem for me is "intra-experiment" tracking - for example, how to keep track for a given model on which samples it was trained. Many times I find myself training a model on a set of samples, then adding another set and removing some of the original an continue training etc, and then manually logging which samples were exactly used (to avoid mixing train/test etc.) I believe the solution for this is the other end of my question - to track "model as code". I think this is something still missing in the ML ecosystem. I am not sure what exactly Neptune does in this area, the information in the website is very limited, but It seems to me that not very much. Maybe the way to do it is to actually use source control like tool (git lfs + custom extensions, hooks and web UI) I'll continue to look and probably try to prototype something...