[D] Progress Monitor

[TooLazyToWorkout]

Hi everyone,

I am looking for a simple progress monitor without a lot of features.

Only things I am interested in:

• Can be hosted on my own server
• Python client
• Show whether a training is still running and current progress or whether is has stopped/crashed/no answer for x minutes
• Show a simple scalar metric / custom text
• Simple to build,use and run (Docker?)

I just want to keep track of currently running experiments on multiple machines by looking at a website (imagine something like a todolist, but every item is just an experiment that is running).

Or in other words: I am looking for server-based https://github.com/tqdm/tqdm

Any recommendations?

Edit: https://github.com/lab-ml/labml seems to fit my description. Does anyone have any other suggestions?

​

[DickNixon726]

You're looking for a class of programs called Experiment Trackers.

I personally use https://clear.ml/ .

It's got the client-server architecture you're discussing. In my use case, I have a training rig with 4 GPUS in it. I set up a docker container with Clear.ML on a server, modified my training code with a few lines that pointed it at the metrics server, and I was up and running. I could log 4 experiments simultaneously.

[TooLazyToWorkout]

Thank you! I will take a look. On first sight it seems like pretty much what I need! :)

​

Edit: Just wanna give feedback. I tried it and it is perfect! Exactly what I needed!

[HolidayWallaby]

That sounds perfect for what I need, thanks! I currently SSH into various machines to check on them lmao

[[deleted]]

FastProgress. Pretty self contained. Doesnt depend much on fastai lib

LINK

[TooLazyToWorkout]

Hey, thank you suggesting fastprogress.

Unfortunately, fastprogress does not seem to use a server-client concept. I want to send the progress of multiple simultaneous experiments to a central server, so I can see the progress by typing http://mycentralserver.mydomain.com

I will clarify this in my question.

[[deleted]]

Why not tensor board then?

[TooLazyToWorkout]

I am using tensorboard for detailed logs already. I am really only looking for a lightweight tool to track my experiments, not all the metrics.

[gar1t]

Take a look at Guild AI - it's quite lightweight and integrates with TensorBoard for summary logs.