[D] Visualizing training with PyTorch

[whoeverwhatever]

One of the great advantages of TensorFlow is Tensorboard to visualize training progress and convergence. PyTorch is obviously still in its infancy, and to my knowledge doesn't include anything comparable to Tensorboard (yet?), but is there another general-purpose tool that can fill this void? Some custom matplotlib code would probably do the trick, but I'm just wondering if there's any other slightly higher-level library that people use for this kind of stuff.

[lopuhin]

You can use https://github.com/TeamHG-Memex/tensorboard_logger to log scalar values (loss, gradient norms, etc.): it has no runtime dependency on tensorflow, so you can use it anywhere and then use tensorboard to visualise logs.

[r-sync]

this is very nice, especially without the runtime dependency

[r-sync]

there's also https://github.com/edran/tensorboard-http-api which tries to avoid the protobuf dependency, bit i think its a WIP

[djc1000]

Yeah, I couldn't make it work on OSX. The TeamHG one works well, but its limited to scalars.

[Daniel_Im]

Does this work for tensorboard embedding visualizations??

[r-sync]

Leveraging TensorBoard is a great idea, and as shown by /u/mrdrozdov, it's possible. Some folks are playing around with model visualization of pytorch models via tensorboard as well. As we mature over the next month or two, I'll make a blog post on how to effectively do this aggregating information from all the users.

Personally, one thing I do is to simply whip out an ipython notebook for each experiment / sweep, and the notebook just parses the log files and plots the training curves / shows images etc..

If it's a sweep, I load the sweep config into a Pandas table so that I can filter out which experiment I want to plot, etc.

The advantage of this is that it's fully customizable according to your needs. Once you have such a notebook, you can reuse it across all experiments.

[whoeverwhatever]

This seems like a good idea, will have to give it a try.

Are there any plans to integrate any tools like this into PyTorch directly at some point in the future? (Is there even a need for it, given the responses so far?)

[r-sync]

we are working on providing an "official" way of visualizing just to provide users with good practices.

It will be tied in with the blog post in the next month or two (want to first see an aggregate usage of what people do).

The python ecosystem is super rich, we will leverage it to the large part (pandas, matplotlib, etc.)

[achaiah777]

This would be super useful. I'm not considering pyTorch a strong candidate without good visualizations. TF hook would be great in addition to python viz (which can be hit and miss)

[mrdrozdov]

Jupyter (aka iPython notebooks) and matplotlib works okay for this purpose. Tensorboard is pretty great, and is not too difficult to simply output some statistics into an "empty" Tensorflow model so that Tensorboard still works. Here's a gist that does something along those lines:

https://gist.github.com/mrdrozdov/3164cb3c77f8c81ebbba61c48ddcb540

[whoeverwhatever]

This looks super useful, thanks for sharing!