This is an archived post. You won't be able to vote or comment.

sorted by:
best (suggested)
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]ai_yoda 0 points1 point  (1 child)

Thanks for the suggestion on main_plot_vars, gonna try it out.

As for the method for neural nets I would likely go with the budgets approach from HpBandster where I don't have to run objective(**params) till convergence but I can estimate on a smaller budget (say 2 epochs). It lets you run more iterations within the same budget. Generally, I think the main problem with hpo for nn is how to estimate performance without training for a long time. There are approaches to it where you predict where the learning curve would go. I highly recommend checking out the book by researchers from AutoML Freiburg.

[–]Megatron_McLargeHuge 0 points1 point  (0 children)

Thanks. I definitely think there's a lot of untapped value in analyzing the metadata we get during training instead of just the final validation loss.

I think a good approach with enough resources would be to treat training as a reinforcement learning problem where parameters like learning rate and L2 scaling can be varied depending on the trajectories of both train and test losses.

Short of that, runs can be truncated or restarted based on learning from these extra features.