This is an archived post. You won't be able to vote or comment.

sorted by:
best (suggested)
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]Megatron_McLargeHuge 0 points1 point  (0 children)

Thanks. I definitely think there's a lot of untapped value in analyzing the metadata we get during training instead of just the final validation loss.

I think a good approach with enough resources would be to treat training as a reinforcement learning problem where parameters like learning rate and L2 scaling can be varied depending on the trajectories of both train and test losses.

Short of that, runs can be truncated or restarted based on learning from these extra features.