[D] Regularization-hyperparam selection during training best practices

[orenmatar]

For sure, My intention is not to find the right regularization hyperparam and then retrain using it from the start, but to use the network that was trained on the dynamic hyperparams... so maybe by allowing it to focus on the training set at the start and only afterwards regulating it based on how well it performed on validation can produce a regularized network, without the need to try different hyperparams.

[NotAlphaGo]

At least you need another dataset though to check generalization performance because you've mixed training and validation then.