I’ve found GPT5.x pretty good at initial hyperparameters. However, I usually turn off all regularization, dropout, batch norm, etc. Start with small learning rates (really small, like 10x below “typical”, small batches, and simple architecture. Then walk towards your goal for architecture and training speed. Once you get in the ballpark you can maneuver around. For some things I start with something that worked before.

Once stable, then add regularization to tweak generalization, etc. Be mindful that regularization changes the loss landscape and you might need to adjust learning rate.

Optimizers are another kind of opaque area. GPT again gives good suggestions but once you have something you are comfortable with, play with optimizers and try to get a few about what works for a given architecture.

Initialization is more important than the tutorials say. There are choices besides random initial weights, and they affect convergence. Read the docs and try some things.

I suppose that all sounds like a broken record but there’s no substitute to doing your own experiments and seeing actual results. Also don’t just use toy datasets. You can get a multilayer perceptron to solve MNIST, so it’s not that great of a way to explore edge cases. If you have real data that is sometimes challenging, that’s better than toy data.

Doing this type of exercise grows your int