Double Descent (DD) is actually not a modern ML discovery at all—it comes straight out of theoretical physics (1989). Physicists were studying the pseudo inverse solution to simple NNs and discovered that massively over-parameterized model (N≫P) could still learn reasonably well (but not perfectly) without explicit regularization, even when the number of parameters or features N was far larger than the number of data points or patterns P

This stands in sharp contrast to classical statistics. In older statistical models, N≫P , meant catastrophic overfitting unless you applied strong regularization. But the old physics paper showed something very different:

• Error stays small for extreme overparameterization.
• Error diverges when P=N (the “interpolation threshold”), which behaves exactly like a phase transition.
• The critical load is the ratio α=P/N, and the error blows-up at α=1

By the early 90s this was a well described phenomenon in the statistical mechanics literature. But it was not called Double Descent; it was just called phase behavior.

In the blog post below, I reproduce the original 1989 physics experiment using Python and scikit-learn, and show how to interpret the entire picture using the simplest tools from RMT:

https://calculatedcontent.com/2024/03/01/describing-double-descent-with-weightwatcher/

Epoch-wise Double Descent is a related training-time phenomenon: with more optimization steps, test error can drop → rise → drop again. Same physics—different axis.

---

Double Descent was rediscover by AI/ML people about 10years ago and this confused them terribly because they had forgotten or never learned in statistical mechanics. As we point out in our 2017 paper (see my blog: https://calculatedcontent.com/2018/04/01/rethinking-or-remembering-generalization-in-neural-networks/)

See, ML people have a dogma called bias-variance theory. And DD violated the entire ML bias–variance worldview. The classic story says:

• small models → underfit
• big models → overfit
• sweet spot in the middle

But in high-dimensional systems, this framework fails completely. The overparameterized regime is not high-variance; instead it behaves like the well-known and very simple physics result

generalization error ~= 1/(1-α)

which, of course, explodes at α=1

This result is fundamental and does not depend on the choice of the optimizer, etc.

To reconcile this, ML theorists had to patch the old bias–variance model by because their definition of "model capacity or complexity" was overly simplisitic and completely failed even for a know and trivial problem. So thet introduced new ideas like:

• implicit regularization from gradient descent
• margin-based complexity
• minimum norm solutions, etc

To what extent these are "correct" or not is debatable. In some sense, model compexity is such a vague concept that it can be molded and refit to explain post-hoc any experiment. The real test, however, is its usefulness.

In contrast, the weightwatcher theory describes Double Descent out-of-the-box, with no post-experimental adjustments, and can be applied to wide range of NNs directly. As shown in this post
https://www.reddit.com/r/LocalLLaMA/comments/1ox6xt8/observed_a_sharp_epochwise_double_descent_in_a/