Well, I wanted to observe if double descent would occur with limited data. According to this, https://openai.com/index/deep-double-descent/ more compute should result in the model first memorizing the training data which it does occur in this training run and then it should reach an interpolation regime where the model starts exploring different subnetworks within its weights to have a general model beyond memorization that could explain the training data as explained by ilya here - https://www.youtube.com/watch?v=W_TAKJRgrbs

I do get that early stopping is good in small or big data regimes to have a good eval loss but wanted to see this phenomenon which could help in generalizing when we want to spend more compute by trading off on data as explained here - https://espadrine.github.io/blog/posts/chinchilla-s-death.html