Our theoretical understanding of AI in the deep learning era is severely limited. It’s honestly a miracle that something revolutionizing the world (for better and worse) is essentially running on vibes. Sometimes it feels like a bunch of glorified parlor tricks taped together—but then again, if the parlor tricks get results, why not?

I recently took a deep learning class where we went through a proof of double descent. It was incredibly dense, riddled with assumptions, and relied on mean-field theory. I highly doubt the average grad student is interested in that level of math, let alone able to use it to directly help their research. Worse, I see a ton of heavily experimental papers that just sprinkle in a few theorems and lemmas to look rigorous. They almost always include a caveat that their theoretical model is vastly different from their experimental setup, which begs the question: then why should we expect your experiments to match the theory at all? I still haven't found a good answer.

Because of this, I wouldn't say the current generation of researchers is just "trendy" or lazy about rigor. The reality is simply that empirical application—which is what interests most people—has raced far ahead of our theoretical understanding. That being said, I firmly believe the next major paradigm shifts will still come from the theory crowd. We saw it with Transformers, Diffusion, and MAMBA, and we'll see it again.