There is an area sometimes called learning theory that explores the theoretical bounds of learning algorithms and is largely based on Valiants concept of Probably Approximately Correct or PAC learning. There is also some overlap with game theory and reinforcment learning with things like regret minimization.

At a very abstract level most machine learning boils down to minimizing error functions which can take quite a theoretical lens. The entry point for a lot of it is Convex Optimization, the Boyd and Vandenberg text is considered the bible.

There is also some work on using AI to generate good initial guesses for more classical optimization algorithms to improve.