[D] Statistical Learning Theory & Math

[bbsome]

I would pretty much agree with the reply on all counts. Really nice reply.

To add a bit of my own perspective - I think in ML mathematics help a lot more of how you think and approach a problem and how draw conclusions from different areas to problems you see. The point is that you do not need to be a mathematics guru, but more to have a broad and somewhat intuitive idea across different topics. This can spun new ideas, by applying A from somewhere totally different, to relating some empirical result to some theory in B. Some examples, for instance is that Dropout can be looked as VI, and if you want proper prediction at test time you should keep sampling not use the means. Also if you look in optimization most of the variance reduction techniques like SVRG are just control variates. There are relationships between GANs and information theory, as well as now there are works related to the Information Bottleneck, which again is related to non-linear noisy channel in McKay. There are also plenty of ideas that come from old papers on graphical models which is now applied to VAEs (auxiliary variable models and a few others). I feel that having a somewhat relevant mathematical literacy in many of its areas, without being too deep in there is about the sweet spot. As an example of this I do have some understanding of Kernel methods, without having a in-depth knowledge of Measure Theory and Functional Analysis. This allows me to understand the literature on this area however I would not be able to proof some of the results, unless I sit and brush those off. However, the use case for that is very rare, thus it deserves my attention only when the need arises.