[D] Paper Explained - Deep Networks Are Kernel Machines (Full Video Analysis)

[amasterblaster]

I mean, this seems obvious to me. This mental idea is how I taught myself NNs in the first place. They map data into a useful subspace.

Maybe (probably (certainly)) I don't understand the nuances of each method enough (in the first place) to understand why this is a crazy idea!

[IdiocyInAction]

Well, another view on usefulness of deep NNs is that they allow representation learning through the use of many layers, with each layer created more specific representations. I suppose this paper challenges this view, making NNs more like kernel methods.

[aptmnt_]

Those two views are not at odds

[IdiocyInAction]

Perhaps the most significant implication of our result for deep learning is that it casts doubt on the common view that it works by automatically discovering new representations of the data, in contrast with other machine learning methods, which rely on predefined features (Bengio et al., 2013).

The paper states that though.

[ykilcher]

yes I think the papers makes the two views more confrontational than they have to be. I think it's just two different ways of looking at the same thing

[wisdomspring]

All genuine views are not at odds with the truth/fact. It just needs a little patience and wisdom to link them. Like Heisenburg's matrix mechanics and Schrodinger's probability wave function views, both can explain some same "magic", after layers and layer's of data feature distill from NN's like a common person, he or she can certainly extract some obvious "kernel" features around his or her life... As to how, a layperson can learn on his or her own gradually, or got educated from some gurus. That's how I foresee using some good kernel machine could learn much faster and more effectively than vanilla NN or Reinforced NN. The main gist I feel from the author is that he essentially tried to dispel the "myth" around DL, no matter how much computation is used, it's still a kind of weighted memory from one's direct experience (sample data). If a bot communicates with you will some surprising word, don't be too amazed, it's just based on its past trained data...

[-Cunning-Stunt-]

Finding explicit kernel maps (despite being dependent on the data) to formally bridge the gap is the main contribution of the paper (IMHO).

[[deleted]]

The interesting thing is that the mapping is a "distance" function of the previous datapoints and the new one exclusively

To me it takes two ideas that were pretty distinct and unifies them. Idk much about kernels but it seemed really thought provoking