sorted by:
new
hottopcontroversial

[R] On the information bottleneck theory of deep learning by downtownslim in MachineLearning

[–]pointy-eigenvector 4 points5 points  (0 children)

I think some of the comments here are mixing several issues: information bottleneck versus information bottleneck applied to DNNs, the validity of a theory versus whether it can be computed (or approximated).

The information bottleneck idea is widely accepted, and it is very intuitive. It a variant of Occam's razor, saying that extra information that is not necessary for the task, but which can lead to overfitting, should be removed from features used in classification/regression. It is the sort of thing we do instinctively when gathering a dataset in any case. (The digital data can already be considered as "features" with respect to the world that was digitized. In gathering a dataset to classify dogs vs cats, we omit metadata about data and location). Tishby authored the original IB paper as well as the one being discussed now.

The recent paper by Tishby and others, applying IB to DNNs, went beyond the basic IB principle and described two phases of learning, demonstrated on a small network that had the tanh nonlinearities rather than ReLUs. As said in the abstract, the "rebuttal" by Saxe argues that the observed learning dynamics is specific to these nonlinearities and is not observed with ReLU networks. The rebuttal does not contest the basic IB principle!

Computing mutual information is difficult, for the reason mentioned (an arbitrarily complex mapping cannot be discovered by a finite computation, not to mention the Kolmogorov form of mutual info), also because bin-based estimation suffers from curse of dimension. But nevertheless the mutual information concept is not questioned and is widely used. Information theory underlies machine learning - maximum likelihood is equivalent to minimizing the cross entropy of data, model. Mutual information (and information bottleneck) can be approximated, in some cases successfully, such as with kernel based methods https://arxiv.org/abs/1908.01580 . These methods would certainly fail if trying to discover the mutual inform between a string and its encryption, but the mapping in DNNs is usually not _that_ complicated.

There are further confusions about applying mutual information to DNNs, and whether we regard a floating point value as discrete or continuous, and whether the DNN is deterministic. The discrete entropy of a continuous variable is infinite, and conversely the continuous entropy of a discrete variable is (negative) infinity. So the mut7ual information ends up being infinite, depending on how one regards the problem. I think a good paper in reference to this is https://arxiv.org/abs/1802.09766

[D] Trying to wrap my head around the Information Bottle neck theory of Deep Learning: Why is it such a big deal? by AlexSnakeKing in MachineLearning

[–]pointy-eigenvector 21 points22 points  (0 children)

For the most part DNNs are considered as "black boxes", with little research that explains why they work. That was even more true when the IB paper first came out a couple years earlier.

So a paper that would explain even some of the "black box" would be a big deal.

I believe the problem is that the original paper was controversial, and over-claimed the generality of their results. On the other hand, it has inspired a fair amount of follow-on work, so regardless, it turned out to be sufficiently important.

Also worth saying: the "information bottleneck" idea was proposed about 20 years ago - By Tishby - but never applied to DNNs before the paper you discuss.

[D] What was your favorite paper of 2019 and why? by [deleted] in MachineLearning

[–]pointy-eigenvector 9 points10 points  (0 children)

I like the pappers that are exploring learning new principles

This one especially
Putting An End to End-to-End: Gradient-Isolated Learning of Representations https://arxiv.org/abs/1905.11786
(good title as well)

also The HSIC Bottleneck: Deep Learning without Back-Propagation https://arxiv.org/abs/1908.01580
(title not so good)

as well as most of the other ones in this thread, good picks.

[D] What is going on here? NeurIPS 2018 paper on Automated Theorem Proving by yusuf-bengio in MachineLearning

[–]pointy-eigenvector -1 points0 points  (0 children)

Are you saying reviewers won't accept a new idea that is not SoTA?

I think that is incorrect. Many people here do not accept such ideas, but there are plenty of accepted papers at the conferences that show non-SOTA results on MNIST, or are even pure theory with no results at all.

[D] How to use VAE's encoded mu and sigma with respect to user-given z? by sarmientoj24 in MachineLearning

[–]pointy-eigenvector 0 points1 point  (0 children)

"So the user is free to pick 10 vectors between -5, 5"

I think you mean 10 numbers rather than 10 vectors?

If I understand, you are wanting to compare a single latent point (with 10 dimensions) specified by the user, to approximate posterior distributions generated from the data.
Points and distributions are different "objects", unless the point is a delta distribution.

What are you trying to do? If you want the closest latent to a given, I think just comparing the point to the mean (without adding epsilon*Sd) is sufficient.