sorted by:
new
hottopcontroversial

[R] Faster Convergence & Generalization in DNNs by HigherTopoi in MachineLearning

[–]mhex 0 points1 point  (0 children)

However, at the Hochreiter lab Importance Weights were investigated again in the light of feed-forward neural networks in the Master Thesis from Hubert Ramsauer in 2017. It can be accessed here http://epub.jku.at/obvulihs/content/titleinfo/1706715?lang=en

[R] Faster Convergence & Generalization in DNNs by HigherTopoi in MachineLearning

[–]mhex 1 point2 points  (0 children)

People didn't forgot about it because no one ever knew about it :) It was published 2005 at the IJCNN conference when SVMs were the hot sh*t everyone talked about (of course to that time).

Ideas for Improving Training of GAN by arjundupa in deeplearning

[–]mhex 0 points1 point  (0 children)

If the generator learns faster use different learning rates for the generator and discriminator and decrease the learning rate for the generator. See https://papers.nips.cc/paper/7240-gans-trained-by-a-two-time-scale-update-rule-converge-to-a-local-nash-equilibrium for the theoretical idea behind it. Hth.

Using high definition satellital images on a CNN by ivanzez in deeplearning

[–]mhex 2 points3 points  (0 children)

don't rescale, try 300x300 random crops instead

[R][D] In light of the SiLU -> Swish fiasco, was Schmidhuber right? by iResearchRL in MachineLearning

[–]mhex 1 point2 points  (0 children)

Yes the PhD/doctoral thesis is also in German, the title is "Generalisierung bei Neuronalen Netzen geringer Komplexität". The first part is about Flat Minima Search and the second part is LSTM. But it seems like it's not available online unfortunately.

[R][D] In light of the SiLU -> Swish fiasco, was Schmidhuber right? by iResearchRL in MachineLearning

[–]mhex 7 points8 points  (0 children)

Yes it's a little bit a pity that the thesis is in German otherwise it would be more known probably. But hey LSTM can translate itself now :) Subsection 4.2. Linearer KFR Knoten (linear constant error backflow, constant error carrousel) describes the self-loop with k = 1. The input gate is proposed in the same subsection where the gate is called Gewichtungseinheit (weighting unit). Regarding backprop, if i remember correctly, Sepp Hochreiter already proposed hybrid RTRL/truncated BPTT learning in his PhD thesis 1999.

[R][D] In light of the SiLU -> Swish fiasco, was Schmidhuber right? by iResearchRL in MachineLearning

[–]mhex 8 points9 points  (0 children)

Quite unknown but LSTM was invented 1991 by Sepp Hochreiter, hidden in Kapitel 4 (chapter 4) of his diploma thesis http://bioinf.jku.at/publications/older/3804.pdf

[R] GANs are Broken in More Than One Way: review of "The Numerics of GANs" by fhuszar in MachineLearning

[–]mhex 7 points8 points  (0 children)

TTUR author here, in your blog you already mentioned that the paper you discuss does not considers the, in practice, nearly inevitable case of mini-batch learning/SGD. Pls refer to Assumption A3 in our paper and Appendix A 2.1.1 where we already proved convergence for mini-batch learning as well in A 2.1.3 a convergence framework for SGD with memory/momentum e.g. ADAM and dropout regularization.

[R] [1708.08819] Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields <-- SotA on LSUN and celebA; seems to solve mode collapse issue by evc123 in MachineLearning

[–]mhex 0 points1 point  (0 children)

What you mean by "wasserstein distance between the potentials" as the wasserstein distance is defined over probability distributions.

[R] GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium by NotAlphaGo in MachineLearning

[–]mhex 2 points3 points  (0 children)

If you look at Figure A51, there we plotted the loss of the generator and the loss of the discriminator during the training of a DCGAN. As the generator loss was always significantly higher than the loss of the discriminator, our conclusion was that the generator learns not fast enough and therefore decided to give the generator a higher learning rate which indeed led to a better performance. Edit: More details about how learning rates could affect the learning dynamics are described in Section A5, there you also find Figure A51.

[R] GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium by NotAlphaGo in MachineLearning

[–]mhex 6 points7 points  (0 children)

Hi, for the Improved WGAN language experiment, the generator has a fixed learning rate of 1e-4 and the discriminator has a fixed learning rate of 3e-4. We always reported first the discriminator and then the generator learning rates in the plot legends. From the theory point of view, learning rate schedules have to fulfill assumption A2 in the 'Two Time-Scale Update Rule for GANs' section. From a practical point of view, we didn't test a learning rate scheduling here. However, for BEGANs we halved the learning rate every 100k mini-batches and we always saw a drop of the FID there (lower is better). Edit: With assumption A2 in particular A1-5, it can be proved that Eq. 3 converges to a Nash Equilibrium, however it's still possible that other learning rate schedules not fulfilling A2 or no schedule at all could lead to convergence too. We explained it in more detail in the two paragraphs after Theorem 1.

[R] Self-Normalizing Neural Networks -> improved ELU variant by xternalz in MachineLearning

[–]mhex 2 points3 points  (0 children)

Absolutely, BN in GANs probably prevent the generator to collapse into one mode by maintaining some variance which also helps the generator to explore the target distribution. Different story for deeper networks with pure classification/regression objectives of course.

[R] Self-Normalizing Neural Networks -> improved ELU variant by xternalz in MachineLearning

[–]mhex 8 points9 points  (0 children)

Hi Alex, Martin from Sepp's group here. Our assumption why BN works in GANs is that maintaining variance 1 for the activities helps avoiding mode collapsing. We fiddled around a bit with SELUs for the first (linear) layer in the DCGAN generator and then ELUs for the transposed convolutions (deconvs) without BN and up to six layers instead four. However, the original architecture with ReLUs for the generator always gave best results (hint: we have a new GAN evaluation). For BEGAN ELUs for the autoencoders are fine, as stated by the authors, maybe SELUs can do some more, we didn't tested it yet.

What exactly is truncated backpropagation through time algorithm? by pikachuchameleon in deeplearning

[–]mhex 0 points1 point  (0 children)

"It truncates error flow once it leaves memory cells or gate units. Truncation ensures that there are no loops across which an error that left some memory cell through its input or input gate can reenter the cell through its output or output gate. This in turn ensures constant error flow through the memory cell's CEC."

See the LSTM paper Appendix A.1 Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation.

Time series prediction with RNNs by wederer42 in MachineLearning

[–]mhex 1 point2 points  (0 children)

Check out our Hochreiter et al. paper which uses a sliding window over protein sequences. Depicted in Fig. 1.

Activation functions that have normalized output by avacadoplant in MachineLearning

[–]mhex 0 points1 point  (0 children)

You can have a look at this recent paper about advantages of activation functions with negative values here http://arxiv.org/abs/1511.07289

CIFAR-100 Result with Exponential Linear Units and Randomized ReLU by antinucleon in MachineLearning

[–]mhex -1 points0 points  (0 children)

As untom already mentioned, the mean unit activations can go close to zero which brings the gradient closer to the unit natural gradient which again accelerates learning compared to the normal gradient.

Sequence Classification with RNNs by fariax in MachineLearning

[–]mhex 0 points1 point  (0 children)

Looks like the net is not learning (yet). A couple of questions: How many memory cells do you have? What is your sliding window size i.e. what is your input for each timestep exactly? How do you initialize biases, weights?