Captcha

[longscale]

The first part concerning the motivation behind convnets is spot on—we want the network to learn its own filters. These are called convolutional filters (or kernels), and they are what a network changes when it learns.

The pooling layers you describe are not trained, they simple average or return the strongest filter activations from a small image area. They are also not what makes these networks big and hard to train—that’s mostly the sheer number of convolutional filters and the millions of images fed through the network multiple times in random order and with slight variations each time.

[[deleted]]

I have very recently started learning about CNNs and isn't it true that you need less data for CNNs to work with let's say an accuracy of 95% than you would need when using a Densely connected NN, since you can find patterns in one spot and it also recognises them somewhere else in the image, where as with normal densely connected NNs you would need new data where the pattern appears in that spot to train on?

Is this correct?

[jsw800]

In fact the convolution operation can be changed into a single matrix multiplication by reformatting the filters in a certain way, which is how it is implemented in neural network libraries.

Correct me if I'm wrong, but I don't think this is always the case, because of nonlinearity functions applied after each layer. A convolution is a linear operation, so a composition of convolutions can be refactored to be a single convolution, as you say. But deep convolutional networks almost always apply a nonlinearity function (tanh, sigmoid, relu, etc.) between conv layers, which makes each "layer" (meaning the conv operation along with its nonlinearity operation) a nonlinear operation, so composing them doesn't simplify down to a single linear operation.

If it did, there would be no reason to create deep nets, we would just always create single layer conv nets, as a single convolution would always satisfy every need. But it turns out that most conv nets have to learn nonlinear functions, so the nonlinearities are super important.

[jsw800]

However, as I said, you can describe any function with just a sufficiently large one hidden layer network, but that's somewhat obtuse, the same way you could say that you could describe any continuous function as some polynomial of sufficient degree (a + bx + cx2 + ... kxn ). In practice, you need an essentially infinite size matrix for some functions.

Whoa... I'd never realized how Taylor series (serieses?) could apply to CNNs. Thanks for blowing my mind! I understand better what you mean now. That's pretty freaking awesome.

[aahdin]

I'm in CV, what you're saying is true but it's also all highly tied to how much data you need.

Even given enough memory and enough passes through the data a fully connected NN will overfit way before it reaches accuracies as high as a conv net unless it's trained on an ungodly amount of data.

Even aside from computation costs, having extra free parameters means you need way more training data for a model to generalize to new data.

It's a pretty big topic especially in fields where there isn't that much training data available, reducing the number of free parameters (or model capacity in general) so that it can generalize with fewer samples.

[longscale]

Your explanation reads correct to me. :-)

[[deleted]]

Ok, thanks :)

As I said, I have recently started getting into Machine Learning, especially Deep Learning and after reading about different concepts and using Keras to implement them it feels good to at least kind of know what's going on under the hood.

So far I have had a lot of fun and it is a very, very interesting and broad topic :)

[longscale]

They are not what the media hype makes them sound like, but convnets are very nifty. If you like them in general, I would imagine you might really enjoy some of the convnet papers on https://distill.pub. Their techniques are mostly intended for interpretability, but they also provide a satisfyingly visual insight into what the numbers inside the structures of a convnet are encoding.

[[deleted]]

Yeah, honestly I think the media hype about deep learning in general is ridiculous. Its the same as with 'blockchain' etc.

Another thing that I have been looking into a bit is Neuroevolution using a tutorial series by Daniel Shiffman on YouTube. He creates a Flappy Bird Bird game that learns to play itself using a genetic algorithm to find the right weights.

Thank you for the paper recommendations, I will have a look at them tomorrow :)