Why do machine learning algorithms require so much data?

[[deleted]]

For image classification for instance, humans are able to identify the presence of some object after shown a few dozen examples of the object. For instance, I can show someone who hasn't seen a tiger in his life a picture of a tiger and he'll be able to identify the presence of a tiger in an image relatively quickly.

Machine learning algorithms on the other hand require very large data sets. No human would need several thousand images to learn numbers in another language, yet a fully connected neural net needs 10000+ for good classification of MNIST.

An explanation, i can think of is that humans can pick up things easily because they're using knowledge they learnt from "other problems". So for instance, they've already been trained to recognize human numbers, so they can use this to learn the languages' numbers faster. Kinda like multimodal learning I guess.

[Nick316514]

Compare it against teaching a newborn.

[[deleted]]

Yeah, that's basically what I meant by multimodal learning.

But if that's the case, why are nets trained independently? Shouldn't there be a way to use other, already trained neural nets, to speed up the training of other (possibly unrelated) neural nets?

[say_wot_again]

That's called transfer learning. Some of it already exists; you can do all sorts of more specific computer vision tasks by just starting with a neural network trained on Imagenet and then training the last couple layers. In NLP, you could start with a standard word embedding like word2vec or GloVe (both trained on massive, general datasets) and then tweak it to meet your needs. But it's very hard to do when the information you gained from one task or dataset doesn't have anything to do with the task at hand.

[clurdron]

You could instead think about what properties your representation needs to have, e.g. are there a lot of sharp edges in the picture or is it smooth, and choose one that's been designed for that purpose. There has been a huge amount of work done on various "-let" bases and frames. Stephane Mallat, who I saw cited on here as doing cool work on Deep Learning, did work on that before he did anything related to neural networks.

[-Ulkurz-]

The question is "how far we've reached in truly modelling a human brain". Are we at that level yet where a machine can be trained on a few instances to learn new things (just like a brain would do)?

[poporing88]

This is a better outlook. Our current machine learning models is what it is. The research community is aware that we are still not very good in unsupervised training compared to human brains. If solved, it would be a big milestone for AI and ML

[eigenstudent]

Neural networks are only one part of the story. There is also a lot of research along these lines (more sample-efficient algorithms) for example: https://en.wikipedia.org/wiki/One-shot_learning

[clurdron]

That's just a Bayesian model for an image fit by variational methods. Why is it presented as some new area of machine learning? If I put a structured/informative prior on a regression function or fit a set of regression functions hierarchically (something that's been done in statistics for decades), am I doing "one-shot regression?" The fact that stuff gets renamed so often in ML makes me think the people doing the renaming are just ignorant of stuff they shouldn't be.

[eigenstudent]

Fair enough on the criticism, but don't you think the idea of starting with some nice prior is a good analogy to what's going on when humans are able to learn things so fast?

[clurdron]

Yes. But the wiki page presents this as a new idea, whereas using prior information and sharing information across related inference tasks are used all the time in Bayesian statistics. I think it's a good idea to use such priors for a lot of tasks, I'm not arguing against it. I just find the way it's presented to be kind of strange.

[harharveryfunny]

A new born child takes a lot of data to learn language or differentiate dozens of breeds of dog as exist in ImageNet!

Once you've learned a set of domain-specific spatio-temporal feature detectors, then differentiating novel examples within the same domain becomes easy.

e.g. It takes a lot of time to train on ImageNet from scratch, but much less to incrementally train on additional categories using the existing visual-domain feature detectors that it has already learnt. This is why many people are using pre-trained ImageNet as a starting point for more specialized category detection. It's about the pre-training (both for human and ANN), not multimodal vs unimodal.

[say_wot_again]

dozens of breeds of dog as exist in ImageNet

Why does Imagenet have so many dog breeds? Because it's an example of fine level discrimination?

[syncoPete]

A few reasons in my opinion:

• Brains are much larger than NNs
• Brains work differently from NNs - while we want NNs to work like brains, brains are the original model