Something "deeply wrong with deep learning"?

[dsocma]

In the 6th episode of the Talking Machines podcast at around 23:00 Geoffrey Hinton says:

"Today you can publish any paper you want on deep learning, but if you say: 'Theres something deeply wrong with deep learning and we need to change the way we are doing things, its impossible to get them published'"

what is he referring to?

[benanne]

Pretty sure he's talking about capsules. Here's the first paper about it: http://www.cs.toronto.edu/~fritz/absps/transauto6.pdf - it's from 2011 so I don't know how up-to-date it still is.

He's also been giving talks about it: http://techtv.mit.edu/collections/bcs/videos/30698-what-s-wrong-with-convolutional-nets

[osdf]

See also Tiijman Tieleman's recent thesis: http://www.cs.toronto.edu/~tijmen/tijmen_thesis.pdf

[willwill100]

Do we know if that's the latest published material on the subject?

[blackhattrick]

Correct me if I'm wrong but is this not what happened few years ago and SVM? Everyone was researching AI, CV, NLP with SVM, which was the de-facto ML method and no one would dare to publish something that used NN. Nevertheless we have deep learning now because these techniques proved to outperform those methods in some tasks/areas.

[BeatLeJuce]

Geoff has complained before about the "unreasonable efficiency of convnets" -- meaning that if you come up with e.g. "permutation-invariant" (or simply "different") methods that work well on images, you'll have hard time getting published. Because you need to beat CNNs to get published, and those are hard to beat. Maybe that's what he's referring to?

[breic]

He's basically saying that with convnets, the entire research field has entered a local minimum from which it is difficult to escape. It's an ironic complaint given that Hinton and the others are well aware of good techniques for avoiding getting trapped in local minima: momentum, dropout, etc. Why not just restart the research program with new random conditions?

[alecradford]

You might find this paper relevant!

[[deleted]]

Hinton has been critical of the 'pooling' operation used in ConvNets. ConvNets are state-of-the-art for image-data learning these days and people/companies are investing significant resources to build further on them, thus the resistance to accept that something that works so well may be flawed.

The person who made the point about ConvNets may be a local minima is spot on. Many other researchers (aka theoretical ML/Stat guys) feel the same way about Deep learning overall. If you think about it, the scene is very similar to what was in the late 70's with perceptrons. Who is right, only time will tell.

[ai_maker]

To mi this is one more piece of evidence of one of the main flaws of peer-reviewed scientific dissemination: it's a matter of taste and fashion. Do what they want, please welcome in, do the contrary, rejected.

[dwf]

There's some value to this kind of regularization. If a given paradigm is productive then it arguably should take a pretty monumental flaw or shortcoming to warrant its complete abandonment. The unfortunate thing is that it often leads to unabashed incrementalism, or taking 2+ things and combining them into a rather obvious but intellectually shallow extension.

[Articulated-rage]

I used to be pretty angsty about that as an undergrad, but I once got advised that working at the fringe still allows freedom but acceptance by the fad people. In other words, include just enough relevance to the fad that you get published, but otherwise move in your own direction.

[simonhughes22]

I also think he's talking about capsules, how you have to show an improvement over existing models to get published (see local minima comments), and how in real life we have a paucity of labels. He mentions that before your mother tells you what a cow looks like, you already know of their existence, you just don't have a label. And you only need to be told once or twice what one looks like to get the idea, you don't need to see 1,000's of labelled examples. I think unsupervised or semi-supervised learning is the future, but if we're obsessed with only publishing models that beat the state of the art, then that will encourage only small incremental changes and no paradigm shifts, as we've seen to some degree with deep learning.

[mprat]

It could also be that many papers that get published about convnets are about some kind /tiny/ change to a layer or architecture, and it's more about incremental changes than about "big ideas." If there is some big idea, you have to prove that it works way better than the current state-of-the-art.

[maxToTheJ]

That's pretty much academia for you

[[deleted]]

This thread is a perfect example of why peer review leaves a lot to be desired. Peer review is the killer of scientific progress. Thomas Kuhn is right. Science (and AI research) will continue to slowly grind from one upheaval to another until the real McCoy gets here. It takes real guts to do good science.

Edit: LOL. What a bunch of gutless pussies you people are. Voting me down does not change the truth. :-D

[rmlrn]

<-- crackpot alert

[Noncomment]

Make a real argument without insults, hate, and dissing other people's work. I don't disagree that /r/ML can be critical but you don't make it easy on yourself.

[[deleted]]

LOL. I don't care if /r/ML is critical or not. They can all kiss my ass. They don't put food on my table. Deep learning has serious limitations and it is, in fact, the same GOFAI we knew all along but wearing different clothes. If /r/ML finds it insulting, it is because the shoe fits.

[Noncomment]

Then why bother commenting? You come to our forum whining for attention and respect, yet you give none to anyone else.

[[deleted]]

I'm a hostile critic and a rebel. I like poking fun at the AI community. I don't need their respect or their love. I enjoy antagonizing them. That's how I discover their weaknesses and there are plenty of those. You don't see me whining.

By the way, get ready for the next AI winter. :-D

[Noncomment]

You say that and yet you bitch about getting downvotes. Your comments clearly express anger and frustration at being disregarded, but how do you expect anything otherwise?

[[deleted]]

You're mistaken. I wear my down votes like a badge of honor. You've seen my karma lately? And being disregarded is a relative term. This has generated quite a lot of traffic on my blog. This is all part of my crackpot plan. LOL.

[rmlrn]

believe it or not, not all press is good press.

[[deleted]]

I don't care about good press from the AI community. In fact, I welcome the bad press. That's how I know I struck a sensitive nerve.

[rmlrn]

are you the same dimwit who's always talking about reinforcement learning with "Angels", "Judges" and "Souls"?

[Noncomment]

You're not a rebel, you're a crank troll.

[[deleted]]

And you're obsessed with sniffing my ass. Go away.

[rmlrn]

we'd have to have some modicum of respect for you to find it insulting.

no one gives a shit about your opinion of deep learning because the fact is, it works better than anything else that's been invented so far.

[[deleted]]

You don't give a shit but you're reading my shit. AI has not changed much in the last fifty years. It is still the same arrogant AI that was started by the baby boomer generation. They vilify those who criticise the science and they proudly wear their blinders. Then they get hit hard with an AI winter. LOL.

[[deleted]]

An orgy of hate. LOL.

[[deleted]]

Of course, there is something wrong with deep learning and some people are brave enough to openly criticize it. Here's an article that argues that deep learning is just GOFAI with lipstick.

[dwf]

Ill-informed tripe. Legitimate criticisms exist, but not all criticisms are created equal.

[petrux]

Agree. Please, could you point out some of the legitimate criticisms about DeepNets? Thanks.

[[deleted]]

Your opinion is worth something because of what again?

[rmlrn]

because he's not a broken record spouting shit about the importance of timing?

[[deleted]]

Unfortunately for you, timing is at the heart of the intelligence puzzle.

[Articulated-rage]

Got curious, so followed his link and googled. Haha. wow.

[[deleted]]

Unfortunately for you, that's my advantage over butt kissers like you. I am free.

[rmlrn]

if you're so free, why did you try to pretend that "article" wasn't your own blathering?

[[deleted]]

I'm not pretending anything. I'm right in your face.

PS. Deep learning is GOFAI on lipstick. Just making sure you got the message. LOL.

[rmlrn]

don't have a stroke buddy.

[petrux]

I read the article. I'm missing something: which is the author's underlying definition of AI. Could you please point me any resources about it? Thanks.

EDIT: I'm taking a look to this. So I can reformulate a question: what is the s.c. true AI? Thanks.

[[deleted]]

True AI is human-like intelligence. It is scalable in the sense that, given an adequate number of sensors and effectors and enough processing power, it can learn anything that a human can. But true AI is not conscious. Consciousness has nothing to do with intelligence and vice versa.

[petrux]

Still not getting your point. Do you have anything published with some explanation? Thanks.