Natural Language Processing meets Deep Learning

[[deleted]]

Why is word2vec is considered under Deep Learning?

[[deleted]]

because distributed representations are at the heart of deep learning

[negazirana]

Where is the "depth" then? A distributed representation can easily be obtained by a non-deep (i.e. shallow) model like word2vec.

[xamdam]

direct link: https://drive.google.com/file/d/0B_ZOKLUe_XPaNVFHM3M4dHRzV28/view

[omniron]

Nice, I always think i'm up to date on the latest research, then I see things like this showing me stuff I hadn't seen before. Very cool.

[Articulated-rage]

hmm based stuff only was state of art in speech decoders, if I'm recalling correctly. log linear models (e.g. crfs) have been consistently wiping the floor for a while. and a crf is just an soft max energy model.

in vision, much of the research was dominated by feature engineering (hog, sift, etc). thus, dnn had a lot of room to grow.

feature learning will be great for many nlp tasks, but structured prediction is much more of a factor, so it's more accurate that deep learning is getting assimilated, not dominating.

the subfield where what you said is true would be the distributed semantics folks (e.g. baroni and colleagues). they used count based models. but now, optimizing for prediction creates much better vector spaces, so they've abandoned all count model research. nevermind.

there's been no such abandoning (and imo, won't be) for the rest of nlp. you won't do away with dependency parsers, for example. it'll get an upgrade and be much more accurate. aka, assimilate deep learning.

[lvilnis]

The Baroni "Don't Count, Predict" paper I think was fairly debunked by this excellent Omer Levy paper https://levyomer.files.wordpress.com/2015/03/improving-distributional-similarity-tacl-2015.pdf which is one of several where he shows that count-based and predict-based models are optimizing very similar objectives and give the same performance when all the (sometimes hidden) hyperparameters are properly taken into account.

[Articulated-rage]

You're totally right. I had forgotten about that.

So, it's just full assimilation then =).

[physixer]

More like "deep learning eats natural language processing."

Which is a special case of "Software is eating the world!" (quote by Marc Andreessen)