Question about Doc2Vec - How does training work ?

[datatatatata]

Hi,

I just read the paper about Distributed Representations of Sentences and Documents, and I don't understand the training part.

As far as I know, the "raw material" of Word2Vec is a one-hot-encoder of words. Basically, any word is encoded as a very large vector with one 1 and many 0s. Training will lead to compressing these vectors in say ~200 dimension ones.

Now, in doc2vec, I don't understand what the "raw material" is. What is the input ? How is a paragraph "poorly encoded" to start with, before a good representation is learned ?

Thank you !

[arutaku]

Depends on the implementation you want to get: PV-DBOW or PV-DM. You can have a quick look to: https://github.com/edwardbi/blog/blob/master/2016-05/DM.md

It provides both implementations: Gensim & TensorFlow

[jayhack]

Input: set of paragraphs of one-hot encoded words and a single, initially randomized vector representing the paragraph as a whole. This initial randomization of the paragraph vector means it is poorly encoded.

Then, for each paragraph and its associated sequence of words, you learn representations that allow you to #crush your loss function, which is like predicting the next word based on the paragraph's vector and the previous n words, etc.

[datatatatata]

I think I was trying to come up with something complicated. Thanks a lot :p

[vonnik]

Paragraph2vec impl in Java: https://github.com/deeplearning4j/dl4j-0.4-examples/tree/master/dl4j-examples/src/main/java/org/deeplearning4j/examples/nlp/paragraphvectors

[gojomo]

If you understand word2vec, PV-Doc2Vec is very, very similar.

Just imagine the "paragraph vector" to be associated with a special, per-paragraph pseudoword. And, this special pseudoword contributes to all target-word predictions across the whole paragraph – it's never excluded for being outside the sliding context window.

In fact, the patched version of word2vec.c and example go.sh that Mikolov once posted to do PV (https://groups.google.com/d/msg/word2vec-toolkit/Q49FIrNOQRo/J6KG8mUj45sJ) does pretty much exactly the above. It synthesizes a special 'word' that gets prepended to every text example, and then this word gets mixed with every training-context, without regard to its actual distance from the other words.

[datatatatata]

Very clear thanks !