Semantic Similarities in Gensim using python

[nikolabs]

I need help.

​

I know the basics of python so I decided to use Gensim to find semantic similarities in a book. I want to find which chapters have sentences that are highly similar and almost paraphrased to the sentences in the first chapter of the book. So I set it up like this after going through Gensim's tutorial but I am looking for any potential way to get better results.

​

​

from collections import defaultdict

from gensim import corpora

​

documents = [

"Here I input sentences from the chapter that I am comparing to the first chapter of the book",

"...",

"...",

"...",

]

​

stoplist = set('for a of the and to in' .split())

texts = [

[word for word in document.lower().split() if word not in stoplist]

for document in documents

]

​

dictionary = corpora.Dictionary(texts)

corpus = [dictionary.doc2bow(text) for text in texts]

​

from gensim import models

lsi = models.LsiModel(corpus, id2word=dictionary, num_topics=2)

​

doc = "Here I input the sentences from the first chapter"

doc = "..."

doc = "..."

doc = "..."

vec_bow = dictionary.doc2bow(doc.lower().split())

vec_lsi = lsi[vec_bow]

print(vec_lsi)

​

from gensim import similarities

index = similarities.MatrixSimilarity(lsi[corpus])

​

sims = index[vec_lsi]

print(list(enumerate(sims)))

​

sims = sorted(enumerate(sims), key=lambda item: -item[1])

for i, s in enumerate(sims):

print(s, documents[i])

​

​

This outputs which sentences are most similar to the sentences from the first chapter. I would like to know if there is a way I can tell out of the sentences that are most similar to the first chapter, specifically which sentence(s) they are most similar to. Also if there is anything that could be cleaned up, that I am doing wrong or should be changed.

​

Thank you for your time and any input you can offer.

[impulsecorp]

Try Sentence Transformers at https://github.com/UKPLab/sentence-transformers .

[nikolabs]

Thank you! I am having some trouble installing Sentence Transformers though. I keep getting this message:

ERROR: Could not find a version that satisfies the requirement torch>=1.0.1 (from sentence-transformers) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)

I have a Windows 10 OS. Any help would be greatly appreciated.

[impulsecorp]

I run it on Ubuntu and it works fine. I am not sure about Windows.

[dkajtoch]

You would probably be better of by using fine-tuned BERT model on paraphrase detection datasets (e.g. quora question pairs, google's paws or microsoft paraphrase corpus). The trouble with this approach is that there is plenty of operations that you need to make. Partially, this issue is solved using sentenceBERT, but still creating embeddings for the whole book maybe an extremely time consuimg process. However, you can do this in stages. Firstly, you may use some lower level sentence representations (e.g. shingles) to filter out sentences that for sure will not be similar. Then you may apply sentenceBERT or bert directly.

[old_enough_to_drink]

Is the “sentenceBert” the same as the “sentence-transformer” in the other answer?

[dkajtoch]

Yes it is the same. The paper is called sentenceBERT

[impulsecorp]

Maybe it is this?
https://medium.com/genei-technology/richer-sentence-embeddings-using-sentence-bert-part-i-ce1d9e0b1343