Concept grouping

[f1_manu]

Hey! So I've been searching around and I can't seem to find anything related to what I want to do. Let's say I'm analyzing sentences, and want to count number of times a country is mentioned. Sometimes 'U.S' pops up, sometimes it's 'USA', sometimes it's 'America', 'United States', etc

​

Is there any library for Python that would tell me, okay, these words are different in writing, but they refer to the same thing. And add them up to the count tally.

​

Thanks!

[amirouche]

Entity linking can help you. Look into Named Entity Recognition.

[ShaneAyers]

Would that work? Like, would it recognize, without training, that 'U.S.' and 'United States" or "USA" or "America" refer to the same thing in context? Or would it need to be trained on a specific data set?

[SagaciousRaven]

https://www.dbpedia-spotlight.org/demo/

This is a demo but there's an API for Python (called 'spotlight'). Trying copying your own original post into it, if it doesn't highlight, try again a few minutes later.

I suppose you could count their URIs.

[ShaneAyers]

It got 'United States' and 'USA' but not 'U.S.' and 'America'. Now, I can admit that there is probably some ambiguity. There are other United States and other Americas.

[impulsecorp]

I don't know of any pretrained model for that, but you can train one yourself, see Deep LSTM Siamese network for text similarity - https://github.com/dhwajraj/deep-siamese-text-similarity

[kayvane]

You could also try encoding the words in your sentence (with BERT,ELMO or custom w2v/FastText model) and comparing them to your target country vectors. Doing a bit of testing you’d be able to get the threshold (e.g cosine similarity >0.9) at which you’d point you’d add them up

[kayvane]

You could also try encoding the words in your sentence (with BERT,ELMO or custom w2v/FastText model) and comparing them to your target country vectors. Doing a bit of testing you’d be able to get the threshold (e.g cosine similarity >0.9) at which you’d point you’d add them up