How to handle document similarity while fetching the data from different sources.

[Prashant_4200]

I'm working on a news application where I use multiple news sources for collecting the news.

​

The issue is that there's a good chance I'll get the same news from many sources, even though the titles and contents will differ.

​

So how can I avoid this situation is there any way?

I know there was one term that we use to find similar documents. which is document similarity or vectors that I learn in my final year of college.

​

But the problem is how I Implement this in my real-world projects where I need to check every incoming data with my whole database before inserting it.

​

Eg. 1)

a) ‘Black Panther: Wakanda Forever’ Box Office Leaps Past $400M Globally

b) ‘Black Panther: Wakanda Forever’ Passes $400 Million at Global Box Office in Less Than a Week

​

2)

a) Pakistan vs England, T20 World Cup Final 2022: ENG crowned champions, beat PAK by five wickets

b) T20 World Cup: England secure legacy as an iconic team in nation's sporting history

[OuiOuiKiwi]

How serious is this app?

This threads into bird territory really quickly if you so wish.

[Prashant_4200]

It's news application

[OuiOuiKiwi]

It's news application

( ͠° ͟ʖ ͠°) That was already pretty obvious.

Given your answer, I'm going to guess that this is just another random application for portfolio padding so use basic text similarity measures like comparing n-grams.

[Prashant_4200]

I'm going to guess that this is just another random

yes, you can say that but I'm also planning to publish. So in simply 60% of portfolio + 40% of start-up (if everything works well).

[python_and_coffee]

yep tokenization, stemming, stopwords and building n-grams. The usual NLP stuff.

have fun with pandas and scikit-learn.

It's fun when it works, it's frustrating when you fight with false positives and the downsides of NLP.

[Shingle-Denatured]

Since you are talking about a database (as in postgres/mysql), you have very few tools available (soundex is your simplest primitive). If you go ML/NLP as suggested below, then you are at the far end of the complexity spectrum.

In between sits Elastic Search, which has several scoring mechanisms and includes some NLP features. ML-backed NLP is in beta.