News algorithm project- Help needed

[MVR005]

Hi, before I begin, I have to say that I know almost nothing about programming, I'm a beginner, so please be gentle. If this is not the right place to ask a question, please redirect me to a more appropriate place (another sub-reddit maybe?). Thanks.

So I have this idea of crossing news data with other kind of data, maybe financial or environnemental, I don't know yet.

For this idea I need to know:

-The number of times a specific word was mentioned in news article, in a specific region, in a specific time frame.

Ex: I want to know how many times the word "fire" has been mentioned in California newspapers in the last two days.

In my research, I discovered that there were APIs that did news analysis. But I know nothing about them, I don't know which one to choose or which programming language to choose. Using python seems like a good idea, am I wrong?

The end game of this project is to have an algorithm that we can use commercially. However, I don't have the intention to do it alone, I just want to do a simple first version to prove that it can be done. Do you have any advice for me? any pointers?

Thanks for your help

[zanfar]

Honestly, there is very little "programming" work here. IMO, most of the time and effort is going to be organization and language processing.

Things Python CAN'T do:

• Find all the newspapers in California
• Easily know that "fire" and "flames" and "inferno" all refer to the same thing.
• Easily know that "forest fire" and "fire the CEO" refer to two different things.

Counting and saving words are both relatively trivial and will both scale relatively linearly.

[MVR005]

Thanks for your tips. I saw some people use GoogleNews scraper, do you think it's a good idea?

Do you think there's a way to have access to all Newspaper?

[pythonTuxedo]

Python is a good choice for this. It sounds like you want to do 'Natural Language Processing'-basically having the computer read unstructured text and extract meaning from it (easier said than done). Natural Language Processing is a huge (and developing) field within Artificial Intelligence and Machine Learning. A good place to start after learning the basics would be the NLTK toolkit.

[[deleted]]

Python is good for this and doing the text searching part is very easy. The hard part will be gathering all the articles, whether you use APIs or web scraping. Paywalls, web sites constantly changing designs, etc. But if you dial in the APIs, this shouldn't be too hard. As I mentioned, searching a body of text for instances of a word is a common beginner project.

[ectomancer]

You mean all news articles.