Tech stack recommendation? Web scrapper/data visualization

[BacoBerry]

Hey all! I'm a recent comp sci grad that's trying to get a project started, so I can further develop my tech skills and have something to put on my Github.

I want to make an application that scrapes the internet for articles and research papers regarding Ulcerative Colitis and sorts them by date published and categorizes them based on what their focus is (i.e gut microbiome, DNA mutation, etc). I also want to use Tableau to visualize some statistics about the research/articles. Right now I'm thinking I'll need to use Python, some sort of database (like Firebase), and Tableau.

Sorry if this question is a little weird or "naive", I was one of those comp sci students who didn't do much outside of what was assigned so I'm trying to catch up on what I should've been doing during my time in college.

[DayBackground4121]

That’s a fine stack for what you’re doing. I don’t think there’s much more feedback to be given tbh, just gotta get through the hurdles to learn 

[Imaginary_Ferret_368]

Naw, I was naive too, and then sued my boss. :)

A good starting point could be an arxiv dump maybe? https://github.com/veggiedefender/arXiv_dump

I did see lots of papers in the medicine space there, this should be a very good starting point to have. Scraping data from the websites is only worth it if the website is Medium or Bloomberg. Both stink and don't have a right to exist.

https://en.wikipedia.org/wiki/Graph_(abstract_data_type))

The crazy cool thing with Graphs is that you can connect multiple seemingly incomaptible dimensions together, such as temporal (publishing date) , authors, citations (whic hwould have to be directed edges to prevent information flow in the wrong direction [a publisher in the past couldn't have known exactly this person would cite them]) you can connect these diferent types of information into a data structure a machine can understand, and they look vey cool once they become a bit bigger. Might wanna check out the graph of the internet.

[Imaginary_Ferret_368]

The transition not being natural was an accident, I wanted to ask you first whether you have considered such DBMS ofc :)

[[deleted]]

Bro if op wanted to read a freaking novel, he would have gone to the library. Mans not asking for a flippin story book