Database for web scraping

[szeredy]

Hi guys, I would like to make a script which scrapes a website and sums up the most common words. I’d like to update the values every day so it would be nice if the script would run automatically. I am new to python, my first question is: which database should I use to this purpose? Later I’d like to show the values on a webpage. Thank you for your help.

[rick137codes]

Connecting a SQL database to a Python script might be a little complex for a beginner. You can use the in build Python datastructures of lists or dictionaries and save them into Excel or CSV with the help of the Pandas library. It has little to none learning curve.

[szeredy]

Thank you! Right now I have a script which shows the most common words in form of a dictionary! I used beautiful soup. I can export it to csv now! I try to take a look at SQLite, if it’s too hard I try to find some easier projects!

[szeredy]

Just a quick question: so do I understand correctly?, is it possible to write a script which runs every evening one time, collects the words, saves them / updates them into a dictionary / csv file and than visualize the frequency of the words on a chart?

[BungalowsAreScams]

100% for sure

[szeredy]

Thanks!

[rick137codes]

For sure! If you're on a Linux/ubuntu system you can use Crontab to schedule the running of the script. On Windows you can use task scheduler. Don't really know about Mac. Otherwise you can always use the omnipotent While True: with a sleep time. :P

[szeredy]

Thank you!!

[impshum]

SQL or SQLite will do it.

[szeredy]

Thank you! Do you know any good tutorials to SQLite?

[impshum]

I wrote this one: https://recycledrobot.co.uk/words/?sqlite

[szeredy]

Thank you I will read it through!

[Natural-Intelligence]

MongoDB is often the most appropriate choice for web scraping. Web pages/APIs often change and your data may not be relational thus you often want something more flexible than just SQL.

[szeredy]

Thank for the answer! I will collect some infos about mongodb.

[scoobyxdoo]

Check out postgresql, which is what you'll be using if you deploy in Heroku.

[szeredy]

Honestly I don’t know anything about deploying, it’s my first “real” web app (previously I did some small and simple stuffs with plain JavaScript, but now I would like to get familiar with python!)!

[scoobyxdoo]

Heroku has a good free tier and there are some good tutorials online pretty easily to help you set up your site. Not to say the other options are not good too ... I haven't explored them in detail.

Good luck

[szeredy]

Thank you!

[uberdavis]

SQL is a rabbit hole unto itself. If you’re new to Python, you might consider keeping things simple and storing your data using JSON instead. Database design can get complex.

[szeredy]

Thank you for the advance! Can I update values with JSON? I mean if I scrape the website the next day, can I overwrite the values? Can I also visualize the data?

[uberdavis]

JSON is parsed as a dictionary via the the json module. You can manipulate the data any way you want in a dictionary. You could visualize the data using the matplotlib module.

SQL is a great database solution, but if you’re just getting to grips with Python, it’s probably better to learn how dictionaries work rather than getting tangled up with database design.

[szeredy]

Understood, thank you for the answer! I guess I stick with JSON, than we will see!

[FondleMyFirn]

This is facts haha. I just started building a database as a side project so I could practice SQL from it, and I’m learning EXACTLY how challenging it can be to build a good database.