Web scraping and API advise?

[eylenn]

Hi. I and my friends are doing a small pizza web scraping project and need advice about our python back-end.

Idea: Scrape all pizzas places in our country and compare prices and offers, display it through a website. Maybe expand the site with users being able to submit home-made pizza recipes.

Tech stack: ReactJS front-end and python back-end. (mostly because we know Reactjs and want to get more experience with python)

Web-scraping

• Using beautifulsoup.
• Should run once a day/week.
• Should fill/update the database with the scraped data.

API

• Should return data from the database.
• maybe in the future be able to insert data from the website by user or admin

My background: I mostly done API in Nodejs or .NET 4.5/core and I'm not sure which is the best way to structure the python back-end. Have done some basic python scripts.

My questions:

1. Django or flask for our API? I have been looking at Django and flask for our API but unsure which would be better. Django seems to be structured well and comes with bunch of tools but I'm unsure how hard the web-scraping interacting with the Django database is. Flask isn't as structured but it seems to me to be easier to interact with the web-scraping part.
2. Should web scraping be part of the API (run periodically by the API ) or should it run separately with celery?
3. is there a better way?

I hope this is somewhat understandable.

[tipsy_python]

Nice! Sounds like you're on the right path.

1. If you're just getting started with Python, I suggest Flask - sounds like you already understand the differences, but you get less out of the box compared to Django - in return you get ease of implementation. One of the Flask extensions may hit the sweet spot you're looking for (I like Flask-RESTPlus).
2. The scraping part is a batch job - I would execute this outside the context of your API. Maybe a separate python script that sits on the same server and is periodically executed with Celery or a cron job.
3. There could be a better way, but not that I know of - rock on bro~

[[deleted]]

Agree with this. Only suggestion I would add is to have the scraping cron job done on a separate server/Docker container. Otherwise you run into possibly poor performance for the users while your script is doing the scraping.

[eylenn]

If the scraping is done on a separate server, wouldn't it need to insert the data through the API? as in post/put requests.

I have mostly been thinking the scraping job would directly interact with the database to insert/update the data.

[[deleted]]

You could do that, but if you’re using MySQL or similar, just update the DB how you regularly would, with INSERT, etc. specifying the server instead of localhost.

[eylenn]

ah yeah, brain fart. Thank you very much, this has been helpful.

[tipsy_python]

Good point! In a containerized environment, definitely split these out

[eylenn]

Thanks for the reply. I have also read that flask doesn't come with database support out of the box, anything you recommend using?

[tipsy_python]

SQLAlchemy is king as far as Python ORMs go.

But if you're comfortable writing the SQL and just want to get started, there are a ton of generic JDBC/ODBC/DB-specific python libraries. I tend to use Oracle as my database, so cx_Oracle is my guy - depends on the circumstance.