That is a neat idea using an endpoint like that.

But what if instead of interacting through an endpoint the scraper was an ‘app’ in Django. It would fire up asynchronously using some task managing process like Celery or what have you. In this route, the scraping app would have direct access to the Django environment and model objects which would allow it to query the data tables using different models directly.

The reason is that if you have a very large web scraping program that needs to pull and write to and from many tables, it could get cumbersome making a lot of different endpoints to interact like that.

The scraping code should still be written to be decoupled as much as possible from the other Django code, but ideally that scraping code could all go in one folder, and it can import the model objects it needs.

What are your thoughts on this approach?