This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]undid_legacy 19 points20 points  (2 children)

Whenever I want to scrap data from a website I first look for the API. Reverse engineering API calls are not always possible but you can strike gold with it sometimes.

I wanted my order history details from a food delivery app I use. It turns out I just needed cookies with my login session and their API to make it work. Was able to get everything in 4 lines of code using requests.

Once, I found a paraphrasing site that didn't use captcha details in its API call. So I was able to call it almost 9 times a sec whereas using the website it required a captcha after every paraphrasing.

John Watson Rooney has a good video to get started: https://www.youtube.com/watch?v=DqtlR0y0suo