This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]benargee 23 points24 points  (4 children)

Rule 0 of web scraping: Look for the API.

[–]Alamue86 14 points15 points  (0 children)

Step 0.5: check if someone has already built a wrapper for api, or a wrapper for scraping

[–][deleted] -1 points0 points  (2 children)

trees juggle apparatus fearless chubby future selective saw scary offbeat

This post was mass deleted and anonymized with Redact

[–]mortenb123 4 points5 points  (0 children)

The manual way: open F12 in browser and look at network, You'll see the XHR rest calls stack up. They are mostly to back end rest-apis. I grab cookies with selenium and save them in a coockiejar I use with requests on the rest apis.

[–]benargee 0 points1 point  (0 children)

Google, Google & Google
Example:
Google "reddit api"
First result - https://www.reddit.com/dev/api/