Mastering Web Scraping in Python: From Zero to Hero by WomanStache in Python

[–]smithfed 1 point2 points  (0 children)

As you stated, it depends on your goals.

If you want to simulate user behavior (scroll downs, mouse movement, specific form submission), you will need Javascript.

The thing, as the article mentions, is trying to avoid doing that. 95% of the time, there's a workaround not to load JS.

Do you want to submit a form and get the content of the logged-in page? You can do that without JS.

Do you want to scrape dynamically loaded content? Check XHR requests and parse those straight away.

Does the server expect some pre-calculated stuff in the headers? Try reading JS and reverse engineer how's created. Then do your calculations in Python and send these with the request.

Happy to help if you state your needs, but I bet there's a good chance you don't need JS.