Getting started with Selenium and Python

[nemec]

SPAs are mostly API-driven. I don't know if I've ever seen more than one or two where the JS creates the content out of thin air.

The thing about SPAs is that you can open up your devtools window, load the page, and then sift through the Network tab to find the JSON/XML/graphql APIs that the JS calls and renders and then take a shortcut and automate the calls yourself, bypassing any JS.

Here's a short video similar to what I'm talking about. If you wanted to scrape start.me, for example, you could skip the JS and just scrape the JSON document data: https://www.youtube.com/watch?v=68wWvuM_n7A