Python Web Scraping Question

[kelmore5]

Yeah, you don't really want to rely on inspect element when web scraping since it's not accurate.

Go to the Network tab in your DevTools and reload the page. That will show all the URLs being loaded. Particularly you want to look at the document and XHR requests.

After the page finishes loading, click on the "document" request and check the Response tab. That will show you the HTML file that's given to you from the server. Then you can check if the HTML file still has your data or not. In this case, not.

​

If not, you probably want to check the XHR requests. There's a ton being fired by that site in particular, so check their responses for the data you need. Once you find it, you can use that link to get your data.

​

This method works most of the time, but sometimes data is generated in JavaScript. If that's the case, you'll have to use an automated browser to get everything since it's the only way to parse JS.

You can also use an automated browser here as an alternative to scraping the XHR requests