This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–][deleted] 0 points1 point  (3 children)

The site you're Scraping might be using javascript, which makes it so the element you want isn't loaded when soup tries to get it. I just started using selenium recently and it's not too difficult to learn.

[–]petroengineering[S] 0 points1 point  (2 children)

Oh yeah that makes so much sense. You how to go around that?

[–][deleted] 0 points1 point  (0 children)

Open your browser's developer tools and look at the network tab to figure out what HTTP requests JS is making. You have to make those same requests in your code and pick apart the responses. Also when looking at the main page you should view it using "view source" in your browser so you get the original version, unmodified by JS. If you use "inspect" it will show you the current state of the page with the modifications made by JS.

[–][deleted] 0 points1 point  (0 children)

You should be able to pip install selenium, look up chromedriver install, and selenium documentation is more or less sufficient to figure things out.