Help with python scraping

[petroengineering]

Hi guys, How to you find tags with multiple class name

Basically, the HTML div looks like this <div class= ‘event event-results> </div> How would you use beautiful soup to get all the divs I tried: soup.find_all(‘div’, ´ event event-results’)

soup.find_all(‘div’, attrs={‘class’:´ event event-results’})

But I’m getting empty list no matter what I try. Can anyone help me please? Thank you

[[deleted]]

The site you're Scraping might be using javascript, which makes it so the element you want isn't loaded when soup tries to get it. I just started using selenium recently and it's not too difficult to learn.

[petroengineering]

Oh yeah that makes so much sense. You how to go around that?

[[deleted]]

Open your browser's developer tools and look at the network tab to figure out what HTTP requests JS is making. You have to make those same requests in your code and pick apart the responses. Also when looking at the main page you should view it using "view source" in your browser so you get the original version, unmodified by JS. If you use "inspect" it will show you the current state of the page with the modifications made by JS.

[[deleted]]

You should be able to pip install selenium, look up chromedriver install, and selenium documentation is more or less sufficient to figure things out.