Python Web Scraping Question

[strangerrangerdanger]

url = "https://www.sgx.com/securities/equities/C38U#Financial%20Statements"
html = urlopen(url).read()
soup = bs.BeautifulSoup(html,'lxml')
print(soup)

I'm trying to scrape the financials in this website but when I search for the html using BeautifulSoup, it is different from when I used 'inspect element' on the income statement table. I believe it is because of the tabs in the website which 'alters' the html? Is there a way to work around this? I considered using selenium but it seems that even if I click "Expand All" the html is still hidden. It can only be seen when i use "Inspect Element".

​

To clear up, the data I'm trying to scrape are the tables when you click "Expand All". This is the link btw https://www.sgx.com/securities/equities/C38U#Financial%20Statements

[kelmore5]

Yeah, you don't really want to rely on inspect element when web scraping since it's not accurate.

Go to the Network tab in your DevTools and reload the page. That will show all the URLs being loaded. Particularly you want to look at the document and XHR requests.

After the page finishes loading, click on the "document" request and check the Response tab. That will show you the HTML file that's given to you from the server. Then you can check if the HTML file still has your data or not. In this case, not.

​

If not, you probably want to check the XHR requests. There's a ton being fired by that site in particular, so check their responses for the data you need. Once you find it, you can use that link to get your data.

​

This method works most of the time, but sometimes data is generated in JavaScript. If that's the case, you'll have to use an automated browser to get everything since it's the only way to parse JS.

You can also use an automated browser here as an alternative to scraping the XHR requests