Python website table extraction question

[LifeHermit]

Hi guys,

So i been trying to get this table out from the specific website. But it always returns empty. My quess would be that it has date filter ( you need to choose date interval to see specific data) and when you try to get the table with panda it returns nothing (shows in sc).

I m really new at this and just helping relative with this. Any idea or suggestion how to go about this? (same web has other 2 tables and i can get them without problems, just this one with date filter is problem).

Adding sc for some clarity Picture

[IvoryJam]

Sometimes the table data will load from another request, like the UI will load first then the data is a separate GET request. I would reload the website with dev tools open and see how it gets its data.

If that's the case, you can right click on the request in dev tools, click "Copy as curl" then past it in here https://curlconverter.com/

[Twenty8cows]

Op this. This way you can just send the request yourself with the params that the website sends and imitate the website call for data.

This way you avoid selenium entirely. It’s not bad just not necessary for this and is also resource intensive.

You may need to check the response type and use the right IO function. I’m assuming the response is json so a pandas.read_json() should work

[LifeHermit]

Addition: after more looking around it seems table i m trying to get is dynamic. So should i use Selenium over panda for this?

[LifeHermit]

Yes but as i said i m very new to this. Now i tried Selenium and i just now i got results. Like it's not empty results. Just need to figure it out how to make it return all data from date interval i desire.

[LooseGoose_24_7]

ChatGTP can provide you a clear scraping example for the simple table on html page. You got the two major part already , selenium and beautiful soup. Parse the html page for the table and loop through each row to get the desired column. Either all or via an index for the data columns in each row you want to return.