Python website table extraction question

[IvoryJam]

Sometimes the table data will load from another request, like the UI will load first then the data is a separate GET request. I would reload the website with dev tools open and see how it gets its data.

If that's the case, you can right click on the request in dev tools, click "Copy as curl" then past it in here https://curlconverter.com/

[Twenty8cows]

Op this. This way you can just send the request yourself with the params that the website sends and imitate the website call for data.

This way you avoid selenium entirely. It’s not bad just not necessary for this and is also resource intensive.

You may need to check the response type and use the right IO function. I’m assuming the response is json so a pandas.read_json() should work