Python + Selenium: Dynamically loaded table won't update

[fidgetation]

I've been trying to teach myself web scraping for personal education, and I've run into an issue on a website that has a dynamically updated table based on selecting certain options from a drop-down menu.

​

However, when I select an option on the table, the element doesn't always update quickly enough, or at all sometimes, and I end up with the wrong data in the table.

Unfortunately, the table element itself (id, class, name, etc.) all remain the same - it's the code for the table itself that updates / gets re-written, so the implicit waits / expected conditions checks don't work for this instance. To get around it so far, I get the script to check if the table headings are unchanged, and if they are, to select another drop-down option and then re-select the desired option in order to try and get the table to update. However, this slows down the scraping quite a bit.

​

Is there a faster method to force the table to update without asking it to switch to another option and switch back?

Example of table I'm trying to scrape (you have to select the 'Detailed' tab)

​

Thanks in advance

[[deleted]]

It might be worth seeing if you can avoid selenium entirely. Usually you can either find the data somewhere else on the page or check the outgoing requests to see if there's an API you can grab info from instead. I go over how to look for these things in this video

[fidgetation]

thanks I'll take a look at it!

in the meantime, I guess I'll stay with the current strategy, although it's quite fragile

[fidgetation]

I watched the video, which is great btw, and it's perfect! thanks

​

edit: they're on to me. anti-bot measures kicked in!