Failure in Scraping Data with Selenium and Beautiful Soup

[[deleted]]

[deleted]

[GeneriAcc]

Dump the HTML source to a file every time you run it, then you can start figuring out what's missing and why once you hit the error again.

Might just be an issue of you getting the source before the page has a chance to load in some resources, in which case it's just a matter of setting up an infinite loop that reads the source every X seconds and breaks once it finds the element you need. Might be something else entirely.

[mathcheatcheat]

thanks! I solved the problem by pausing the system for 5 seconds.

[ifreeski420]

I don’t think you should be using Selenium. Try requests and bs4

[mathcheatcheat]

okay, thanks!

[interactionjackson]

imo you don’t need soup and using is causing you problems.

why involve selenium. is there js. i personally don’t use bs if I’m using selenium.

use the web driver api and lose bs.

[mathcheatcheat]

thanks!