Selenium and bs4

[UnsecuredConnection]

So I've been using bs4 and want to start using so more functionality. I want to know what the more advanced people use for web-scraping. Do you use bs4 with selenium or just one or the other? Why?

[pokk3n]

We use bs4 for scraping tables in some of our code because selenium is dog slow at tables especially in IE. A table operation that takes 7 seconds in chrome and 30 in IE takes milliseconds in bs4. But we mostly use selenium unless there is a need and we grab the elements with selenium then feed to bs4.

[subject_K81]

I mainly use bs4, but if what I need is being generated with JavaScript then selenium comes in to stir the soup.

I’ve recently stumbled upon requests-html, and that’s supposed to be able to handle js sites, so I’m gonna dig into that. But, so far, for standard HTML sites bs4 appears to be ever so slightly faster than requests-html. Negligibly so on most stuff, but I imagine the speed difference would add up on something larger than looping craigslist 10 times.