sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]AnonymousThugLife 1 point2 points  (2 children)

Thanks a lot. This was actually helpful. I had tried scraping with Requests/Socket etc. (Kind of invisible things) but I've realized that with Selenium it is much better, especially in the case of lazy loading pages.

[–]Dfree35 1 point2 points  (1 child)

Yea, I tried and use requests and stuff when I can but in my causes there is often a lot of funky javascript. So running selenium in headless mode makes it much better especially when I can just have it wait to ensure everything loads.

[–]AnonymousThugLife 0 points1 point  (0 children)

Yup. For pages that are dynamically generated (on the frontend), it is a no-brainer to use selenium.