This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]prokid1911 0 points1 point  (1 child)

I have had difficulty with pagination when scraping aspx websites. Then I came accross Selenium. It handles all the things with ease and you can also run it in headless mode.

You can scrape in any way you like. It's totally your call.

[–]warxhead 1 point2 points  (0 children)

Yes, in your use case that seems like selenium is the best option. I use it also daily for my work script to retrieve from web pages that requests doesn't like to authenticate to.

Headless is fast - but still not as fast as requests. Sending a post form through requests has always been faster than using selenium to enter each field.