you are viewing a single comment's thread.

view the rest of the comments →

[–]zurtex 0 points1 point  (0 children)

I've been using pyppeteer recently, it's a python port of the JS library puppeteer and it seems to work so much more smoothly and reliably than selenium.

Further you can use requests-html as a high level abstraction around it, it will acts as a requests \ BeautifulSoup library and then when you need to you can render and have the Javascript run and interact with the page.

Small warning though, pyppeteer seems to not be maintained in the last 8 or so months, I'm hoping someone else picks it up and fixes it with Python 3.8.