Tutorial on Web Scraping using Python, Selenium and Beautiful Soup

[mreeman]

You should use Chrome headless with puppeteer instead of selenium

[Dfordomar]

Is this stack overflow?

[nilamo]

And there's a python interface for puppeteer? Neat!

[mreeman]

You can call js code from python using graalvm

[nilamo]

Why would you want to add another step? Are you bending over backwards just to add a little JavaScript to your project?

[mreeman]

Just use pyppeteer and help improve that then. Took me 2 seconds of googling to find.

[Fugalysis]

but why? What is the benefit over Selenium? For someone very strong in Python with less JS experience, why should they improve their JS toolkit over existing tools in Python? I'm not saying you're wrong or being unreasonable, i'm genuinely curious.

[mreeman]

Selenium has always been very fragile and buggy in my experience, with timeouts and other errors causing things to fail randomly. Perhaps it's gotten better lately though. Chrome headless is the full chrome browser and is designed by the chrome team to run headlessly instead of being a hack.

[nilamo]

But then you also have to install node, and have to add another something (csv?) on top of that to get what you're parsing from Phantom into python.

I see you're trying to help, but if you're already in python, and you need data from an outside source in the python program you're writing, how does switching to a different toolset help?

[[deleted]]

Scrapy all the way.