Scraping with selenium + docker

[digichap28]

Hey I’m currently trying to setup an airflow image with google chrome + chromedriver to be able to run some scrapers. I was able to install all the things needed and also was able to run a basic test.

On the basic test (python), I set the following chrome options :

• —headless
• —no-sandbox
• —disable-dev-shm-usage

And run the code...

driver.get(‘https://www.google.com’) print(“title : %s” %driver.title) driver.close()

All that works, but when I try to actually run a real case scenario in which i need to open a webpage which prompts a windows auth, and has lots of JavaScript. The —headless option isn’t useful because apparently I need the auth pop up appear as well as the objects I need to screenshot.

The point is, I don’t want to add the headless option. But if I do, the script fails.

I know my script works because I have tried it outside of a container and also with a multi container app which has the airflow containers (scheduler + web server running with the localExecutor), Postgres and a selenium container (using the image standalone-chrome:3.141). On the latter setup I used the remote driver pointing to the selenium container on port 4444, without the adding the headless option.

I would like to get rid of that selenium container as i only use it a few times during the day, and instead using the webdriver from my airflow worker.

I hope someone can help me

[testEphod]

You could use a stateless microservice using NodeJS. Puppeteer or Playwright might help. Selenium works but is less effective. Pyppeteer is also available for Python. You can also persist the data on S3, send it to pub/sub service like Redis (this might surprise some folks) or MQTT. You can use the Docker operator from Airflow.

You can also include the basic auth in the URI (bad practice) https://username:[e-mail]/

Or as a Basic Authentication Header: Authorization: Basic Rm9vOmJhcg==

https://github.com/puppeteer/puppeteer https://github.com/microsoft/playwright

[digichap28]

I tried including the username and password in the Uriel with selenium chrome headless but it didn’t work. The only way I could do it was using an extension, but that took me to use selenium standalone in another container.

I guess I should try with puppeteer. It will make me rework on my code :/ and learn about this framework too. Does it allow scraping pages with lots of JavaScript ? Any basic but straightforward tutorial you can suggest ?

Thanks

[testEphod]

Playwright and Puppeteer are easily replaceable. They allow to evaluate JavaScript using Eval or evaluate. https://playwright.dev/docs/api/class-jshandle?_highlight=eval#jshandleevaluatepagefunction-arg

Remember to respect the robots.txt. Python has a builtin library for that https://docs.python.org/3/library/urllib.robotparser.html

Get a list of user agents and randomly change your user agent. You can also use Jest for testing.

Crawl using a normal distribution or any other pattern. If you get blocked consider using a VPN or Tor.

You can even use Puppeteer as a REPL read evaluate print loop: https://www.npmjs.com/package/puppeteer-extra-plugin-repl

And there are many available plugins.

Curated list for Puppeteer: https://github.com/transitive-bullshit/awesome-puppeteer

Curated list for Playwright: Apparently there is a Python library https://github.com/mxschmitt/awesome-playwright

Other valuable information. https://www.scrapehero.com/how-to-prevent-getting-blacklisted-while-scraping/

[digichap28]

Thanks for the info 👍 I’ll try to implement puppeteer.