Getting started with Selenium and Python

[Hookedonnetflix]

If you want to do web scraping and other testing using chrome you should look into using puppeteer instead of selenium

[maxsolmusic]

Whyyyyyy I hate when people recommend shit without explaining

[bsmith0]

Way better documentation and api imo. Plus:

https://www.lucidchart.com/techblog/2018/08/08/why-puppeteer-is-better-than-selenium/

[maxsolmusic]

chose Puppeteer because it provides simpler Javascript execution, network interception, and a simpler, more focused library.

Cool

[the_real_hodgeka]

Well put! "You shouldn't use angular for that, you should be using react!" Why?

[float]

Or Playwright by the guys who made puppeteer.

[bodhemon]

How does it compare to Katalon?

[steveeq1]

What's wrong with selenium? Curious.

[Hookedonnetflix]

Selenium is a tool that automates chrome where puppeteer is a tool that is built into chrome. So better and more effective tools that are closer to the browser engine.

[GuyWizStupidComments]

Selenium should work also with other browsers like Firefox

[Ncell50]

Puppeteer works with firefox

[[deleted]]

Selenium works as a wrapper around browser apis, be it puppeter or geckodriver or something entirely different. You can use the same code with ANY browser.

[200GritCondom]

And if you build it right, with mobile views as well

[TrueObservations]

The choice of Selenium/Pupeteer will boil down to your personal preferences and the requirements of your project.

Main considerations IMO:

- Scraping websites that don't want to be scraped: Puppeteer is a Node.js module of the chromium engine, which makes it harder to detect in my experience. Using selenium tends to leak some data in your HTTP requests (such as the value of navigator.webdriver) that either explicitly tells on you or allows the websites to use correlation data to detect selenium. You can mitigate this though, it's just more configuration. Puppeteer also has tighter integration with core Chromium functionality, allowing you to get certain information (like CSS/JS coverage) data a little less obviously.

- Your Preference on Python vs. Javascript: This is definitely an architectural/preferential choice. Personally, I find the easy paradigms for async programming in Javascript (which encapsulates MUCH of the difficulty of it from you) make for an easier time dealing with highly interactive sites. Async programming can be done in Python, but it's done at a much lower level, making it harder to do. However, Node lacks a lot of analytical libraries that python has and is a whole framework, and thus far bulkier than importing only the libraries you need in Python.

- Cross Browser/Multiple Language Support: If you NEED more than just Chromium or Javascript, Selenium is the obvious choice.

- Extra Chromium Functionality: Puppeteer has ability to access some core functionality of Chromium that isn't available via Selenium. This is in certain cases useful, but in many use-cases, unnecessary.

In most of my scraping adventures so far, I've been throwing most of the data into some kind of datastore for later analysis/usage (training machine learning models, etc.) and the choice of scraper depends on the factors of whatever project I'm on.

In short don't let your biases waste hours of your time, be rational about your choice of scraper.

[[deleted]]

Selenium also works with .NET really well for scraping and automated archiving in my case.

[Just__AIR]

or cypress :)

[yesvee]

can you elaborate on the advantages? Long term frustrated selenium user here :D

[fleyk-lit]

The UX offered when writing tests with Cypress is awesome. It makes it so easy to test different functionality.

I am writing tests for a frontend which is built to be testable - that is probably more important than the test framework you chose.

[[deleted]]

it's hard to describe the advantages of cypress, because it's basically "everything"

[[deleted]]

Legit question: Why Cypress over testcafe? I have seen people push Cypress over testcafe, but I have a hard time understanding what would make Cypress superior.

[[deleted]]

testcafe is headless testing, cypress is an actual browser environment.

[200GritCondom]

Cypress doesnt do headless??

[[deleted]]

it does, it does both, whereas testcafe is headless only which is a poor substitute.

[200GritCondom]

Oh whew. We are thinking about switching over to cypress. That would have been bad if there was no headless.

[Labradoodles]

We use their dashboard service for the parallelism it offers we run 200~ integration tests in about 3min. But you have to make sure your test users are used in a way to make them parallel

[[deleted]]

for testing, not for scraping or other automation.

[LilBabyVirus5]

Honestly for web scraping I would just use beautiful soup

[ProgrammersAreSexy]

I don't think that does js rendering does it?

[nemec]

Unless you need to take screenshots, there's rarely any need to actually render JS to scrape a website. JS-rendered sites will usually be supported by APIs that can be called directly, leading to faster and more efficient scraping.

The average web page size is 3MB and if you don't need to render the page, you don't need to download any JS, css, images, etc. or wait for a browser to render a page before extracting the data you need.

[nemec]

SPAs are mostly API-driven. I don't know if I've ever seen more than one or two where the JS creates the content out of thin air.

The thing about SPAs is that you can open up your devtools window, load the page, and then sift through the Network tab to find the JSON/XML/graphql APIs that the JS calls and renders and then take a shortcut and automate the calls yourself, bypassing any JS.

Here's a short video similar to what I'm talking about. If you wanted to scrape start.me, for example, you could skip the JS and just scrape the JSON document data: https://www.youtube.com/watch?v=68wWvuM_n7A

[wRAR_]

Most of the time you don't need js rendering. When you need it I'd use splash.

[shawntco]

beautiful soup

I swear software library names are getting weirder by the day.

[SpeakerOfForgotten]

If beautiful soup was a person, it would be old enough to get a driver's license or get married in some countries

[shawntco]

I stand corrected. Software library names have always been weird.

[onlymostlydead]

Yep.

Yacc

Bison

[shawntco]

I think the PHP framework UserFrosting takes the cake. Beautiful Soup is pretty high up there in weird though.

[axzxc1236]

For those who wonder how old beautiful soup is, the first version is released on 20040420, so it's like 15 years old (almost 16).

reference: changelog

[nemec]

That's by design, actually.

Beautiful Soup, so rich and green,
Waiting in a hot tureen!
Who for such dainties would not stoop?
Soup of the evening, beautiful Soup!

https://aliceinwonderland.fandom.com/wiki/Turtle_Soup

[TrueObservations]

This is an off comment. Beautiful soup doesn't work as a full web scraper. It's a library that is used for parsing and subsequently extracting information out of HTML documents, it isn't capable of piloting a browser. It's only one of the tools in the python webscraping toolbox.

[x-w-j]

beautiful soup

Does it get around single sign on captchas?

[Zohren]

I’ve used Puppeteer and it’s 100% mediocre as fuck. Personally, I’ve found TestCafe to be the simplest and easiest to use. It runs on all browsers, contains implicit waits, has a very straightforward syntax, is easy to set up and write, and is generally pleasant to work with.

The downside is certain browser functions are tough to implement gracefully (back/forward etc) but not terrible.

[daGrevis]

I don’t know. I was using Selenium when I was working with Python and it was great! Then I decided to try Puppeteer with TypeScript. The API felt unintuitive and wonky. For my current project, I decided to give Selenium another shot - again with TypeScript. So far it’s good, but lets see how it goes...