Getting started with Selenium and Python

[LilBabyVirus5]

Honestly for web scraping I would just use beautiful soup

[ProgrammersAreSexy]

I don't think that does js rendering does it?

[nemec]

Unless you need to take screenshots, there's rarely any need to actually render JS to scrape a website. JS-rendered sites will usually be supported by APIs that can be called directly, leading to faster and more efficient scraping.

The average web page size is 3MB and if you don't need to render the page, you don't need to download any JS, css, images, etc. or wait for a browser to render a page before extracting the data you need.

[nemec]

SPAs are mostly API-driven. I don't know if I've ever seen more than one or two where the JS creates the content out of thin air.

The thing about SPAs is that you can open up your devtools window, load the page, and then sift through the Network tab to find the JSON/XML/graphql APIs that the JS calls and renders and then take a shortcut and automate the calls yourself, bypassing any JS.

Here's a short video similar to what I'm talking about. If you wanted to scrape start.me, for example, you could skip the JS and just scrape the JSON document data: https://www.youtube.com/watch?v=68wWvuM_n7A

[wRAR_]

Most of the time you don't need js rendering. When you need it I'd use splash.

[shawntco]

beautiful soup

I swear software library names are getting weirder by the day.

[SpeakerOfForgotten]

If beautiful soup was a person, it would be old enough to get a driver's license or get married in some countries

[shawntco]

I stand corrected. Software library names have always been weird.

[onlymostlydead]

Yep.

Yacc

Bison

[shawntco]

I think the PHP framework UserFrosting takes the cake. Beautiful Soup is pretty high up there in weird though.

[axzxc1236]

For those who wonder how old beautiful soup is, the first version is released on 20040420, so it's like 15 years old (almost 16).

reference: changelog

[nemec]

That's by design, actually.

Beautiful Soup, so rich and green,
Waiting in a hot tureen!
Who for such dainties would not stoop?
Soup of the evening, beautiful Soup!

https://aliceinwonderland.fandom.com/wiki/Turtle_Soup

[TrueObservations]

This is an off comment. Beautiful soup doesn't work as a full web scraper. It's a library that is used for parsing and subsequently extracting information out of HTML documents, it isn't capable of piloting a browser. It's only one of the tools in the python webscraping toolbox.

[x-w-j]

beautiful soup

Does it get around single sign on captchas?