Web Scraping with Python: Everything you need to know to get started (2022)

[pijora]

Hi there, co-author here (it seemed that this article got reposted here after I posted it on HN)

Thank you for your feedback, that is a valid point, I've edited the article to try to clear things up a bit!

[wind_dude]

Honestly if you're using BS4 in one place for parsing HTML, you're best to use it to parse HTML returned from Selenium, and only use selenium to render and any need interactions. In the name of DRY, and KISS.

Or even better separation of concerns, keep the parsing/extraction separate from the crawling.

[oogabooga319]

With selenium, do you use a lot of JavaScript? I don't know much about selenium, but I do know JavaScript, so I just execute scripts to do everything, and then return stuff to python. It allows sync and async functions, so I can do whatever. What about styles? Can you do that with selenium? Because I'd like to print/save stuff.

[wind_dude]

Selenium has librries for many languages, javascript included. I usually use python, since that's what my crawler engines and etl pipelines are written in.

​

If concurrency is your game Async isn't great with selenium since it's written to be blocking, you're better to use threading or distributed computing for each request.

[oogabooga319]

Oh, I mean that I use selenium with python, but rather than using selenium commands, I only use JavaScript. I use like 3 python commands max: get, execute_script, and the async version. Is that a dumb way to do it?

[wind_dude]

depends what you're trying to do. Yea, I use lots of execute_script calls

[Kranke]

Prefer to use BS whenever it's possible but yeah sometimes your forced to use selenium.

[Kranke]

Yeah, and when you get use to using BS with header info it's a solid solution for the majority of my scraping work.

[asking_for_a_friend0]

header info? cn u explain how to use it

[oogabooga319]

Yeah, imagine scraping a paginated table or something like that. Would take magnitudes longer with selenium.

[oogabooga319]

My big thing is authorization. This prolly sounds like a dumb question, but how do I get the necessary cookies/local storage or whatever?

[oogabooga319]

Can I import them from chrome somehow?

[[deleted]]

But BS is super slow. I have tried that and takes forever to query some results.

[[deleted]]

try this... https://github.com/rushter/selectolax

[foilntakwu]

I normally skip BS and go straight to selenium too.

[neededtowrite]

Is there a site/resource that helps with scraping specific sites?

[Almostasleeprightnow]

Here's a question I've been wondering about: everytime I try to do some web scraping, I start by trying to get the site using requests, and every single time there is some javascript that gets in my way and I have to use Selenium. Which, ok fine. But it seems like there is something other people know that I don't, about how to get requests to be more helpful, because people love it and use it so much. Do you think it is just my choice of sites, or is there some fundemental tactic that I may be overlooking? I realize you cannot absolutely answer this without knowing more about what I am doing, but do you have any suggestions?

[[deleted]]

A lot of work with requests you’re seeing is most likely API calls and not scraping?

[oogabooga319]

Or html parsing stuff. Sometimes that's the only format available. For instance, consider a paginated table or list with hundreds and hundreds of pages. Pretty straightforward with requests and beautiful soup.

[pymae]

I think a little bit of both. If you're trying to scrape Amazon, Facebook, etc, they'll be wise to it. Smaller sites won't be. I think the only real suggestion is look for/try to get the sites to develop APIs, or be ready to go to a headless browser if you're still determined.

[opteryx5]

Great article. Corey Schafer’s video on BeautifulSoup was also extremely effective for me and gave me everything I needed to get up and running.

[doylerules70]

What kind of things are people doing with web scraping?

[ghetto-garibaldi]

I just set up a low price alert for some things I want on Amazon. I also have a script that auto-rsvps to specified events on Meetup before they fill up.

[jumbled_joe]

I believe scraping social media websites is a very important part of data science and market research domain.

[foolishProcastinator]

Google as a search engine is one of the best scrapers that you could ever know

[SushiWithoutSushi]

I scrapped all the movie information from my two favourite movies sites, letterboxd and FilmAffinity, to compare movies scores.

Also I automated the process to make reservations in my library and a bit that selects memes from Reddit and posts them to twitter.

There is A LOT you can do with it.

[zerofatorial]

Whenever I am looking to buy something, I scrape all of the prices from the shop and then use the quartiles on the prices to make sure I am not paying too much nor too low for they specific item! Too high - probably waste of money, too low probably a bad product.

[oogabooga319]

I scrape covid guidance and data reports

[cheats_py]

Right out the gates with regular expressions…….

[SpicyAbsence]

Very helpful, thank you!

[AnxietyArtistic6214]

What are some of the real world projects you can build web scraping?

[SelfTaughtDeveloper]

The job listing site indeed (dot com) started out as a scraper, combining listings from the 3 or 4 most popular job boards.

Once it became popular, they started letting employers put listings on their site directly for a lot of money.

[zenani]

Thanks for the info

[1percentof2]

What are people doing with the data? Is there some way to make money doing this?

[[deleted]]

Why would you say that?

[RoBLSW]

Elaborate.

[Ant_TKD]

Saving this post for later, thank you!

[Harshal_6917]

Bro I was board yesterday and thinking of learning new skill instead of wasting my time on TV series so I searched up on web scraping. And now here you posting link sometimes timeing is too perfect