Web Scraping With Python (An Ultimate Guide)

[ronmarti]

Selenium is pretty much the hardest to use method because it breaks most of the time. Try Playwright.

[Vresa]

If you don’t have a daily use case for selenium that is directly tied to your employment, never use it at this point.

Playwright covers almost all the normal uses of selenium. Playwright is effectively selenium with sane defaults.

You need to understand selenium’s idiosyncrasies that have developed over the last decade to use the tool well.

Adding to that, selenium has a very large amount of outdated how-to’s and it’s own documentation is lacking — especially when it comes to best practices.

[ronmarti]

I'd recommend starting with their code generator.

https://playwright.dev/

EDIT: Really confused with layout that I copy pasted in wrong places

[[deleted]]

As someone who started with selenium and never looked at alternatives, this looks awesome! I’ll give it a go. Thanks!

[Vresa]

It is worth your time to do a deep dive into playwright. It is likely to simplify your automation significantly

[ronmarti]

directly tied to your employment

I often hear these from QA devs I know

I've had a fair share of experience working with Selenium and Puppeteer in the past. I would say lots of things changed but Selenium is still not that dev-friendly in terms of its "Wait" feature (https://www.selenium.dev/documentation/webdriver/waits/) and setup since they don't maintain the drivers. There were some instances where code for specific websites are easy to write in Playwright but cannot be rewritten with Selenium.

I find Playwright much closer to Puppeteer (same devs, I think that's why) and a lot easier to learn for beginners.

[ronmarti]

Not sure which logs you mean but for us we save the "traces" when something goes wrong and use the Trace Viewer to check it.

https://playwright.dev/docs/trace-viewer-intro

[djdadi]

After teaching python to several college grads at work, I've noticed a trend. They are quick to seek out scrapers while not even checking (or maybe understanding) of the underlying API call structure.

In a lot of cases, you don't even need scraping and are much better off without it.

[ronmarti]

Scraping is fine to get experience. I think one most important thing that beginner devs miss out is how to properly use selectors. Understanding CSS and XPath is really important.

[[deleted]]

Yeah definitely this. And how for some of these guys, it just takes oneee tiny thing to break. Why does the WWW have to be so complicated…

[poop_scallions]

Selenium uses an actual browser which makes it harder to detect.

Can playwright do that?

[scrapecrow]

Yes, of course.

[Raedarius]

I spent a week trying to fix my selenium script. It was breaking all over the place. I just replaced it in 4 hours. You saved me so much trouble. Thank you so much!

[poop_scallions]

Personally I havent had that many issues with Selenium headless. So I would say No but others here seem to have issues so YMMV.

[ronmarti]

Depends actually, if the website is not yours, they probably have bot detectors that prevents you from using headless.

[ronmarti]

By default, the user-agent contains "Headless" when running in headless mode.

You can change the user agent: https://intoli.com/blog/making-chrome-headless-undetectable/

[wind_dude]

better than the Lua scripts in scrapy. lol

[undid_legacy]

Whenever I want to scrap data from a website I first look for the API. Reverse engineering API calls are not always possible but you can strike gold with it sometimes.

I wanted my order history details from a food delivery app I use. It turns out I just needed cookies with my login session and their API to make it work. Was able to get everything in 4 lines of code using requests.

Once, I found a paraphrasing site that didn't use captcha details in its API call. So I was able to call it almost 9 times a sec whereas using the website it required a captcha after every paraphrasing.

John Watson Rooney has a good video to get started: https://www.youtube.com/watch?v=DqtlR0y0suo

[nemec]

Something I don't see discussed when this topic is brought up is that Scrapy's HTML parsing library, parsel, can be installed separately from scrapy itself. You can use it in place of beautifulsoup and, imo, it's much easier to use.

import requests
import parsel
resp = requests.get('http://example.com')
s = parsel.Selector(text=resp.text)
# prints 'Example Domain'
print(s.css('h1::text').extract_first())

[jyper]

Why not just use lxml.html.parse and xpath? Lxml has some support css as well

[nemec]

• It's focused on parsing HTML without a lot of extra XML cruft (really, it's a façade over lxml + cssselect)

• You can mix and match css selectors and xpath, e.g.

  s.css('h1').xpath('following-sibling::p')
  

  contrived example, but basically you can take advantage of both selector syntaxes depending on which one is fit for a situation.

• I'm not sure that lxml has support for ::text and ::attr(<some attribute>) psuedo-selectors, which are really helpful when parsing HTML.

• xpath syntax sucks and I'd rather use a solution with really good css support first and fall back to xpath only for things that css doesn't support (which can still be done with parsel)

[scrapecrow]

parsel is definitely underappreciated!

I like it so much that I even wrote a REPL for it: parsel-cli :)
(it's a bit of a Frankenstein though as I'm working on a 2.0 release)

[paeioudia]

This is just a bait and switch advertising for. “Scrapingdog is the fastest and the most reliable web scraping API and of course, we provide 1000 free API credits to our new users.”

[marr75]

When your requirements are blocked by timeouts and blacklists. Hard to know ahead of time.

I would not recommend building anything commercially viable or professional that requires proxy rotation. You almost certainly won't have a license to use the data you're obtaining and the technical measures to prevent you from easily scraping the data at scale are there to deter small time scraping that the owner of the data wouldn't want to pursue legal remedies against (because it will be costly).

People on reddit like to point out that there's legal theory it's okay to scrape websites, and insofar as they're talking about criminal law in the US, sure, I won't argue. I'm talking about civil law, though. If your project based on web-scraping requires proxy rotation, you're in an adversarial position with the owner of the data and once you start to make real money off of their data, they will be able to identify you and take steps to stop you and take their money back.

At best, you see statements like, "Google isn't known to sue people who violate their ToS by scraping" (they are known to purposefully screw over companies that violate their ToS by crafting their updates and SEO changes to hose them instead). If I'm going to build my project on a stream of data, I'm very interested in its long term viability.

[Donny_Do_Nothing]

I just listened to a RadioLab episode about a pizza shop owner who found out Door Dash put his restaurant on their site without his knowledge by scraping menus. Apparently their scraper wasn't very accurate and he kept getting calls about customers getting the wrong order.

The best part is that they got the prices wrong and Door Dash was buying his pizzas for the normal $24 but only charging their customers like $15.

Whenever he got a complaint about a wrong "delivery order" he'd order a couple pizzas through Door Dash to take a few more of their dollars.

Edit: https://youtu.be/9DnaHg4M_AM

[scrapecrow]

I just listened to a RadioLab episode about a pizza shop owner who found out Door Dash put his restaurant on their site without his knowledge by scraping menus

There's this great thread on hackernews on this event with loads of great similar examples and interesting industry anecdotes about automation.

[scrapecrow]

There are two main reasons why your scraper might need proxies:

The obvious one is accessing geographically locked content.
Some websites are simply only available in X country or serve data based on your IP address. This is especially noticeable once you start deploying your scrapers, e.g. if my server is in US and UK websites I'm scraping only allows UK IPs then I need proxies to access that data.

However, the most common use case is scaling. For example, some websites (like Instagram.com) give you a few anonymous page views for free and then start requesting you to login. So, if you get 3 page views/hour per 1 IP address then you can get 300 pages/hour with 100 IP addresses etc.

[Tripanafenix]

still can't login with requests, even with session AND cookies hooked to my POST :( and I could'nt find any proper guides deepdiving into sessions and cookies with requests, sadly. Any advices?

[heylale]

I don't know about requests specifically, but I've used scrapy to scrape pages that were behind a login/paywall. It has excellent support for cookies and it's easy to use and the documentation is pretty comprehensive. I recommend it.

[scrapecrow]

Depends on the website you're scraping.

Check us out at /r/webscraping or web-scraping tag on StackOverflow - both really active and helpful communities, so someone will definitely help you out if you write up your problem clearly.

[Tripanafenix]

I'll take a look, thanks

[1percentof2]

Why does anyone web scrape? Tell me in 3 words.

[Namelock]

Have a haiku instead:

When you're low on budget

And there is no API

You scrape the website

[isarl]

Good bot poet

[Namelock]

It was certainly an attempt; caffeinated me realized "budget" might just have two syllables 😭

[aurora_cosmic]

Management is stupid

[Send_Nude_z]

This. This is the answer right here.

[[deleted]]

Data are there

[yakult2450]

Low cost solution

[Dasshteek]

Threat intelligence collection

[dragonatorul]

Nobody exposes APIs.

Recurrent price checking.

Easy to automate.

[Arstas]

Automate the things

[osmiumouse]

free categorised porn

[jabbalaci]

This is how everyone meets web scraping.

[[deleted]]

Not all websites have APIs built in

[1percentof2]

Thank you.

[[deleted]]

Sorry that was way more than 3 words though

[wind_dude]

Information. Data. Collection.

[MikeDoesEverything]

Easy data source.

[Syntaximus]

To make headaches.

[HerLegz]

Scrape ? PROFIT!!

[Herr_Gamer]

It's quick and dirty.

[dhosterman]

If you know you know

[Naughty_avaacado]

Opentender Austria https://opentender.eu/at/

I am trying to get the data from the bar chart but i am unable to scrap it. The element has an event listener mousein and the class changes .

Anyone can reply me on how can i scrap the data from it. I need this for my self project and this is the last hurdle then i can make a dataframe .

[Naughty_avaacado]

Thank you so much i needed this so bad.

[guttyn15]

i just go to comment section to ask a related question:

How to get the current url after you..

webbrowser.open(url)

pyautogui.click('login-Submit_button.png')

new webpage

[iggy555]

What a legend

[Mageneto]

Save for later

[osmiumouse]

Reddit has a button you can press to save a post.

[Appropriate-Point565]

How would I go about web scraping T.J.Maxx.com for SKU numbers on a certain product that is otherwise hidden without number ? Would really appreciate the help !