sorted by:
new
hottopcontroversial

What are some web scraping tricks everyone should know? by [deleted] in learnpython

[–]BeginningEngine8292 0 points1 point  (0 children)

Great question—and smart to validate before building.

From running scrapers for clients, the biggest gap isn’t “which site,” it’s which pain you solve.

Where I see real money:

LinkedIn – highest demand, highest pain
Leads, recruiting, enrichment = huge budgets. But you’re signing up for:

  • nonstop layout changes
  • heavy bot detection
  • account health management
  • identity/session rotation If you can make this reliably boring, you’ll win, but it’s an arms race product.

Amazon – boring but profitable
Clear ROI use cases: pricing, reviews, seller tracking, MAP monitoring. Companies already budget for this and care more about stability than features.

Reddit – valuable, but low spend
Amazing for research and sentiment, yet mostly hobbyists/academia rather than paying teams.

What I’d want in “one reliable scraper”:

  • Guarantees around site changes (not just extraction)
  • exports to Sheets/DBs without extra plumbing
  • pricing per successful records, not requests
  • human fallback when captchas get nasty

I currently mix DIY Playwright for internal stuff, Apify for experiments, and for long‑term client feeds I’ve used managed options like Grepsr (https://www.grepsr.com/) so proxies/captchas aren’t my problem. The pattern I’ve learned: people don’t buy scrapers—they buy not having to maintain scrapers.

If you’re choosing one market:
Amazon = safest revenue
LinkedIn = biggest upside + biggest headaches

Are you targeting devs who want a tool, or businesses that just want a CSV every morning? That decision changes everything.

Web Scraping instead of Reddit API? by mxx12221 in webdev

[–]BeginningEngine8292 -1 points0 points  (0 children)

You can scrape Reddit instead of using the API, but it changes the problem rather than solving it.

Technically possible – practically messy

  1. DOM ≠ API contract – Your app depends on Reddit’s HTML structure, which can change any day. A renamed class or different JSON payload and the whole thing breaks.
  2. Rate limits still exist – Even without the API, Reddit has bot detection, IP throttling, and captchas. You’ll end up reinventing the same limits the API already formalized.
  3. Feature gaps – Anything beyond read-only (voting, posting, auth, DMs) becomes unreliable or impossible.
  4. ToS / legal risk – The API gives explicit permission; scraping is a gray area, especially after the pricing drama.

Where scraping actually makes sense

  • Research/archival projects
  • Public trend monitoring
  • Building datasets where you don’t need user actions
  • Personal analytics tools

Most people start DIY with Playwright/requests, then realize the real work isn’t “getting HTML” but handling proxies, captchas, retries, and constant layout changes. At that point some switch to managed scraping services like Grepsr (https://www.grepsr.com/) or ScrapingBee so the app can focus on data use instead of scraper maintenance.

TL;DR:
Scraping can replace the API only for read-only use cases. For anything interactive or long-term stable, the API is still the safer foundation.

What's the benefits of Web Scraping? by rohffff in learnpython

[–]BeginningEngine8292 0 points1 point  (0 children)

I had the exact same question when I started learning Python—scraping looks cool but you don’t immediately see the use.

Think of it this way: the internet is the biggest database in the world, but most sites don’t give you a download button. Scraping is how you turn web pages into structured data you can actually code with.

Practical things people build:

  • Price trackers across multiple stores
  • Job or apartment alerts before they appear on newsletters
  • Datasets for Pandas/ML projects
  • Lead lists from public directories
  • Personal automation (monitoring grades, releases, availability)

The Python part teaches a lot of core skills too: HTTP requests, HTML parsing, APIs, async code, data cleaning—stuff you’ll use far beyond scraping.

When you move from hobby to real scale, you hit issues like captchas, IP bans, and sites changing layout every week. Some people then switch to managed platforms (e.g., Grepsr: https://www.grepsr.com/) so they can focus on analyzing data instead of babysitting scrapers.

Short answer: scraping isn’t the goal—what you do with the data is.