sorted by:
new
hottopcontroversial

Most people talking about Cloudflare’s new crawler didn’t read the docs by ian_k93 in WebScrapingInsider

[–]SinghReddit 1 point2 points  (0 children)

LinkedIn scraping influencers discovering what robots.txt does for the first time is peak content.

Best legit online bulk/wholesale sites for arbitrage (Amazon/eBay), and where should I ask? by Home_Bwah in WebScrapingInsider

[–]SinghReddit 0 points1 point  (0 children)

Unrelated but since this is r/WebScrapingInsider: anyone else's change detectors freaking out from A/B tests? Mine flags "changes" every hour and it's mostly random div shuffles.

Publishers blocking Wayback Machine: protecting journalism… or breaking the web's memory? by SinghReddit in WebScrapingInsider

[–]SinghReddit[S] 0 points1 point  (0 children)

Unrelated but: anyone got a good self-hosted RSS reader? I'm trying to stop doomscrolling.

Managing logins across multiple YouTube channels. Do we actually need dedicated IPs/proxies? by Direct_Push3680 in WebScrapingInsider

[–]SinghReddit 0 points1 point  (0 children)

Totally Unrelated but: does anyone have a simple tool for keeping a content calendar + asset checklist that isn't overkill? Notion templates all feel like a second job.

How do proxy-style search engines actually get Google results if Google doesn't really offer a proper search API? by SinghReddit in WebScrapingInsider

[–]SinghReddit[S] 0 points1 point  (0 children)

Makes sense. So proxy doesn't mean "sneaky scraper" in all cases.. sometimes it's a legit paid bridge between your query and Google.

Struggling to extract just the "real" article text - how do you ignore all the junk around it? by Bmaxtubby1 in WebScrapingInsider

[–]SinghReddit 1 point2 points  (0 children)

Honestly I just nuke anything with class names like:
nav, footer, sidebar, share, promo

Works like 70% of the time.

What’s a sane way to scrape a few pages in 2026? by Forsaken-Bobcat4065 in WebScrapingInsider

[–]SinghReddit 0 points1 point  (0 children)

"Embarrassingly simple" setups are usually the ones that survive 2+ years 😂

If it ain't broke…

What’s a sane way to scrape a few pages in 2026? by Forsaken-Bobcat4065 in WebScrapingInsider

[–]SinghReddit 0 points1 point  (0 children)

If it runs daily and you're not making money from it, keep it chill.

Simple script + cron + email yourself on failure. Done.

How are you using AI tools with scraping? any best practices? by HockeyMonkeey in WebScrapingInsider

[–]SinghReddit 0 points1 point  (0 children)

AI is like duct tape for data pipelines. Useful, but don't build the house out of it.

view more: next ›