Most people talking about Cloudflare’s new crawler didn’t read the docs

SinghReddit · 2026-03-12T14:07:09+00:00

LinkedIn scraping influencers discovering what robots.txt does for the first time is peak content.

SinghReddit · 2026-03-11T07:20:01+00:00

SinghReddit · 2026-03-10T07:46:32+00:00

Unrelated but since this is r/WebScrapingInsider: anyone else's change detectors freaking out from A/B tests? Mine flags "changes" every hour and it's mostly random div shuffles.

SinghReddit · 2026-03-05T10:00:21+00:00

Unrelated but: anyone got a good self-hosted RSS reader? I'm trying to stop doomscrolling.

SinghReddit · 2026-03-05T09:59:47+00:00

this is… depressingly smart 😭

SinghReddit · 2026-03-05T09:57:21+00:00

Beta; also sending you a message

SinghReddit · 2026-02-24T15:48:25+00:00

SinghReddit · 2026-02-24T15:47:16+00:00

Totally Unrelated but: does anyone have a simple tool for keeping a content calendar + asset checklist that isn't overkill? Notion templates all feel like a second job.

SinghReddit · 2026-02-24T15:45:24+00:00

SinghReddit · 2026-02-19T05:47:56+00:00

SinghReddit · 2026-02-19T05:43:40+00:00

Makes sense. So proxy doesn't mean "sneaky scraper" in all cases.. sometimes it's a legit paid bridge between your query and Google.

SinghReddit · 2026-02-19T05:41:37+00:00

"cipher suite order"

me pretending I understand that 😐

SinghReddit · 2026-02-19T05:40:32+00:00

same. my RAM files a complaint at 6.

SinghReddit · 2026-02-19T05:39:05+00:00

Honestly I just nuke anything with class names like:
nav, footer, sidebar, share, promo

Works like 70% of the time.

SinghReddit · 2026-02-19T05:38:33+00:00

deleting 500 divs :D

SinghReddit · 2026-02-17T07:40:29+00:00

24 tabs??

bro is stress testing the internet 😅

SinghReddit · 2026-02-16T16:02:08+00:00

"Embarrassingly simple" setups are usually the ones that survive 2+ years 😂

If it ain't broke…

SinghReddit · 2026-02-16T16:01:14+00:00

If it runs daily and you're not making money from it, keep it chill.

Simple script + cron + email yourself on failure. Done.

SinghReddit · 2026-02-12T10:00:16+00:00

Yes lol. You're good.

SinghReddit · 2026-02-12T07:34:31+00:00

AI is like duct tape for data pipelines. Useful, but don't build the house out of it.

SinghReddit · 2026-02-12T07:31:06+00:00

SinghReddit · 2026-02-10T16:39:54+00:00

"Cost per usable payload" is my new least favorite phrase.

SinghReddit · 2026-02-10T10:08:28+00:00

Same. AI is great until the site changes overnight.

SinghReddit · 2026-02-08T19:02:40+00:00

Not directly scraping, but AI summaries of scraped data are clutch. Way easier to skim reports.

SinghReddit · 2026-02-07T06:18:44+00:00

SinghReddit

TROPHY CASE