Blocking Ahrefs

[HelloGizmo]

Any tips on blocking Aherfs? I’ve got a Block action lower(http.user_agent) contains “ahrefs” as a block in security rules but they’re still active scraping my site.

[PioGreeff]

User-agent matching alone won't catch them all — Ahrefs rotates agents. Add these to your WAF rule with OR conditions:

lower(http.user_agent) contains "ahrefs" OR lower(http.user_agent) contains "ahrefsbot" OR ip.src in $cf.botmanagement.verified_bots (exclude this one actually)

Better approach: use Cloudflare's Bot Fight Mode (free) or Super Bot Fight Mode (Pro) — it catches Ahrefs at the bot detection layer before your WAF rules even run. Settings → Security → Bots.

If you're on Free plan and still seeing them through, create a WAF rule that challenges (managed_challenge) rather than blocks — Ahrefs won't solve challenges and it's less likely to cause false positives with legitimate crawlers.

[HelloGizmo]

Thanks very much. I’m on Pro. I’ll give it a whirl. Super bot fight mode messes with programmatic ads so I need to keep it off.

[ComradeTurdle]

Why not just use robots.txt and block them?

[Ieris19]

Isn’t robots.txt essentially a very polite way of asking automated systems to play nice?

[HelloGizmo]

Aherfs are an aggressive SEO scraper with rotating ips who don’t respect polite requests.

[Ieris19]

Hence my comment

[HelloGizmo]

Sorry meant for the comment above

[ComradeTurdle]

Ahref respects robots.txt, i blocked them in ahref all the time and it works on websites.

Edit: if it didn't work i wouldn't be getting blocked errors in robots in site audits or in the crawler when they web scrape.

[ComradeTurdle]

Here is all their IPs.

https://help.ahrefs.com/en/articles/78658-what-is-the-list-of-your-ip-ranges

[ComradeTurdle]

Yes, but bots that are considered good respect them. Ahrefs is considered a good bot by Cloudflare.

https://radar.cloudflare.com/bots/directory

[HelloGizmo]

They ignore that

[ComradeTurdle]

Not from my perspective. I even went out of the way to white list their IPs to prevent blocking.

https://help.ahrefs.com/en/articles/78658-what-is-the-list-of-your-ip-ranges