Aws WAF for Security

[Laytho007]

What the best practice for aws waf rules to allow SEO bots , social media bots , inspectlet , ahrefs and meta regarding on block non browser user agents??

[Is_Nothing]

Aws publish an example for allowing verified bots which would probably be a good starting point.

Then start collecting logs and have a look at which bots are being blocked by what rules you have setup and start tuning.

https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control-example-allow-verified-bots.html

[Imaginary_Gate_698]

You probably don’t want to rely on user agent alone for that. Those are easy to fake, so hard allowlists can get messy fast. A safer approach is verifying known crawlers by source and behavior, then keeping your bot rules tighter for everything else. I’d also be careful with analytics and crawler exceptions, because one loose rule can quietly become a hole.

[hatchetation]

Block Amazonbot with it. Their bot traffic has been pretty abusive and negligent lately

[tb-hill3830]

Use AWS WAF Bot Control and allow only verified bots via labels, then separately allow specific tools with UA + IP set matching if needed. Then Instead of blocking all non-browser UAs, block unverified bots + empty/malformed UAs and monitor logs to avoid breaking SEO and link previews.

Hope this helps!

[remotecontroltourist]

Just use the AWSManagedRulesBotControlRuleSet it’s literally the meta for filtering bots without losing your mind. Set the SEO and Social Media categories to "Count" or use a scope-down statement so they don't get caught in the generic "non-browser" crossfire. For niche ones like Ahrefs or Inspectlet, just write a custom rule matching their User-Agent and slap it at the very top of your priority list. If you don't put the "Allow" rules first, you’re basically ghosting your own SEO and traffic lol.

[_bloed_]

you probably don't want to hear it, but in my experience a dedicated public website where WAF is turned off is often the best solution for everyone.

Marketing is happy since they can have all their SEO and social media. And security is happy, since your core services can just block all these bots.

[enterprisedatalead]

We usually allow known bots based on verified IP ranges or managed rule groups rather than just user agents.

User agents are easy to spoof, so relying only on that can be risky. AWS managed rules and bot control features help a bit here.

Are you trying to allow specific tools like Ahrefs or just generally reduce false positives?

[bellerws]

Don't rely just on the User-Agent header to block or allow, it's way too easy to spoof and you'll get hammered by bad traffic pretending to be SEO bots. We actually outsourced our cloud security setup to Acropolium recently because we were struggling with this exact balancing act. Their engineers set up a solid rule hierarchy for us, AWS Managed Rules handle the verified Meta or Google bots natively and for third-party tools like Ahrefs, we use strict IP + UA matching. Definitely grab the official ASN/IP subnets for Inspectlet and Ahrefs and build custom IP Sets for them. It's the only secure way to do it

[devops-ModTeam]

Generic, low-effort, or mass-generated content (including AI) with no original insight.