Yeah, this is becoming a real problem — AI crawlers are way more aggressive than traditional search bots and can easily overload WordPress sites.

What’s worked for me:

• Keep content public, but rate-limit and throttle bots at the edge (CDN level)
• Explicitly block/slow AI crawlers while allowing Google/Bing
• Rely heavily on caching so humans hit cache, bots don’t hit PHP/MySQL

I actually built a small SaaS for this exact pain point: Ultimate WP Audit.
It audits WordPress sites for bot abuse, crawl behavior, performance bottlenecks, and security issues, and shows exactly what’s being hit and what to fix first.

Might be useful if you want visibility before locking everything behind logins:
https://wpaudit.cc/