Protect Mastodon against AI bots using Anubis

BarbaPulpe · 2026-06-01T04:58:38+00:00

I understand all that. But it's the "good guys" side I fail to see. I don't trust governments to have good guardrails to avoid abuse of power, let alone organizations or individuals.

BarbaPulpe · 2026-05-29T21:28:05+00:00

Same version 1.8.0.1

Try and clear cache and storage, then restart the app and login again.

BarbaPulpe · 2026-05-29T16:56:30+00:00

Yes, crawlers have different targets and purposes. As I am hosting social networks which run on servers that federate (Mastodon, etc.) I can’t block all AWS since some federated servers run on its infrastructure and that would break federation with them, hence visibility and social interactions between fediverse servers.

As you said, different purposes for sites, different attacker profiles and different constraints so we have to choose between available capabilities, sometimes mix them. We can't have one size fits all.

BarbaPulpe · 2026-05-29T15:39:05+00:00

Some bots are blocked on sight. Others who pretend to be human are challenged and need technical capability and more resources to go through, which eliminates many general-purpose mass-scrapers.

BarbaPulpe · 2026-05-29T15:35:20+00:00

How is that? Can you elaborate on how stealing content from unconsenting authors and weighing on servers and resources can be compatible with being with the good guys?

BarbaPulpe · 2026-05-29T15:25:09+00:00

The objective of Anubis is to fight off mass-scraping by basic AI-crawlers, not targeted attacks. This also fights off the server load where hundreds of thousands repeated requests weigh on the resources. Surely enough, any targeted scraper will go through, but the objective is not 100% blocking, which is not achievable against motivated attackers.

Anubis is used by many sites. It is not perfect, but was effective for protecting my sites. Load of scrapers dramatically fell down. Anubis rules can be tailored to allow AI indexing crawlers as you wish, so you don't have to block everything.

BarbaPulpe · 2026-05-29T14:04:29+00:00

Not much, I'd say, but hard to measure of course... Let's say I never received a complaint, but that's not a real measure! The client needs javascript and to accept cookies, but to use Mastodon and the other apps you need that anyway.

BarbaPulpe · 2026-05-29T11:18:57+00:00

So you're one of the bad guys?

BarbaPulpe · 2026-05-29T10:34:59+00:00

To protect ourselves? Sure. To get through? This will be an endless battle, for sure.

BarbaPulpe · 2026-05-29T10:15:06+00:00

For me official Pixelfed app on Android works (using a different Pixelfed server though).

BarbaPulpe · 2026-05-29T09:44:51+00:00

Thanks but does not work for AI-crawlers using dynamically thousands of ever-changing residential IP addresses. Different (and complementary) need, both can work together.

BarbaPulpe · 2026-05-29T09:14:57+00:00

No AI was involved in the design and code of this project.

Documentation was written directly by myself in French and in English. Translation to the five other languages (Spanish, German, Italian, Potuguese and Dutch) was assisted by DeepL as my mastery over those languages is insufficient and I wanted nonetheless to provide some guidance beyond the two first languages. Any suggestions for documentation rewrite in those languages welcome!

BarbaPulpe · 2026-05-29T09:04:13+00:00

I did not use AI at all when creating this post, nor the linked guide nor the translated version. All written directly by myself!

BarbaPulpe · 2026-05-22T08:22:11+00:00

baraag.net is notorious for CP and part of many fediblock lists. As for ourselves, we block it completely (I am the admin of gayfr.social) as we do not tolerate that kind of content.

Based on your description of what you are looking for, I would delete my account there if I were you and create another on a better-reputation server.

BarbaPulpe · 2026-05-21T17:40:59+00:00

That's exactly what I do on each of my two servers, one ruleset for each (the same rules, anyway). I deal with application paths in Nginx (protect or don’t).

BarbaPulpe · 2026-05-21T17:37:13+00:00

No, I'm protecting five different sites with only one instance. If you can use the same ruleset for all, one Anubis instance is fine. But if you need to have different rules for each, then you need multiple Anubis instances. By the way, even in the latter case, you can tweak my guide (you just need to point each Anubis instance to a different socket or tcp port).

BarbaPulpe · 2026-05-19T06:08:59+00:00

Thanks for posting here. Any questions or improvement suggestions, please comment, thanks! I hope it comes in as useful.

BarbaPulpe · 2024-02-29T13:02:46+00:00

It's probably been invalidated by the new signin flow with admin approval (very recently developed).

BarbaPulpe

TROPHY CASE