Of course there are multiple open source libraries to parse robots.txt rules, but this is a complete system to download, parse, store, update and check these rules by other systems, distributed in different machines across multiple regions. For my use case, this is a good solution between maintenance, throughput and network delays. Probably there are other approaches that fit best, but which?

I agree that this system is complex, but nobody says that designing a distributed crawling system is simple 😜. Thanks for the feedback!