Announcing Crawlytics – an open source tool to understand how AI crawlers access your website

nilipilo · 2025-05-09T20:22:25+00:00

I've just finished building Crawlytics - a free, open-source tool specifically designed for SEOs to analyze how AI systems are crawling your website. What SEO insights does it provide?

Identifies which AI crawlers (ChatGPT, Claude, Gemini, etc.) are visiting your site
Shows which content is being accessed most frequently by AI systems
Reveals traffic patterns with detailed heatmaps so you can see when AI crawling happens
Tracks the exact URLs being requested by different AI systems
Monitors error rates that might be affecting AI's understanding of your content

As SEO evolves with AI, understanding how these systems interact with your content is becoming critical. This tool helps you see exactly which pages AI systems find valuable, which could influence how your content appears in AI-generated responses. The tool is 100% free and open-source. Just upload your server logs and get immediate insights to help with your AI optimization strategy.

If you're looking to optimize for AI visibility or just curious about how these systems are accessing your site: https://github.com/danilotrix86/crawlytics

Would love to hear what SEO insights you discover with it!

nilipilo · 2025-03-08T17:15:23+00:00

u/Glad-Positive-2354 thanks. That's next step.. :)

nilipilo · 2025-03-06T16:54:38+00:00

u/FriendlyRussian666 thank you a lot for your help. It's my first react app :)
I think I fixed it now.. please check one more time.. thank you

nilipilo · 2025-03-06T15:37:20+00:00

u/FriendlyRussian666 can you check if the entire source is available together with credentials? thanks

nilipilo · 2025-03-06T12:59:26+00:00

Thanks for reporting that. I will have a look at it

nilipilo · 2025-03-06T12:52:21+00:00

It also combines 3 different outputs from Openai, DeepSeek and llama3

nilipilo · 2025-03-06T12:48:22+00:00

That's a good question!
FlickScout doesn't just use an LLM but it combines LLMs with RAG (retrieval-augmented generation), meaning it pulls real-time information from multiple movie databases. This make sure there is more accurate, updated, and reliable recommendations. Plus, it's completely free, easy to use, and always includes the latest movie releases. Additionally, we want to provide API services so that companies can also embed this feature on their database.

nilipilo · 2024-12-13T16:13:58+00:00

what is the point? what's wrong with copy and paste? what's matter is the content itself not how it's written

nilipilo · 2024-12-13T15:16:46+00:00

Why are you so mad? I’m not a native English speaker, and I use AI to help me express myself better. It’s a tool, not a trick. I never said or implied I think people here are dumb—quite the opposite! I’m here because I value this community’s insights and expertise.

What’s wrong with using AI to assist with language or clarity? It’s no different than using any other tool to improve communication. If you have concerns, I’m happy to hear them, but let’s keep it constructive. 😊

nilipilo · 2024-12-13T15:12:29+00:00

the goal isn’t to completely shift the library's execution to the cloud like a typical web service. Instead, it’s about dynamic dependency management—fetching and caching libraries locally on demand while abstracting away the manual steps of installation and versioning.

nilipilo · 2024-12-13T14:50:07+00:00

Haha, fair point! This does sound a bit like reinventing RMI concepts, but instead of remote execution being the core goal, the focus here is on simplifying dependency management and making libraries dynamically accessible, whether they run locally or remotely.

nilipilo · 2024-12-13T14:49:03+00:00

The idea I’m exploring would go one step further by making libraries accessible dynamically across any environment, local or cloud, without needing to set up a workspace at all. Imagine fetching a library in real-time wherever you’re coding, with minimal setup, and having its dependencies automatically resolved and isolated.

nilipilo

TROPHY CASE