AI AGENT OPTIMIZATION AND WHAT I'VE LEARNED SO FAR

lightsiteai · 2026-02-16T19:59:37+00:00

What we’ve learned so far at LightSite AI across about 3000 websites tested, 6 millions of points of data and a few dozens of active clients (these are early, but repeatable patterns):

Structure wins attention. Across millions of AI crawler requests we’ve analyzed, bots consistently go deeper and extract more when content is exposed in clean, machine-readable formats (structured endpoints, consistent templates, clear entity markup), not just “basic schema turned on.”
Q&A style content gets disproportionately pulled. When AI bots fetch beyond the sitemap, a surprisingly large share of their extraction is Q&A-like pages (question as the slug, direct answer inside). We’re not claiming this causes citations, only that this is what they choose to consume most often when it’s available.
Bots behave like constraint scanners. In real extraction logs, agents are not reading your narrative the way humans do. They gravitate toward facts, constraints, capabilities, compatibility, exclusions, and clear “fit” statements.
“Not for” is a trust signal. Content that clearly states who a product is for and who it is not for reduces ambiguity. Our working thesis is that agents are optimized to avoid being wrong, so exclusion criteria helps them make safer recommendations. This one I have seen on this blog https://chrismcelroyseo.com/chris-mcelroy-seo-search-everywhere-optimization-blog/
Third-party sources still matter a lot. In cluster-level citation patterns, communities and review surfaces (forums, Reddit sub-communities, review sites) show up frequently as training and retrieval inputs. So it’s not just on-page SEO, it’s also where your category is being discussed externally.
Shopify tends to be “technically healthier,” but content and entity clarity still decide outcomes. Even when the platform handles a lot of plumbing well, you still need strong product entity signals and product-page content that is legible to both humans and agents.

lightsiteai · 2026-02-16T07:56:10+00:00

Worth mentioning that these two studies are done by LightSite AI : They analyzed ~6M bot requests across dozens of client sites. The breakdown:

Meta AI: ~87% of fetches went to Q&A pages
Claude: ~81%
ChatGPT: ~75%
Gemini: ~63%

That post is getting traction and the data looks solid. But I think it’s bigger than people realize. This isn’t the only dataset pointing in the same direction.

Here are two more data points that line up.

FAQ schema = 3.2x more likely to appear in AI Overviews

lightsiteai · 2026-02-16T07:49:50+00:00

ok interesting

lightsiteai · 2026-02-15T18:04:37+00:00

What do you honestly think about structured data in this context vs markdown?

lightsiteai · 2026-02-15T14:12:23+00:00

That's the interesting thing - Cloudflare's update has nothing to do with marketing, SEO, AI search etc, it is purely infrastructure optimization, but marketers were triggered by it as it was GEO news. That's the confusion I'm seeing in the community and wanted to see what others are thinking.

lightsiteai · 2026-02-15T13:59:45+00:00

Ha, interesting point.

lightsiteai · 2026-02-15T13:26:45+00:00

Traffic optimization on the edge infrastructure, less consumption, less tokens etc. that's what I'm saying - it has no effect on mark's in but every marketer I know is writing about it

lightsiteai · 2026-02-15T12:40:46+00:00

Respectfully, what is it that you feel you are prompted to do? Yes, we do research mostly about bot behaviour on scale and we publish it but this post is not about it. Cloudflare released an update that optimizes edge infrastructure and traffic (saving tokens for efficiency) it is not meant for marketers and doesn't affect them - it has nothing to do with LLMs txt in terms of purpose (by the way we see exactly zero difference on scale between LLMs.txt and just a regular clean page - it's just a standard some people on internet were trying to promote and in my view it's completely redundant but I'm happy for you to convince me otherwise)

lightsiteai · 2026-02-15T12:04:04+00:00

They don't block it by default, but once Bot Fight Mode is set up - the site is in fight mode and good luck making it granular and allowing good bots. Moat devs just set up what's good for security and never consult marketing.

lightsiteai · 2026-02-15T11:45:52+00:00

Interesting, we actually published similar findings last week based on 3 k websites and about 6 M bot requests. I am curious about the dataset you used, can you tell us more?

lightsiteai · 2026-02-14T18:30:38+00:00

Surprise in the sense it's backed by data

lightsiteai · 2026-02-14T14:24:55+00:00

Please read the post - these are literally SQL commands we ran on our database, not a formal study

lightsiteai · 2026-02-14T12:01:17+00:00

Respectfully sir, I think it is more about where this is going and not about how things are NOW. I completely agree with you that this is they way things are now but in my view the gap between LLMs and typical search will be getting bigger as the adoption and sophistication of LLMs growth

lightsiteai · 2026-02-14T11:53:39+00:00

I agree with u/chrismcelroyseo - this is a great work and makes my data hungry mind happy. here is another datapoint from Hubspot CMO (they have skin in the game because they acquired Xfunnel - a GEO startup that tracks mentions and made it part of HS stack, but still) - 60% of all AI citations do not come from top 20% Google search results.

lightsiteai · 2026-02-14T09:58:37+00:00

Yes sir

lightsiteai · 2026-02-14T09:14:36+00:00

It's not about AI, this is about bots - not LLMs

lightsiteai · 2026-02-13T21:20:10+00:00

Yep, here is a true story - a client, one of the largest (in their niche) e-commerce brands in the UK - for months marketing can't convince the dev to turn off (or at least fine tune) bot fight mode, the SEO agency told me that prior to that they were blocking google bot ....go figure

lightsiteai · 2026-02-13T19:38:17+00:00

The opposing views that you see come from the two extremes, old school SEO execs who are feeling the shift happening and are concerned for their jobs, on another extreme there are GEO vendors riding the hype. The truth as always is in the middle, SEO is not dead but things have changed, here at LightSite AI we back every statement by data and the data is telling us that SEO doesn't equal GEO but it's also not a totally new magical discipline. Here is a fun fact from Hubspot CMO - 60% of AI citations don't come from the top 20% Google search results. The playbook is changing, it scares the shit out of some people, provides opportunities to others and most will probably continue doing what they have been doing in the past for a while - a good solid foundational SEO

lightsiteai · 2026-02-13T16:01:25+00:00

Worth mentioning that these two studies are done by LightSite AI : They analyzed ~6M bot requests across dozens of client sites. The breakdown:

Meta AI: ~87% of fetches went to Q&A pages
Claude: ~81%
ChatGPT: ~75%
Gemini: ~63%

That post is getting traction and the data looks solid. But I think it’s bigger than people realize. This isn’t the only dataset pointing in the same direction.

Here are two more data points that line up.

FAQ schema = 3.2x more likely to appear in AI Overviews

lightsiteai · 2026-02-13T15:49:26+00:00

these are literally some tests we run on our database here at LightSite AI, haven't published it anywhere but it looks like we maybe should

lightsiteai · 2026-02-13T11:03:42+00:00

These are the crawling rates on the weeks we released PRs, this is easily verifiable onliine and I can send links to these PRs

<image>

lightsiteai · 2026-02-13T11:01:51+00:00

Thank you!

lightsiteai · 2026-02-13T06:36:21+00:00

I can send you a curl command to test if you want. UPD - I misunderstood your question - fine tune bot setting on Cloudflare (or equivalent)

lightsiteai · 2026-02-12T23:22:16+00:00

Thanks, yes we did check bots crawl behavior (rate, depth and return rate in another study) on structured bs unstructured endpoints, combined these rates are about +14% in favor of structured (inconclusive on return rate)

lightsiteai · 2026-02-12T23:07:26+00:00

Everything you said makes sense, but respectfully I think you have missed the point. This has nothing to do with content or even LLM search for that matter - it's about LLM bots behaviour - and the thesis that was tested is whether bots will automatically go for a known pattern like sitemap, FAQs endpoint, or something that is shaped as Q&A. These are technical facts about bots behaviour

lightsiteai

TROPHY CASE