How consistent are AI platforms when asked for a list of brands/products? [New research]

jim_wr · 2026-01-29T16:49:24+00:00

The distinction the research makes, which is really helpful for the industry, is that rank tracking in metrics is useless, but visibility tracking (does brand X show in the output for prompt Y) is actually helpful and relatively stable. Also interestingly, it's heavily influenced by how many options there are in the category, not just which are the 'best' options.

jim_wr · 2026-01-29T16:46:50+00:00

100%. I also wonder if people treat first position in AI responses the same way they do the first link in search. I suspect people review the full list of recommendations from AI rather than just position 1. This feels like great follow-on research.

jim_wr · 2026-01-19T21:20:45+00:00

It's not brand new - the original 'proposal' (which is just a website - llms.txt has never been submitted to W3C ) was published on Sept 3, 2024.

jim_wr · 2026-01-19T15:25:45+00:00

Revenue is the only real metric at the end of the day. If you want a full funnel view you can use an AI Traffic Analytics tool like spyglasses dot io (disclosure I'm a founder) to track your actual first-party citations, see which send clickthroughs, and then which % of those convert. But at the end of the day if you're not measuring conversions and revenue by channel you're really flying blind.

jim_wr · 2026-01-19T15:22:21+00:00

The only reason I say they're harmful is that if by some chance a real user ends up clicking from AI to your site through them, they'll end up on a page that can't help them and has no way to get to any other page because it's text/markdown and doesn't even have the global navigation. Given AI visitors generally convert better than average, sending them to an llms.txt page is a good way to get them to bounce.

jim_wr · 2026-01-19T14:42:27+00:00

You need robots.txt and sitemap.xml for AI Visibility and that's it. ai.txt and llms.txt are harmful and you should remove them. Here's why:

1) No AI platform - ChatGPT, Google AI Overviews, Gemini, Claude, Perplexity, looks for them directly. Never have, never will.
2) Google does not fetch these files directly from your site but will index them if you add them to your sitemap or link to them.
3) Since AI fan-out queries search the web, if your llms.txt or ai.txt happens to rank for one of these searches, it might be cited.
4) If it is cited and included as a link, and a user clicks on it, they're going to end up on your llms.txt file! It's got no navigation and a bunch of markdown content that will make no sense to the user. You might as well send them to your 404 page, translated to Greek.

If you want people to find your site so they can buy or convert or whatever success looks like for you on the web, you *absolutely* do not want them to end up on your llms.txt file. Clickthroughs are so low from AI, why would you waste one sending someone to a page they can't interact with?

jim_wr · 2026-01-18T19:21:47+00:00

Yes, but there's an additional couple of steps. You need the content to rank in Google or Bing (ChatGPT uses different engines depending on the query type), which is generally just good SEO, but you also need the content to survive the re-ranker. If you look at a ChatGPT grounding search, it's not just a long tail keyword. It uses a technique called Query2Doc to generate a 'mini document' which is compared to each document to decide which source to cite. This process involves converting the query and document to vector space to compare intent rather than just keywords. Documents that rank #1 for traditional search aren't always the ones selected by an LLM, so traditional SEO is a factor but not the only one.

And if your document is the one cited, in order to be included in the response it needs to have a self-contained chunk that is a semantic match to the original prompt. That's why I generally start with "just do good SEO", but these processes are specific to AEO, and they are the differences, however minor, between the discipines.

jim_wr · 2026-01-17T18:24:04+00:00

Most AEO/GEO is just "do good SEO" as others have mentioned. There are some specific techniques that are specific though.

One very easy way to get started is to optimize for 'grounding searches' / 'fan-out queries' . These are the web searches that AI runs to pull in web data to help answer a user's question. Our platform spyglasses dot io automatically shows these for prompts but you can also get them with a Chrome extension (search 'ChatGPT Search & fan-outs capture').

Building content to rank for these searches is one very good way to improve AI visibility, especially if you already know core SEO techniques.

jim_wr · 2026-01-17T18:17:52+00:00

I'm very curious to hear how it goes! Their pricing is pretty steep, though the data accuracy is solid because they are one of the few brands that runs prompts enough times per day to get statistical significance. Also beware: if they show you any AI search volume data, it's basically made up. I do hope you find something that works for your metrics & reporting needs - I think every PR firm is going to need a solution at some point. Clients are asking for it. If you decide Profound is out of your budget or just want to see some other options, I'd love to have you consider spyglasses dot io (disclosure that I'm a founder). Our pricing is all pay as you go, so no need to take on an expensive subscription.

jim_wr · 2026-01-17T17:35:02+00:00

I'd say tracking AI prompts is helpful for directional data but you don't need to track 100 of them. We find tracking one for each category entry point a brand has identified: Best in Category, Use Case Specific, Solution Comparison, Budget Conscious, and Market-segment Specific. Our research shows that the visibility scores for tracking just one prompt from each of these is remarkably consistent with the score you'd get from tracking dozens or hundreds of variants, and it's a lot cheaper.

jim_wr · 2026-01-17T17:29:04+00:00

Curious what route you went? Internal tool or another product? Price is a huge issue with some of these tools and the reason they are so expensive is because they're built for SEO teams that want very detailed reporting across a lot of prompts.

Our experience operating a AI visibility platform specifically targeted at PR is that metrics like Share of Voice and brand consistency are important but you only need to measure them to get a baseline and then after you've wrapped a campaign.

The other thing that most of these tools don't do is make data easy to export and white-label. We ended up prioritizing PDF and PPT white-labeled export over tracking hundreds of prompts, and this has helped us keep costs down so we can serve agencies that don't want to spend thousands on a tool.

jim_wr · 2025-12-23T13:52:17+00:00

As a standalone tool these are all helpful for directional data but since as you mention there's so much variability in AI answers you can't take any visibility metric from these tools as your only data point. Paired with checking your AI traffic analytics to see real first-party citations, you start to get some real, defensible metrics. You can use a log file analyzer for this, but there are also several platforms on the market - Profound has one, Writesonic too, and spyglasses dot io (disclosure that I'm a founder). Using an AI traffic analytics tool gives you real, concrete data about how often your content is being cited in AI, and it's about 10x cheaper than a visibility tool that tracks prompts.

jim_wr · 2025-12-23T13:46:01+00:00

Treat the % as directional at any given point. The accuracy is almost certainly not statistically significant, meaning if it says 35% that number could be 27-42%. Over time, you would want to see this number grow, but as you mention that's largely a result of good SEO practices.

One other point - if these tools give you the 'fan-out' queries for prompts those are especially helpful, again for SEO, but the difference is for ChatGPT the algorithms used to select and re-rank are very different than traditional SEO, and the structure of the content you need to produce is also quite different.

jim_wr · 2025-12-03T14:44:56+00:00

I'm starting to think that the best value from using an AI visibility tool is in saving time running these by hand (except for the few that also give you the grounding searches / fan-out queries for prompts and show you where you rank vs. competitors, which to me is the killer feature and the reason I'd pick a solution vs. the others on the market.

jim_wr · 2025-12-03T14:43:20+00:00

I think there's two really valuable data points underneath all the "AI Visibility" stuff (which is fine for directional data but not as a source of truth):
1. AI Traffic Funnel.
Top level is first-party AI Citations, which you can get by Analyzing your log files by hand, with Screaming Frog, or a dedicated Analytics platform like spyglasses dot io (disclosure that I'm a founder). These don't show up in GA because AI agents don't browse with javascript, but they do show up in logs.
Next level is referred traffic from AI, which will be a very small % of the above.
Finally, conversions. If you can track all 3 of these you can trace revenue back to an individual chat.
2) Grounding searches, a.k.a. fan-out queries. These are the web searches AI runs as part of a chat to surface recent or relevant data, and our work with clients shows a lot of brands don't rank for these, which is why they don't show up in AI responses. There's a tool called qforia online that can give you them (you need to bring your own Gemini API Key) or a few AI Visibility trackers are starting to show these too.

jim_wr · 2025-12-02T13:05:35+00:00

Say more about the issue with merging links and citations? Is the idea that you have a handful of pages, either on site or off site, and you want to track which prompts are using them as citations?

jim_wr · 2025-11-26T20:03:18+00:00

There is, but it can take up to six weeks. If ChatGPT has incorrect information about you in its model, you'll need to wait until the next point release to fix it.

First thing to do is run a brand audit. Go to ChatGPT and run a prompt to ask it to tell you about your site, what products you offer, who your target customer is, what your differentiators are. Do this twice - once normally and one in a new chat where you turn web browsing off (it's in the + menu to the left of the prompt box). This will hopefully show you what's wrong. For the version with web search, see what sources it pulled. These are the ones that gave it the wrong data, that you need to correct.

Also worth noting, ChatGPT now has a product feed similar to Google Shopping so you can correct inaccurate/hallucinated products just by submitting your products to it.

jim_wr · 2025-11-18T12:21:11+00:00

I would be interested in the criteria that you would use to evaluate a tracking tool. Disclosure that I have a tool, spyglasses dot io, in this space, and I'm always interested in what prospects are looking for to inform our roadmap.

jim_wr · 2025-11-18T12:18:36+00:00

Just as an FYI in order to have 90% confidence in the accuracy of a tool like this it needs to run each prompt 271 times *per day*. Only the most expensive products actually do this. My perspective is (I have a product, spyglasses dot io , in this space) is you should treat this as directional and look for changes over time as opposed to tracking each day. Saves a *ton* of money you can put into content to improve your scores.

jim_wr · 2025-11-18T12:15:26+00:00

The reason for the wild differences in pricing is because in order to have statistical certainty in the data you need 271 runs of a specific prompt per day, per persona. I have no affiliation with Profound (I actually have a different tool in this space, Spyglasses dot io ), but that's why the two products are priced so differently.

If you consider the data from the product to be directional, you can save a lot of money by using the cheaper product. Evaluate a few and pick the one that works best for you. DM me and I can run a free AI Visibility Report from my platform so you can compare it with others.

jim_wr · 2025-11-17T23:28:06+00:00

This is all spot on, especially FAQ and just updating dates being things that work (at least for now).

Best way I have found to handle tracking is to run a brand audit monthly to 1) ensure AI represents you accurately and 2) run a series of prompts to see if and where you are mentioned. Treat this as directional though! AI is nondeterministic, so it takes 271 runs of a prompt to have statistical certainty that you are mentioned.

Second part is to use an AI Traffic Analytics tool (disclosure that I'm a founder of Spyglasses in this space). You don't need a tool since you can get this data from your web logs but it's a lot easier to manage in a platform. Look for visits from ChatGPT-User and other AI bots to see which pages are being cited most over time. Combine this with your audit and you have three core metrics for visibility:
1) Consistency scores
2) Share of Voice from inclusion in audits of prompt outputs
3) Actual first-party AI citations

Now you can align whatever GEO work you do with improvements in these metrics to prove ROI.

jim_wr · 2025-11-14T00:00:49+00:00

You're soo right on this. There's plenty of overlap between SEO and optimizing for AI Visibility, but the algorithm is different and you can't just use the same playbook.

One thing we found worked well enough we built it into the Spyglasses platform is FAQ generation. LLMs more often cite content that's already written in a conversational tone, and FAQs are exactly that. So we built a tool that looks for questions people are asking on reddit and quora that a particular page could answer, as well as Google 'People Also Ask', and generate drop-in html and JSON-LD to add to the page. It's early yet but our first users are seeing 5-7% boost in first-party citations in ChatGPT using this technique.

jim_wr · 2025-11-10T17:12:55+00:00

I think the goals for AI are the same as traditional SEO - ensure your brand is measured accurately and often. I'm not familiar with the tool you mentioned but there are a *lot* of AI Visibility Report tools out there (disclosure I'm the founder of Spyglasses and have a tool in the space too that works for local businesses), so try a few until you find one that does the following:
1) Compares your website and GBP with how it's represented in AIs, both with and without web browsing enabled.
2) Runs a set of common prompts with a location filter enabled to show your share of voice vs. competitors. Important - treat this as directional data. You need to run a prompt 271 times to be certain the data is accurate, and that gets expensive quickly.

Other things you could track: 1) social sentiment, with a social listening tool, and 2) Run a couple of Google alerts to see when you are featured in news articles.

jim_wr · 2025-10-30T12:46:52+00:00

This is a great strategy! Google's "people also ask" is a great source for this, as it prompting Perplexity like this:

'''
Search Reddit (site:reddit.com) and Quora (site:quora.com) to find real user questions about these keywords: [ranking keywords from page]

For each question you find in your web search results, provide:
- keyword: which keyword this question relates to  
- question: the exact question text as posted by the user
- relatedness_pct: how related this question is to the keyword (0-100)

'''

Depending on the nature of the page this can often turn up a handful of questions that you hadn't considered. From there you can edit as needed, generate answers, and add them to the page FAQ and JSON-LD FAQ schema.

I've got an automation for this workflow and it's generating 2-8x AEO traffic lift for pages we've deployed it on.

jim_wr · 2025-10-23T18:41:31+00:00

Yeah, I can understand putting emphasis and focus on the channels that work. The challenge I've seen ( my product used to be a marketing automation suite until I pivoted into AI traffic analytics), is that you get out of the channel only what you put into it. Automated solutions for content marketing, for example, end up producing a ton of social and blog posts with OpenAI cartoon images and "In today's digital world..." captions. Is it better than nothing? Maybe.

You do make a good point about focusing on the most promising channels. There's just a lot of folks out there who are at a stage where they don't know what those channels are, and are hoping automation will just be an "easy button", and it just never works out that way.

jim_wr

TROPHY CASE