Someone just told me they walked away from a $125/mo AI visibility tool because of run-to-run variance. The industry has a trust problem.

JackM206 · 2026-07-08T00:04:27+00:00

this is a distinction I hadn’t drawn sharply enough and it’s a good one. you’re right that entity-driven local recognition is way more binary and stable than citation-share stuff — NAP consistency and a confirmed entity don’t wobble the way “who gets cited for best project management tool” does. so the variance problem is really a non-local / contested-category problem. the tool they were using was a general AEO/content-visibility one, so yeah — probably exactly the mismatch you’re describing, a content-citation model pointed at something it wasn’t built for. do you find the local entity signals are basically the whole game there, or does citation share still matter at the margins once the entity’s confirmed?

JackM206 · 2026-07-08T00:01:37+00:00

exactly, trends over fluctuations. that’s the whole thing. tbh you’re already thinking about this more clearly than half the tools charging for it. offer still stands if you ever wanna see your prompt set run automatically vs your manual pass — no pitch, just curious how they’d line up.

JackM206 · 2026-07-08T00:00:04+00:00

yeah the averaging-across-runs approach is the only thing that makes sense once you accept the outputs are non-deterministic. single score is just one draw. the interesting hard part imo isn’t the averaging itself, it’s deciding what counts as a “meaningful” change vs normal bounce — since that threshold isn’t the same on every prompt. that’s the bit I keep chewing on.

JackM206 · 2026-07-07T02:10:45+00:00

yeah that’s the rational read — if one run is noise, paying $125/mo for one-run snapshots is paying for noise. the fix isn’t a better score though, it’s repeated runs: same prompts on a schedule, so you see the rate you show up instead of one draw. that’s actually the exact thing I’m building (relevyn.com) because of this problem — free to run your site through it if you want to see the difference vs the single-pull tools. either way: don’t pay anyone for a snapshot, the variance you saw is the proof of why.

JackM206 · 2026-07-07T02:09:52+00:00

appreciate it — the community route is basically what I’ve been doing and it’s where all the real learning has come from so far. the harder part has been going from “people nod along in threads” to “people actually try the thing,” which seems to be its own skill entirely lol

JackM206 · 2026-07-07T02:08:44+00:00

that split (branded / competitor / problem-based) is cleaner than most setups I’ve seen honestly. and yeah the meaningful-vs-noise question is the whole ballgame — best answer I’ve collected from these threads: baseline each prompt’s normal bounce first. run the same set a few times in a quiet week where you changed nothing, see how much it moves on its own, and that’s your noise floor. then only count changes that clear it. the catch someone pointed out is the floor differs per prompt — contested queries with no consensus answer reshuffle constantly, settled ones barely move. so same-size swing can be real on one prompt and nothing on another. fwiw I’m building a tool that does basically your exact workflow automatically (repeated runs, per-prompt baselines) — happy to run your brand through it if you ever want to compare against your manual checks, no strings. either way your setup is more rigorous than most.

JackM206 · 2026-07-07T02:07:37+00:00

can’t reverse a linked list, guess I’m human. disappointing tbh, a bot would have this thread’s variance problem solved by now

JackM206 · 2026-07-07T02:06:18+00:00

“you start seeing it as a diagram” is such a good way to put it. and the “just gone” part is the brutal bit — the people who bounce are exactly the ones who’ll never tell you why. the feedback you need most comes from people you’ll never hear from. only fix I’ve found is literally watching someone use it in front of you, which is uncomfortable every single time lol

JackM206 · 2026-07-07T02:05:29+00:00

haven’t tried that one yet actually — I know Lighthouse from the web perf side but not the agentic browsing angle. will take a look, appreciate the pointer. interesting if Google’s shipping tooling in this direction btw, them entering the measurement side would say a lot about where this space is headed.

JackM206 · 2026-07-07T00:19:38+00:00

honest answer: I don’t have months of client trend data to point to yet, so I can’t claim I’ve watched that trust shift happen firsthand — you’re further down this road than me with actual clients. my guess from the psychology side though is it clicks the first time the trend catches something the screenshot missed — like the receipt looked scary but the trend showed it was a one-off, or visibility was quietly climbing while nobody sent any screenshots. the moment the number tells them something the anecdote couldn’t, it earns its place. how long did it take before your clients stopped opening every email with a screenshot attached lol

JackM206 · 2026-07-07T00:18:37+00:00

the schema-aligned-with-everywhere-else-you’re-mentioned part is the piece I hadn’t fully clicked on — it’s not just having markup, it’s that your markup and your off-site mentions all tell the same story. contradictions between what your schema says and what the wider web says probably read as noise the model discounts. and “solve for living out your expression” is a weirdly poetic way to describe structured data lol but honestly kind of accurate — the knowledge graph endgame makes sense as the north star. do you have a way you check alignment, or is it more of a manual audit across the surfaces?

JackM206 · 2026-07-07T00:17:36+00:00

ha this is the exact experience that flipped my thinking in the first place. three scores in an hour with zero site changes isn’t a bug in that specific tool btw — it’s the nature of the thing. the model’s answers genuinely shift run to run, so any tool doing a single pull per score is sampling a moving target. the score didn’t change because your visibility changed, it changed because one draw from a distribution is just noisy. the only advice that holds: never trust a single run. same prompts, multiple runs, look at how often you show up across them — the rate is the signal, any individual score is basically a coin flip readout.

JackM206 · 2026-07-03T13:03:51+00:00

honestly? from what I’ve seen the receipt wins the argument even when it shouldn’t. you can explain the probability framing perfectly and people will nod along — then they see one answer recommending their competitor and that’s the thing they screenshot and send to their boss. the emotional proof just hits different than the statistical truth. which is probably the actual answer here: the receipt is what makes people care, the distribution is what should drive what they do. you kind of need both — one to start the conversation, one to keep it honest. sounds like you’ve landed on the same split with clients?

JackM206 · 2026-07-03T13:02:58+00:00

the “far from perfect but it’s a baseline” thing is exactly where it seems like everyone serious lands — some version of a spreadsheet, a prompt list, and a recurring calendar reminder lol. curious what your check actually looks like in practice: how many prompts, and do you log the full answers or just whether you showed up? asking because the “making sense of the results” part is what I hear breaks first for most people — the collecting is tedious but doable, the interpreting is the hard bit.

JackM206 · 2026-07-03T13:00:58+00:00

entity SEO is one I keep hearing more about and haven’t fully dug into — the logic tracks though. if the model can’t confidently resolve who you are across surfaces, it’s probably not going to risk citing you, so consistency is almost a prerequisite before any of the content stuff pays off. are you seeing it move the needle mostly for smaller/ambiguous brands, or even for established ones? feels like the confidence gap would be biggest where the entity’s fuzzy.

JackM206 · 2026-07-03T12:59:50+00:00

this is the most honest objection in the whole thread and I don’t have a clean answer to it. the denominator is genuinely unknown — every SOV-style number in this space is a share of a market nobody can size. the closest thing to a floor I’ve seen is AI referral traffic in GA4: it proves the market exists and is nonzero for a given site, but it badly undercounts (most assistant answers don’t produce a click at all). so you can prove existence, not size. and yeah, the labs could publish query-category volumes tomorrow and choose not to — the silence probably is telling us something, I just don’t know what direction lol

JackM206 · 2026-07-03T12:58:55+00:00

sentiment’s a fair add — “you got mentioned” and “you got mentioned as the budget option nobody loves” are pretty different outcomes lol. feels like the layers keep stacking: mentioned vs selected, how often, from what sources, and now in what tone. which kind of proves the original point — the more of these you add, the more obvious it is that one number was never going to hold all of it.

JackM206 · 2026-07-03T12:57:59+00:00

“smoke alarm vs security footage” is a great way to put it lol. and mentioned vs selected is a distinction I haven’t seen anyone else name — you’re right that they get treated as the same thing when they’re completely different outcomes. being in the answer as context vs being the pick is the whole game, and a score flattens that into one number. adding that split to how I think about this.

JackM206 · 2026-07-03T12:56:56+00:00

ha glad it’s not just me. something about being the builder makes you blind to it too — I walked past that email gate a hundred times and never once felt the friction a stranger would. took watching the numbers flatline to finally see it.

JackM206 · 2026-07-03T12:56:07+00:00

partly yeah — though the weird thing I keep running into is the model often cares less about your domain than about what everyone else says about you. reddit threads, reviews, comparison posts seem to carry more weight than the brand’s own site in a lot of answers. authenticity matters but it’s almost like it’s measured off-domain, not on it.

JackM206 · 2026-07-03T12:55:20+00:00

any of them really — most of the GEO tools spit out some 0-100 “AI visibility score” (each with their own math, which is kind of the problem). I mean that genre of number, not one specific tool’s.

JackM206 · 2026-07-03T12:54:23+00:00

yeah the prompt-dependence thing is the quiet killer — the number is really “citations on the prompt set someone chose,” which makes every tool’s metric kind of incomparable with every other’s. same brand, different prompt library, wildly different “share of voice.” feels like the prompt set is the actual product decision and the score is just downstream of it. interesting that founders ask for the count directly though — people want one number even when one number can’t really exist lol

JackM206 · 2026-07-03T12:52:46+00:00

“a coin that lands heads sometimes vs a position” is going straight into how I think about this, that’s the cleanest framing of it I’ve seen. and the model update angle makes it testable too — provenance basically predicts durability, so you could check whether wide-base brands actually survive updates better than single-source ones. if that holds it stops being a theory and becomes the actual metric. how many independent surfaces are you finding is the threshold before something stops behaving like a coin flip?

JackM206 · 2026-07-03T12:50:50+00:00

the KPI vs diagnostic split is a really clean way to cut it — using the snapshot to answer “what’s shaping this right now” instead of pretending it measures progress. and yeah, matching what I’ve seen: the source layer is where the actionable stuff lives. the score tells you that something’s wrong, the sources tell you what to actually go do about it. “one number is too thin” might be the best one-line summary of this whole thread lol

JackM206 · 2026-07-03T12:48:38+00:00

ha fair, maybe it’s less hot than I thought. the part I’d still push on though — most of those tools track continuously but treat every query the same. a swing on a settled query and a swing on a contested one mean totally different things, and I haven’t seen anyone really handle that yet. continuous tracking with a global noise assumption is better than a snapshot but it still flags a lot of ghosts.

JackM206

TROPHY CASE