Most teams are measuring AI visibility wrong.
I noticed it pretty quickly when I started paying attention to how people were talking about AI Search visibility. Everyone was doing the same thing:
Type a question into ChatGPT. See what comes up. Take a screenshot. Repeat a few times. Call it research.
I get why. It’s fast, it’s easy, and it feels like data. But it’s not. Not really.
AI Search isn’t deterministic. Ask the same question multiple times and you’ll get different answers. Change the phrasing slightly and the results shift again. Which means small sample sizes are misleading, random prompts create bias, and one-off screenshots prove nothing.
That’s what Signals Over Noise is trying to fix.
- A handful of prompts
- A few screenshots
- A lot of assumptions
- One-off results treated as patterns
It looks like data. It feels like insight. But it’s not reliable. It’s noise.
AI Search isn’t deterministic. So your measurement method can’t be either.
The problem with testing a few prompts isn’t just that the sample is small. It’s that the results you’re seeing are probability outcomes — and you’re treating them like facts.
Results aren’t stable
Ask the same question five times and you might get five different answers. The model isn’t broken — that’s just how probabilistic systems work. A single result tells you almost nothing.
Random prompts create bias
The queries you think to test aren’t necessarily the queries your audience is actually using. You’re measuring visibility for your prompts, not theirs. That’s a significant gap.
Screenshots aren’t strategy
A screenshot shows you what happened once. It doesn’t show you what happens consistently, how often you appear, or whether things are getting better or worse.
Persona → Entity → Sampling → Mention Rate
This is the system I developed to get reliable data from AI Search. Each stage feeds the next. The goal isn’t just to see what shows up — it’s to understand how consistently you show up, under what conditions, and whether that’s changing.
How your audience actually thinks, searches, and decides
Before you can measure visibility, you need to know what you’re measuring visibility for. Not the queries you’d write — the queries your audience actually uses. That distinction matters a lot more than most people realize.
AI systems surface results based on how real people ask real questions. If your query set doesn’t reflect actual search behavior, you’re measuring the wrong thing from the start.
What you want to be known for, clearly defined
Your topics, expertise areas, services, locations — whatever is most important for your brand to be associated with. These become the anchors for every query set you build.
AI systems think in entities and relationships. The clearer your entity definition, the more precisely you can measure whether AI actually understands and recognizes you for the right things.
Structured query sets with enough runs to see patterns
This is where most measurement approaches fall apart. Not enough queries, not enough runs, no consistency. You need a sample size that gives you signal, not just noise.
- 20 × 5Directional signal — good for a first look
- 40 × 10Validated trends — reliable enough to act on
- 80 × 30Decision-grade data — confident benchmarking
We increase runs for confidence, not coverage.
How often you appear, not just whether you appeared once
Mention rate is the core metric — the percentage of AI responses, across your full query set and all runs, that include your brand. It’s a number you can track over time and actually use.
Not “we showed up in 3 screenshots.” But “we appeared in 34% of responses across 400 data points.” That’s something you can benchmark, improve, and report.
Mention Rate Tool
I built this because doing all of the above by hand was taking forever. It automates the structured sampling, tracks mention rates across AI platforms, and gives you actual data — without the spreadsheet nightmare.
- Structured query sets built around your entities
- Consistent sampling across AI platforms
- Mention rate scoring you can track over time
- Actual data — not screenshots
This is a measurement system. Not a hack.
I want to be honest about what Signals Over Noise is and isn’t. There’s a lot of noise out there about AI visibility — a lot of shortcuts, hacks, and “just do this one thing” advice. This isn’t that.
- Prompt engineering
- Testing a few queries and calling it research
- Screenshots as strategy
- Gaming AI responses
- A one-time audit with no follow-up
- A repeatable measurement methodology
- Structured sampling with defined sample sizes
- Data you can track and compare over time
- A framework for actually understanding AI visibility
- Something you can build a real strategy on
From guessing to actually knowing.
This is what changes when you have real data instead of assumptions.
Confidence in your visibility data
You know what you’re measuring, how you measured it, and why the sample size is large enough to trust. No more “well, it showed up when I tested it.”
Clarity on where you actually stand
A mention rate tells you exactly where you are — and because it’s consistent and repeatable, you can compare it to a previous period and see whether things are moving.
Direction for what to fix and improve
When you break down mention rate by entity, persona, or query type, patterns emerge. You can see where visibility is strong, where it’s weak, and where the opportunity is.
A way to track progress over time
You ran a baseline. You made changes. You ran again. Did mention rate go up? That’s how you know whether what you’re doing is actually working.
Signals Over Noise connects to everything else on this site.
Discover → Structure → Reinforce → Measure
Signals Over Noise is the measurement layer of the framework. It’s how you know whether the rest of it is working.
Field notes, experiments & observations
I’m learning by running this method on real data and documenting my learning as I go.
Built to make this method usable
The Mention Rate Tool and Schema Generator both came directly out of trying to run this system efficiently.