Why Do Two Audit Readings Differ When Nothing Changed?
Artificial intelligence (AI) engines do not return the same answer twice. SparkToro and Gumshoe.ai research ran each prompt 60 to 100 times per platform, and nearly every response was unique in its brand list, its order and its length.
The category itself stayed recognisable. When 142 people wrote their own headphone prompts, Bose, Sony, Sennheiser and Apple still appeared in 55% to 77% of 994 responses.
| What stayed steady across runs | What moved between runs |
|---|---|
| The set of brands an engine considers for a category | The exact brand list |
| How often a leading brand appears across many runs | The order of the brands |
| The number of brands returned |
This is why a generative engine optimisation (GEO) programme reports a rate across many runs. The research was run with Gumshoe.ai, a vendor of AI tracking tools, and is not peer reviewed.
How Far Apart Do Two Readings Need to Be?
The gap must be wider than the margin of error on each reading. A preprint from IQRush shows why. It sampled 200 queries on OpenAI's SearchGPT for running gear and put 95% intervals around each source's citation share. The two leading sources looked different on a dashboard, but their intervals overlapped.
| Source | Citation share | 95% interval |
|---|---|---|
| Tom's Guide | about 9.5% | about 5.5% to 12.5% |
| Runner's World | about 6.0% | about 4.0% to 8.0% |
The same logic applies to your own results. The paper's own example is a SearchGPT domain moving from 8% to 11%. That gain cannot be credited to a content change, because it sits inside the typical interval.
- Gap wider than the margin: treat the change as real, and note the date it began.
- Gap narrower than the margin: hold judgement, and add runs or lengthen the window.
- Engines differ: to narrow citation-share intervals to five points, the paper needed about 40 to 50 queries on Gemini, about 100 on Perplexity and 150 or more on SearchGPT.
The author works at IQRush, which sells AI visibility measurement. The study covered three consumer topics over nine days with ChatGPT-written queries, and business topics were not tested. Treat the numbers as the shape of the problem.
How Long a Window Does Each Reading Need?
Two to four weeks. The University of St Gallen tracked four AI search engines for 45 to 46 days. Its preprint found that cited sources overlapped by only 34% to 42% between consecutive days. A one-day or one-week reading mostly measures that churn, so the authors recommend rolling windows of two to four weeks.
| Window length | Margin of error on one brand's mention rate (95%) |
|---|---|
| 7 days | plus or minus 26 points |
| 14 days | plus or minus 16 points |
| 21 days | plus or minus 10.5 points |
| 28 days | plus or minus 6.5 points |
Mention rate here means the share of runs in which a brand is named. The prompts were Swiss-German consumer queries, eight per vertical, so expect different margins on business-to-business (B2B) topics. Run count per prompt matters as well, and our guide to what an AI citation audit costs covers the thresholds.
What Should Stay the Same Between Two Readings?
Everything you control, which means fixing the prompt set, the engines, the collection method and the location before the first run. Then track each engine as its own series. Ahrefs found that Google's AI Mode and AI Overviews cited the same pages just 13.7% of the time for the same query, so blending the two hides real movement.
| Hold constant | What goes wrong if it drifts |
|---|---|
| Prompt wording | A reworded prompt gets a different answer, so the change reflects the prompt |
| Engine and Google surface | A blended score moves when one engine moves, and masks the others |
| Collection method | Switching between manual checks and automated collection changes what is counted |
| Location and language | Answers shift by market, so a new location starts a new series |
Add new prompts as a separate set. A cross-engine AI citation audit keeps the original set intact and reports each engine on its own line. Ahrefs sells search optimisation software.
How Do You Confirm a Change Is Real Before Acting on It?
Check a second, independent signal. Google launched dedicated Search Console reports on 3 June 2026. They show how often your pages appeared in AI Overviews and AI Mode, by page and date. A real gain in your tracked prompts should show there within the same weeks.
- Search Console impressions: covers Google's AI surfaces only, and reports impressions for pages by date.
- Referral visits: covers the other engines, as visits arriving from AI assistants.
- A dated change log: site changes, robots.txt edits and engine updates, so timing can be matched to the movement.
If the change is a drop, check crawler access first. Our checklist of what to fix first lists the access checks in order.
What Should the First Two Weeks of Tracking Look Like?
Freeze a prompt set and run it for two weeks before touching a page. That gives you a first window and its margin. Change one thing, run a second window, and compare the two.
Find out whether your AI visibility movement is real or noise. Scope an AI visibility audit through our answer engine optimisation service.
Content Creation
Find out whether your AI visibility movement is real or noise. Scope an AI visibility audit with Intelligent Resourcing.
FAQs
How many points does an AI visibility score need to move to count?
There is no fixed number. It depends on the window length and the runs per prompt. A longer window narrows the margin, so a smaller gap can count. A short window needs a large gap before the change means anything.
Can one bad week mean visibility has really dropped?
Usually not. Answers shift day to day for reasons that have nothing to do with your site. A drop counts when it persists across a full window. A sudden fall to zero after a site change is the exception, and should be investigated at once.
Should AI Overviews and AI Mode be tracked together?
No. They are separate Google surfaces that often cite different sources. Track each as its own series. A blended score hides which surface moved and which did not.
What does "not enough data" mean in an AI audit?
It means the gap between two readings is smaller than the margin of error. The honest answer is to extend the window or add runs. Reporting a winner anyway turns chance variation into a false result.
Does a rising score prove a content fix worked?
Not on its own. The gap must beat the margin, the timing must follow the fix, and a second signal should agree. Engine updates and competitor activity can move a score without any change on your site.