An AI assistant can name a brand in one answer and leave it out of the next, even when you submit the same prompt. That makes a single answer useful evidence of what happened once. It takes a set of comparable answers to describe how often it happens.
The prompt is only part of the situation. The assistant's model, search mode, conversation history, location, retrieved sources and available products can also affect an answer. Keeping the wording unchanged does not establish that all those conditions stayed fixed.
Start with a question a real buyer would ask. Keep the exact wording and record the assistant, mode, date and relevant context alongside the full response. Use a fresh conversation when that is part of the test design. Separate a completed answer that omits the brand from a failed request or a run that produced no answer.
Then report a numerator and denominator. In an illustrative sample, eight brand appearances in 20 valid answers is a 40% observed appearance rate. Twelve in a comparable sample of 20 is 60%, a difference of 20 percentage points. Those figures describe the samples; they do not by themselves prove that a site change caused an improvement or that the underlying rate is now exactly 60%.
Repeated questions in one sitting can reveal variation within that sitting. Observations across days can reveal variation over time. Both are useful, but they answer different questions, and closely related observations may provide less independent information than their count suggests. There is no universal number of repetitions that makes every comparison reliable.
The sampling example shows this distinction using a historical set of recorded answers. Its practical lesson is to retain the observations and their context, then make the strength of the conclusion match the evidence.
A useful update to a colleague sounds like this: “The brand appeared in 12 of 20 answers this week, compared with eight of 20 under the same test setup last week. We will check whether the pattern persists.” That is a claim someone else can inspect and a sensible basis for the next decision.
Common questions
My brand appears in half the answers. Did visibility drop?
That result alone cannot tell you. Compare it with an earlier sample collected under equivalent conditions, and keep the sample sizes and uncertainty in view.
Does repeating the same question teach us anything?
Yes. It can show how variable answers are within the conditions you tested. It does not automatically establish how answers vary across other days, users or assistant modes.
Can a brand be permanently in or out of an assistant's answers?
An observed streak does not establish permanence. Sources, products, models and user context can change, so describe the period and conditions in which the streak occurred.
Should a failed request count as a missing brand?
No. A completed answer that leaves the brand out is an observed omission; a failed request provides no answer to evaluate. Record the failure separately so it does not silently become a zero.