AI visibility, recommendation quality and revenue describe different stages of discovery. A brand can appear often without being a good match for the request. A favorable answer can help a buyer without producing an observable referral. A sale can occur after several influences.
Define each outcome separately
Question | Evidence to retain | Limit |
|---|---|---|
Did the brand appear? | Complete answer and a checked brand mention | An appearance can be incidental or negative |
Was it recommended for the task? | Relevant passage, buyer constraints and review rubric | A favorable tone alone does not establish suitability |
Did the buyer visit? | A visit under a declared referral or attribution definition | Some influenced visits have no identifiable AI referrer |
Did the buyer purchase? | Order linked under the stated measurement method | An attributed purchase does not automatically prove causation |
Keep missing sentiment or unknown attribution separate from a negative recommendation or zero revenue. Those are different observations.
Use a rubric a second reviewer can apply
For a product recommendation, check whether the answer addresses the requested budget, use case, availability and material constraints. Record the evidence for each judgment. A glowing description of an unavailable item should not receive the same task-success label as a purchasable option that meets the request.
Read a selection of the actual answers with another reviewer. Discuss disagreements about whether a recommendation fits the buyer’s request before using the result to choose an action.
Keep rates and counts together
Consider this illustrative referral report:
Period | Identified AI-referral sessions | Orders attributed under the same rule | Session conversion rate |
|---|---|---|---|
Before | 100 | 4 | 4% |
After | 200 | 6 | 3% |
Attributed orders increased from four to six while the session conversion rate fell from 4% to 3%. A claim that “conversion improved” would be wrong for this metric; a claim that “attributed orders increased” would describe the table. Neither establishes which change caused the outcome.
Also identify whether revenue is gross, net of returns or another measure. A change in order count and a change in revenue can point in different directions when order values change.
Match the comparison to the claim
If the claim concerns visibility and conversion, the analysis needs both variables on a comparable unit, such as brand-period observations. A plot of visibility against favorability cannot by itself show either metric's relationship with conversion.
Keep date windows, brands and measurement settings aligned. Account for promotions, inventory changes, seasonality, traffic mix and site changes when interpreting the result. Segment results where a pooled average would conceal important differences, and report each segment's sample size.
A correlation can identify a relationship worth investigating. It does not establish that increasing the score will increase revenue, nor that the same relationship holds in every category.
Design a useful intervention
Choose one specific change and define success before looking at the result. For example, a merchant could correct missing size information on a selected group of product pages and compare relevant outcomes with a suitable unchanged group. Keep price and availability changes in the record, and explain why the comparison group is credible.
Where random assignment is feasible, it strengthens the design. Where it is not, describe the study as observational and state the remaining alternative explanations. Report an uncertain result as uncertain rather than selecting a favorable metric afterward.
Choose the action from the diagnosed problem
Low appearance rates call for examining coverage of relevant buyer questions and the sources assistants can use. Frequent appearances with poor task fit call for inspecting inaccurate claims, missing constraints or unfavorable evidence. Visits without purchases call for investigating the offer and buying experience as well as the recommendation.
That sequence keeps the work tied to a buyer outcome. The AI visibility guide helps interpret the answers, and the benchmark guide helps assess the relevance of a comparison.