Visa published the first results of its Agentic Commerce Benchmark on November 5, 2026. It gave 22 AI models from 12 labs 15 shopping tasks, such as buying a TV, a laptop or a personal care appliance, with three attempts at each. A run passed when the model uncovered the customer's requirements through conversation and then bought a product that met them, or declined to buy when nothing fit. Across all models, 57.2% of runs passed. The best models passed 82.2% and the lowest scorer 28.9%.
The failures are what reach a store. In 54% of failed runs the model still made a purchase, either of a product that missed the customer's stated requirements or after missing some of those requirements. For a merchant, that is a sale made against the wrong brief, the kind of order that can come back as a return.
Asking questions separated the models. The five that uncovered the most hidden requirements passed 70.2% of runs, and the five that uncovered the fewest passed 44.4%. Price helped less: the five most expensive models passed 67.1% of runs, against 50.2% for the five cheapest. All three attempts at a scenario passed 34.2% of the time.
Visa's TV scenario shows what a model has to find out. The customer mentions a brand a relative recommended and a membership with a third-party supplier, and the model must uncover the budget, format compatibility, HDR10+ support for family videos and gaming features. A model can match requirements like those only against product details a store has written down, so the specifications on a product page are the part of this test a merchant controls. Pick one product your customers choose by specification and check that every attribute a buyer would ask about is stated on its page in plain words.
Visa said it will add shopping scenarios and product categories and extend the benchmark to checkout, payment and order placement.
Common questions
What did Visa's Agentic Commerce Benchmark test?
Twenty-two AI models from 12 labs each took on 15 shopping challenges, such as buying a TV, a laptop or a personal care appliance, with three attempts per challenge. A run passed when the model uncovered the customer's requirements in conversation and made the right decision to buy a fitting product or not to buy at all.
How often did AI models pass?
Across all models, 57.2% of runs passed. The best models passed 82.2% of runs and the lowest scorer 28.9%. All three attempts at the same scenario passed 34.2% of the time.
What happened when a run failed?
In 54% of failed runs the model still made a purchase, either of a product that did not meet the customer's requirements or after missing some of them.
Which models did better?
Models that asked more questions. The five that uncovered the most hidden requirements passed 70.2% of runs, against 44.4% for the five that uncovered the fewest. The five most expensive models passed 67.1%, against 50.2% for the five cheapest.