Visa launched its Agentic Commerce Benchmark November 5: shopping agents failed 42.8% of runs, roughly 43%. Worse, failure didn’t necessarily stop a purchase.
The test covered 22 models from 12 labs across 15 challenges involving televisions, laptops, and personal care appliances. According to Visa’s results, the overall pass rate was 57.2%. The best-performing models passed 82.2% of attempts; the worst passed 28.9%.
Google and Meta agents shared first; Claude ranked third at 76%, OpenAI sixth at 64%, Fortune reported.
According to Visa, agents had to uncover every task-defined customer requirement through conversation, then correctly decide whether to buy. The simulated customer was another agent, volunteering some information while withholding other requirements, Fortune reported.
If nothing fitted, declining the purchase was the right answer. A lucky guess did not count.
The uncomfortable bit: 54% of failed runs still ended in a purchase, Visa says. Those agents missed identified requirements or bought before uncovering everything.
Visa’s five strongest requirement-discovering models averaged 70.2%; its five weakest, 44.4%. The five highest-cost models passed 67.1%; the five cheapest, 50.2%.
Television requirements included budget, format compatibility, HDR10+ for family videos, and gaming needs. Fortune also described Dolby Vision and a 30-day return window.
Each model received three attempts per scenario, but all three passed in only 34.2% of model-scenario combinations, according to Visa.
Selection isn’t checkout
Checkout, payment, order placement, additional scenarios, and product categories are planned extensions. Visa has shared the benchmark with leading model providers for feedback; replays and an interactive version are public.
“We’re not there yet from a trust or technology perspective,” Rubail Birwadker, Visa’s global head of growth, told Fortune.
Visa’s September 2026 Trust Index found that 72% of U.S. consumers had used an AI assistant, but only 23% trusted AI to handle payments on their behalf, according to the company’s benchmark announcement.
For founder-led retailers considering what shopping agents mean for their storefronts, the practical recommendation is precise product information over automatic-discovery promises. Spell out features, compatibility, intended use, price, and return policies; make limitations as clear as selling points.
That’s “an inference from the benchmark, not a merchant-performance finding.” Visa’s test establishes neither retailers winning or losing customers nor listing rewrites improving recommendations.
“Clear listings could help an assistant evaluate a match; they cannot make it ask the missing question.”
Sources
- https://usa.visa.com/about-visa/newsroom/press-releases.releaseId.22936.html
- https://fortune.com/2026/11/05/ai-agents-shopping-ranking-best-agent/
- https://thenextweb.com/news/visa-ai-shopping-benchmark-agentic-commerce-ai-models-fail
- https://www.digitaltransactions.net/visa-preps-agentic-commerce-benchmark-and-forecasts-a-4-5-holiday-spending-jump/
- https://www.localogy.com/2026/11/visa-tests-ai-shopping-judgment/
- https://thumb.wikimedia.org/wikipedia/commons/thumb/c/c4/Person_using_a_credit_card_to_make_an_online_purchase_on_a_laptop_at_home.jpg/500px-Person_using_a_credit_card_to_make_an_online_purchase_on_a_laptop_at_home.jpg?utm_campaign=index&utm_content=thumbnail&utm_source=commons.wikimedia.org

