Key takeaway: Merchants and processors should expect agentic purchases to create new “not as described” and “not authorized” disputes. Visa’s testing shows AI agents can complete a purchase even when they have not captured what the shopper actually wanted.
Visa released the Visa Agentic Commerce Benchmark on November 5, 2026, testing 22 AI models from 12 labs on realistic shopping tasks. The models passed 57.2% of test runs, according to the company.
The benchmark covers 15 shopping challenges, such as buying a TV, a laptop, or a personal care appliance, with each model given three attempts per scenario. Models must work out a consumer’s needs through conversation and choose not to buy when nothing fits. A run passes only if the AI uncovers every task-defined requirement and makes the correct buy or no-buy decision, which Visa said prevents lucky guesses.
Results varied widely. The best models passed 82.2% of attempts, while the weakest passed 28.9%. All three attempts passed in only 34.2% of cases across models and scenarios. About 54% of failed runs still ended with the AI making a purchase.
Visa also pointed to its September 2026 Trust Index, which found that 72% of US consumers had used an AI assistant but only 23% trusted AI to handle payment transactions on their behalf.
Rubail Birwadker, Global Head of Growth at Visa, said “AI agents have the potential to power the next era of commerce, but trust is key to making it work.” Visa said it plans to add more scenarios and product categories and to extend testing to checkout, payment, and order placement.
Why it matters: An agent that buys the wrong item, or buys when it should have declined, leaves the consumer holding a purchase they never wanted. That outcome maps directly to “not as described” and “not authorized” disputes, and the low consumer trust figure suggests shoppers may be quick to contest those charges. Merchants and processors may need to decide how they document agent-initiated orders and who carries the liability when an agent gets the purchase wrong.
Source: Visa