Visa Tested 22 AI Shopping Agents — They Failed 43% of the Time, and Your Brand Data Is Why

Visa Tested 22 AI Shopping Agents — They Failed 43% of the Time, and Your Brand Data Is Why

Jasper Koers 11 min read Brand Intelligence

On November 5, 2026, Visa published the first independent benchmark of AI shopping agents. The Agentic Commerce Benchmark put 22 AI models from 12 different labs through 15 realistic shopping scenarios — buying televisions, laptops, personal care appliances, and other products that real consumers purchase every day.

The headline number: the agents failed 42.8% of their test runs. But the more troubling finding is what happened after they failed. In 54% of those failed runs, the AI agent still completed a purchase — recommending or buying a product that did not match the customer's actual requirements.

This is not a hypothetical problem. These are the same AI models that power ChatGPT Shopping, Google AI Mode checkout, and the autonomous purchasing agents that Walmart, Amazon, and Shopify are building into their platforms. When these agents fail, real money moves to the wrong product.

What Visa Actually Tested

The benchmark is more rigorous than the typical AI evaluation. Each of the 15 scenarios included both stated requirements — things the simulated customer explicitly asked for — and hidden requirements that a competent human sales assistant would uncover through follow-up questions.

For example, a customer might ask for a television with a specific screen size and budget. But a good sales assistant would also ask about the room's lighting conditions, viewing distance, whether they play video games (which requires different refresh rate considerations), and whether they need smart TV features or already have a streaming device. These contextual requirements determine whether the purchased product actually satisfies the customer, even though the customer did not articulate them upfront.

Each model got three attempts at every scenario. All three attempts passed only 34.2% of the time. This means that even when an agent gets it right once, it frequently gets the same task wrong on the next try. The consistency problem is as significant as the accuracy problem.

The Performance Spread Is Enormous

The best-performing models achieved an 82.2% pass rate. The worst managed just 28.9%. That is a nearly 3x difference in reliability across models that are all marketed as capable shopping assistants.

But the more actionable finding is what separated winners from losers. The five models that uncovered the most hidden buying requirements averaged a 70.2% pass rate. The five that uncovered the fewest averaged 44.4%. The ability to ask follow-up questions — to probe beyond the stated request — was the single strongest predictor of success.

This has direct implications for brand data. An AI agent can only ask intelligent follow-up questions if it understands the product category well enough to know what questions matter. That understanding comes from structured product data: specifications, compatibility information, use-case metadata, and category-specific attributes that let the agent reason about which requirements are relevant.

Why Agents Fail: The Data Problem

Visa's benchmark reveals a problem that looks like an AI capability gap but is actually a data quality gap. Consider what happens when an AI shopping agent encounters a typical product page:

Scenario 1: Marketing prose instead of structured specifications. The product page says "Experience stunning visuals with our cutting-edge display technology." The agent needs to know the resolution is 3840x2160, the refresh rate is 120Hz, the panel type is OLED, and HDR10+ is supported. That information might exist somewhere in the page, but it is buried in a paragraph of marketing copy or scattered across tabs that the agent may or may not navigate to.

Scenario 2: Inconsistent data across sources. The brand's own website lists the product weight as 15.2 kg. The retailer's listing says 14.8 kg. The specification sheet PDF on the brand's support page says 15.0 kg without the stand. An AI agent cross-referencing these sources encounters three different numbers and has no way to determine which is authoritative.

Scenario 3: Missing category-specific attributes. A laptop listing includes processor, RAM, and storage — the basics. But it omits thermal design power, display color gamut coverage, thunderbolt port count, and keyboard travel distance — attributes that matter for specific use cases like video editing, software development, or portable work. The agent cannot ask whether the customer needs color-accurate display work if the data does not include color gamut information.

These are not edge cases. They are the norm. And they explain why agents that are capable of sophisticated reasoning still fail nearly half the time — they are reasoning over incomplete, inconsistent, and unstructured data.

The 54% Problem: Buying Wrong Is Worse Than Not Buying

The most consequential finding in Visa's benchmark is not the 42.8% failure rate. It is that 54% of failures resulted in a purchase. The agent did not say "I cannot find a suitable product" or "I need more information." It went ahead and bought something that did not meet the customer's requirements.

This creates a specific commercial problem. A failed search that produces no result is an inconvenience. A failed search that produces a wrong purchase generates a return, damages brand trust, and costs the retailer money. At scale, this is a significant liability.

For brands, this means the cost of bad data has changed. In the traditional e-commerce model, poor product data meant lower conversion rates — customers could not find what they needed and left. In the agentic commerce model, poor product data means wrong conversions — agents buy the wrong product and the customer returns it.

The economic incentive to fix product data has flipped from "improve sales" to "prevent losses."

What the Benchmark Means for Brand Data Strategy

Visa's benchmark is the first rigorous measurement of how AI agents perform as shoppers, and it establishes several things that were previously assumptions:

1. Structured product data is the bottleneck, not model capability. The top models are already good enough to be useful shoppers when they have the right data. The 82.2% pass rate of the best models is commercially viable. The gap between 82% and 43% is not primarily about smarter AI — it is about better data for the AI to work with.

2. Cross-source consistency is a trust signal. When an agent encounters conflicting information about the same product from different sources, it either picks one arbitrarily or averages the conflicting data points. Neither produces reliable results. Brands that maintain consistent data across their own website, retail partner listings, and third-party databases give agents a coherent picture to work with.

3. Category-specific attributes determine recommendation quality. Generic product data — name, price, basic specs — is not enough. The benchmark's hidden requirements test specifically measures whether agents can match products to contextual needs that generic listings do not address. Brands that publish rich, category-specific attribute data enable agents to make nuanced recommendations instead of basic keyword matches.

4. The consistency tax is real. All three attempts passed only 34.2% of the time. This means brands cannot rely on agents getting it right once. Agents need to reliably extract and interpret the data every time, which requires consistent formatting, clear attribute labels, and structured data that does not change meaning depending on how the page renders.

Where Fetching Company Fits

The verification layer — ensuring your brand data is consistent, structured, and machine-readable across every surface an AI agent might check — is exactly what brand intelligence APIs exist for.

When Visa's benchmark tests whether an agent can identify the right product for a customer, it implicitly tests whether the brand's data is structured well enough for the agent to reason about. Logo consistency, contact data accuracy, social profile linkage, and structured brand metadata are all trust signals that agents use to verify they are dealing with an authoritative source.

Our API extracts and structures the brand data layer: logos, colors, fonts, contact information, and social profiles from any URL. This is the foundation that sits underneath product data — the brand identity verification that lets agents trust the source they are reading product specifications from.

An agent that encounters a product listing needs to answer two questions: "Is this the right product?" and "Is this a trustworthy source?" Visa's benchmark measures the first question. Brand data consistency determines the second.

What to Do Now

The Visa benchmark is not an abstract research paper. It is a preview of how the largest payment network in the world will evaluate AI agents that handle transactions on its rails. Visa has stated it will extend the benchmark to checkout and payment flows under its Intelligent Commerce initiative.

Audit your product data for agent readability. Take your top 20 products and ask: if an AI agent read each product page, could it extract every specification as a structured data point without parsing marketing copy? Count how many attributes exist only in prose paragraphs rather than in structured fields or schema markup.

Test cross-source consistency. Pick five products and compare the data on your own website against the data on your top three retail partners. Every discrepancy in specifications, dimensions, weights, or compatibility information is a potential agent failure point.

Publish category-specific attributes. Go beyond the generic spec sheet. For electronics, publish thermal data, port specifications, and display characteristics. For apparel, publish exact measurements, fabric composition percentages, and care instructions as structured data. For food products, publish nutritional data, allergens, and sourcing information. The hidden requirements in Visa's benchmark map directly to these category-specific attributes.

Verify your brand identity layer. Use a brand intelligence API to extract your current brand fingerprint from your website and compare it against what appears in directories, retail platforms, and social profiles. Every inconsistency erodes the trust score that agents assign to your brand as a source of product information.

Monitor agent traffic patterns. Check your server logs for AI crawler and agent user agents. Understand which pages agents visit most frequently, which pages they bounce from, and which products they struggle to extract data for. This gives you a prioritized list of pages to fix.

The Benchmark Is Just the Beginning

Visa's 42.8% failure rate will improve. Models will get better at asking follow-up questions and reasoning about product requirements. But model improvement cannot compensate for missing data. An agent that perfectly understands what questions to ask still cannot make a good recommendation if the answers do not exist in a structured, consistent, machine-readable format.

The brands that fix their data now will be the ones that agents recommend confidently once the models catch up. The brands that wait will find themselves in Visa's next benchmark as the products that agents skip over — or worse, the products that agents buy by mistake and customers return.

Agentic commerce is not coming. It is being benchmarked. And the benchmark says your data is the bottleneck.

Fix your brand data before the agents do it wrong. Extract your brand fingerprint with Fetching Company and verify that every surface an AI agent checks tells the same story — structured, consistent, and machine-readable.

Share this article

Ready to try the API?

Extract brand data from any website with a single API call. Start free.