Why AI Search Converts Better Than Traditional Search

In this week’s episode of Voices of Search, we spoke with Joe Doran, chief product officer at Botify. Joe helped Microsoft build a paid search platform and now leads product strategy at Botify, where he helps enterprise retailers adapt their infrastructure to an AI-driven discovery environment. His background sits at the intersection of crawl technology, structured data, and commerce—which makes him uniquely positioned to explain what’s actually happening beneath the surface of agentic search.

Our conversation covered why traditional search was generating false positives all along, why most enterprise retailers are only indexed at 40 to 50% of their actual product catalog, and why the product feed—not the product page—is quickly becoming the primary competitive battleground in e-commerce.

Key Takeaways From This Episode:

• Most enterprise retailers are only indexed at 40 to 50% of their pages, according to Google Search Console data. That means nearly half their products have no chance of being discovered—by Google or any LLM.
• AI search is not compressing the buying cycle—it’s eliminating the false positives that traditional search created. Fewer clicks, but higher-intent arrivals at the product page.
• LLM crawlers see only 30 to 40% of what’s on a product page due to JavaScript rendering limitations—missing reviews, specs, promotions, and other signals that inform AI recommendations.
• Product feeds are the new on-page SEO. Full, accurate, and complete structured data fed directly to LLMs is now the primary lever for product discoverability.
• Prompt volume and share of voice metrics are probabilistic and synthetic. Citation rate correlated with crawl volume is a more reliable signal of actual AI visibility.

The False Positive Problem Traditional Search Never Solved

The conversation opened with a framing that reorients how most retailers should be thinking about the shift to AI search. Rather than treating lower click volume as a loss, Joe offered a different lens: traditional search was wasteful by design.

“You would make a query, you would go to a product page, and it wasn’t what you wanted,” he said. “You’d go back, do another query. So it’s actually more efficient—you got rid of the bad clicks, the clicks that were never leading to a conversion.”

The reason AI search does this better comes down to how chatbots handle constraints. A user looking for an outfit for a specific event, within a specific budget, with specific fit requirements is giving the system multiple simultaneous inputs that a keyword query could never process. 

“Those multiple constraints are really hard in a traditional search solution,” Joe said. “The chatbots do a great job of that product discovery, narrowing it down and allowing you to find what the full portfolio of products is out there.”

What this means for retailers is that AI referral traffic is arriving at product pages with stronger intent. Conversion rates on those visits are trending higher—not because more people are discovering products through AI, but because the ones who do are more qualified when they land.

Half Your Products Don’t Exist to Search Engines

Before any of the AI-specific strategy matters, Joe pointed to a more fundamental problem that most enterprise retailers haven’t fully reckoned with: indexation.

“When we look at retailers across size and scale, we’re only seeing them—and this is reported through Google Search Console—their indexation sits between 40 and 50%,” he said. “Which means only 40 to 50% of their web pages are actually being indexed, let alone crawled.”

Why the Problem Is Getting Worse, Not Better

The issue compounds in multiple directions:

• Retailers are expanding product catalogs aggressively. A tier-one retailer with less than a million products is now rare.
• JavaScript-heavy pages cost three to five times more for crawlers to render than a clean HTML page. Google’s crawler is the most sophisticated in the world, and even they prefer not to render JavaScript because of the budget cost.
• LLM crawlers from ChatGPT, Perplexity, and others are roughly 20 years behind Google’s crawl efficiency. They don’t have the budget or sophistication to work through complex pages, so they simply move on.

The compounding effect is what Joe called the real danger. “If I as a crawler can’t efficiently crawl the breadth of your site, I’m going to continually pull back on it because there are other parts I could spend on,” he said. 

Lower crawl efficiency signals lower content quality to the systems doing the crawling, which triggers even more budget reduction—and the cycle accelerates.

What Crawlers Actually See on Your Product Pages

Beyond indexation, there’s a separate problem happening at the page level. Even when a bot does crawl a product page, the content it sees is dramatically incomplete.

“Our friends at the crawlers and bots cannot render JavaScript,” Joe said. “So what ends up happening is they see 30 to 40% of the content that’s on the page.”

The things they miss are exactly the things that matter most for AI recommendations:

• Reviews, which are almost always rendered through third-party JavaScript
• Specs and detailed descriptions, which are frequently pulled through API calls rather than baked into the HTML
• Promotional pricing and offers, which are dynamically rendered

Joe described showing retailers a side-by-side comparison of what a crawler sees versus what a consumer sees. “They just get irate,” he said. “Because they’re missing like half the information that’s there.”

For LLM crawlers specifically, the problem has an additional layer. When ChatGPT retrieves a product to include in a response, it runs a confirmation crawl against the live page to verify that the data in its index still matches. 

“If it can’t read it, it’s not going to be able to verify that data,” Joe said. “It doesn’t match what’s in my index, so I’m not going to show it. No mention, no citation.”

The Product Feed Is Now the Primary Competitive Lever

This is where the conversation shifted from diagnosis to strategy. If crawlers can’t reliably read product pages, and LLMs need structured data to make accurate recommendations, then the product feed becomes the mechanism through which retailers actually compete for AI visibility.

“Anyone that’s a retailer should be leaning in very heavily to go deliver full, accurate, complete product feeds to the LLM,” Joe said. “Any of them that’ll actually take it.”

What Full and Complete Actually Means Now

The requirements for product feeds have expanded significantly beyond what Google Shopping historically demanded. The old model needed five fields: title, description, price, reviews, and an image. That was enough to fill an ad slot. AI answer engines need far more to generate contextually accurate recommendations.

When OpenAI launched its Agentic Commerce Protocol, it had 75 required fields. It trimmed to 42 after recognizing that retailers couldn’t source the additional data at scale—but those fields are still wanted. The areas where most feeds fall short:

• Review content, not just star ratings. LLMs want the actual text of 10 to 15 reviews, not the two or three displayed on the page.
• Q&A content on individual product pages, which most retailers don’t have and can’t produce at the scale of millions of SKUs.
• Missing attributes like dimensions, materials, pattern details, and category classifications that exist on the product page but were never systematically extracted into the product information system.

“We’ll see that nine out of 10 records or products have at least three to seven errors in them,” Joe said. “Missing data, incorrect data, data that’s inconsistent with what’s on the product description page, or fields that were just never populated.”

The underlying cause is structural. Retailers have spent 20 to 30 years augmenting product pages with third-party data—reviews from external providers, specs from manufacturer feeds, promotional content from marketing tools—none of which lives in a single source. 

“They’ve built this castle of a beautiful product description page,” Joe said. “There’s not one source of it.”

Share of Voice Is Not the Metric You Think It Is

One of the sharpest points in the conversation was Joe’s pushback on the AI visibility metrics most retailers are currently obsessing over: prompt volume and share of voice.

“Both of those metrics are probabilistic. Both of them are synthetic,” he said. “No one is publishing accurate, transparent data around what is actually being answered by the search engines and what the volume is there.”

The panel providers that supply this data are working from small samples—often Chrome plugin data—that may not represent a given retailer’s actual customer base. Extrapolating those panels to broad population-level conclusions, and then making infrastructure investment decisions based on them, is a significant risk.

What Joe trusts instead is a simpler signal: citation rate correlated with crawl volume. “If you increase citation rate, I see a correlated increase in crawl volume, which means they’re doing a retrieval crawl to go get it,” he said. That relationship between being cited and being crawled is measurable, directly tied to infrastructure, and actionable—unlike prompt volume estimates built on synthetic panels.

The Attribution Problem No One Has Solved Yet

The conversation closed on the measurement challenge that sits beneath everything else being discussed in e-commerce right now: attribution.

“I think humans are complex and messy people,” Joe said. “I do not believe they just go to a single chatbot, make a request, and then purchase. I think the shopping experience is messy.” A single purchase might touch traditional search, agentic search, a retargeting ad, a TV spot, and a site search—across multiple devices, over multiple days.

The session-based conversion model that paid and organic search teams have both optimized against for a decade was never capturing that complexity. It was just the only thing measurable at scale. The shift to AI-driven discovery doesn’t create the attribution problem—it exposes the one that was always there.

Joe’s position is that the brands that get ahead of this will be the ones that anchor on the purchase as the outcome and build measurement infrastructure that attributes channel contributions holistically rather than in silos. 

The retailers that will look back five years from now wishing they’d started earlier aren’t the ones that missed a new tactic. They’re the ones that kept measuring the wrong thing while the landscape shifted underneath them.

The Future of E-Commerce Discovery

The shift toward agentic search isn’t an update to be gamed—it is a fundamental re-architecting of how commerce happens online. Fewer clicks do not signal a dying search channel, but the elimination of the false positives that bloated traditional search for decades. 

Capturing those higher-intent shoppers requires a complete realignment of enterprise search strategy: solving the catalog indexation gap, treating rich product feeds as the primary discovery lever over unrenderable JavaScript pages, and grounding performance in verifiable citation rates rather than synthetic metrics. Above all, it requires abandoning obsolete, single-session tracking in favor of holistic revenue measurement. 

Ultimately, agentic search won’t be won by the brands with the cleverest optimization tactics, but by the ones that build the infrastructure to make their entire catalog visible, verifiable, and ready for recommendations.

Voices of Search is a daily SEO and content marketing podcast hosted by Jordan Koene and Tyson Stockton. The show delivers actionable strategies and data-driven insights to help marketers navigate the ever-evolving world of search engine optimization and content marketing. New episodes air weekly, covering everything from technical SEO to AI discovery, featuring industry leaders and practitioners sharing real-world frameworks and proven tactics.

Subscribe to Voices of Search on Apple Podcasts, Spotify, or your favorite podcast platform. Follow Previsible on LinkedIn for updates and subscribe to the VOS YouTube channel for video episodes and clips.