Why AI Can’t Replace Your Baseline Research

40 out of 45. That’s how many citations in KPMG’s October 2025 report on agentic AI turned out to be fabricated or misattributed. 17% and 34%. Those are the hallucination rates Stanford researchers measured in Lexis+ AI and Westlaw’s AI-Assisted Research, two legal tools built specifically for accurate case retrieval. AU$440,000. Deloitte refunded the Australian government after an AI-assisted report cited academic papers that don’t exist and quoted a court case that was never heard.

None of these tools were malfunctioning. They were doing exactly what a large language model does: predicting the most statistically plausible next words. Plausible and true are different properties, and nothing forces them to line up.

Now run the same test on your last CRO research doc. Ask an AI tool why your cart abandonment is high, and it hands you five clean, confident reasons: shipping surprises, checkout friction, weak trust signals, no guest checkout, slow load times. Every one is plausible. None of them pulled from a session recording, a support ticket, or an actual visitor to your actual store. It’s the KPMG problem wearing a CRO hoodie.

What baseline research actually is

“Research” in CRO isn’t a vibe check. CXL’s ResearchXL model, one of the industry’s most-used frameworks, breaks it into six concrete inputs, and AI has a real, limited role in exactly one column of this picture.

Good baseline research produces a sentence like this: “23% of mobile users abandon within 8 seconds of reaching the shipping page. 6 of 12 session recordings show repeated up-down scrolling before exit. 3 support tickets independently cite unclear delivery timelines.” Every clause has a receipt. Hand it to a skeptical stakeholder and they can go check it themselves.

The 85/15 problem

In 2024, Stanford and Google DeepMind built AI agents from two-hour interviews with 1,052 real people, then tested how well those agents predicted the same people’s answers on a follow-up survey. Result: 85% accuracy, matching how well people replicate their own answers two weeks later. That’s genuinely strong, and worth using for early-stage directional thinking.

Here’s the part that matters more for you: the missing 15% wasn’t random noise. Researchers traced it to personal anecdotes, emotional contradictions, irrational decisions, and context-dependent behavior. That’s not a rounding error. That’s the exact category of detail that separates a generic hypothesis from a winning one. A synthetic persona can tell you “price-sensitive shoppers exist.” It can’t tell you that a real customer typed the name of your competitor into a support chat last Tuesday because she was comparing prices mid-checkout, which is the sentence that actually writes your test.

Do the math on what 85% means for a research program, not a single answer.

If you treat three unverified AI-generated insights as each independently reliable at that 85% rate, the odds all three are simultaneously correct drop to roughly 61%. Build a test brief on three synthetic-only insights and you’re closer to a coin flip than a research-backed decision. This is exactly why the verification step below isn’t optional.

Where AI actually earns its keep

Maze’s 2026 industry report found 69% of researchers now use AI in their workflow, up 19 points year over year, with 63% reporting faster turnaround and 60% reporting better team efficiency. That’s real. Tagging 500 open-ended survey responses by hand is slow, and a tired researcher tags row 400 differently than row 20. AI clustering doesn’t get tired and doesn’t drift.

The same report draws the line clearly: AI has taken over discovery-stage work, churn-driver analysis at scale, and concept testing. Survey research still holds for regulated tracking, benchmarks, and anything that needs to be statistically projectable to a real population. Translation for CRO: use it to process what you collected, not to skip collecting.

The implementation playbook

Here’s how to actually run this, starting with your next research cycle.

Step 1 — Collect real data only. Session recordings, on-site surveys, support tickets, user testing, analytics segmentation. All from real visitors on your real site. No synthetic substitutes here. Every downstream step inherits whatever bias or fabrication enters at this stage, so this is the one place “close enough” isn’t good enough.

Step 2 — Let AI synthesize, not originate. Once you have real transcripts and verbatims, feed them in with a prompt structured like this:

“Here are 15 support tickets about our checkout flow [paste raw data]. Cluster these into recurring themes. For each theme, cite the specific ticket number it came from. Do not include any theme that appears in fewer than 3 sources. Do not infer reasons that aren’t stated directly in the text.”

That last line matters most. It blocks the model from doing the one thing it defaults to: filling gaps with plausible-sounding invention. Here’s what that actually looks like, run twice, on the same underlying question.

Without guidance, a vague prompt gets a vague, unsourced answer:Fluent, confident, and built entirely from patterns in training data. Not one line traces back to your store. Every business asking this exact question gets this exact answer.

With the right dataset and explicit guardrails, the same underlying question gets a sourced, falsifiable answer instead: Same assistant, same general topic, different inputs. The prompt hands over real tickets and a rule: cite the source, and drop anything under three mentions. The response comes back with ticket numbers attached and one theme dropped for insufficient evidence, exactly as instructed. The difference between these two screenshots isn’t the model. It’s what you gave it to work with, and whether you told it verification was mandatory.

Step 3 — Verify before you hypothesize. Open the source file for every theme AI hands back. If it says “shipping cost is a top concern,” count the actual tickets that say so. This is a five-minute check that would have caught the KPMG and Deloitte failures before publication. It’s the highest-leverage five minutes in the entire process.

Step 4 — Keep synthetic personas in ideation, never in decisions. Use them to stress-test a hypothesis or brainstorm edge cases early. Never cite them as evidence in a research summary, and never let one be the deciding vote on a go/no-go call.

Quick reference: what to trust AI with Print this, tape it above your desk, or paste it into your test-brief template. Every “NO” is a task where AI is guessing, however good it sounds doing it.

The five-question audit

Run this against your next test brief before it reaches a stakeholder:

1. Can every claimed insight be traced to a specific recording, ticket, survey response, or number?
2. If AI touched the research, did a human check the source material, not just the AI’s summary of it?
3. Would this insight survive someone asking “show me the actual quote”?
4. Is any part of this based on a synthetic persona, and is it labeled directional rather than evidential?
5. Does the sample size actually support the confidence level in the brief?

Fail more than one, and the brief isn’t ready. This isn’t gatekeeping for its own sake. It’s the difference between a research-led testing program and one that’s guessing with better formatting.

Do this now

Pull your last three test hypotheses. For each one, find the specific recording, ticket, or verbatim it’s actually built on. No receipt, no evidence base, just a guess wearing a data label.

Want a second set of eyes on it? Opti Phoenix runs a free mini audit that traces your last quarter of test hypotheses back to source and flags which ones are evidence-based versus assumption-based. If the research process itself needs rebuilding. Let’s Talk Growth!!