We ran 120 A/B experiments with an agentic AI framework. Here’s everything we learned.

Not a product review. Not a hype piece. A complete technical breakdown of the agentic development system Optiphoenix built internally – the agents, the integrations, the human review checkpoints, and the real numbers behind the results.

120 experiments · 3-month pilot

Table of Contents

1. Why we built this – the problem with rigid AI tooling
2. The framework architecture – a complete breakdown
3. The five specialised AI agents and what each one does
4. MCP integrations – how AI talks to Figma and Optimizely
5. Standards enforcement – the non-negotiable layer
6. Human-in-the-loop – why we intentionally didn’t automate the final review
7. Results – real numbers from 120 experiments
8. Onboarding and scalability benefits
9. What happens as LLMs improve
10. Frequently asked questions

1. Why we built this – the problem with Prompt based AI tools  AI tooling

By late 2024, every major experimentation platform was racing to add “AI-powered” features. Optimizely launched its AI variation development agent. VWO introduced AI-driven test creation and variation building directly inside its editor. AB Tasty launched Evi, an agentic AI that automates the full A/B testing journey from hypothesis to post-test analysis. The underlying promise was seductive: let AI handle the experiment build, focus your team on strategy.

We tested several of these tools. They worked – for the use cases they were designed for. The problem is that experimentation isn’t a fixed workflow. These prompt-based AI assistants don’t give users control over the code being written. It puts you in a position where you’re never quite certain whether the experiment will actually get built – or built correctly. Each attempt is either a hit or a miss, and when it’s a miss, you’re not debugging code, you’re rephrasing prompts. One tweak leads to another, the context drifts, and what started as a 30-minute build turns into an hour of back-and-forth with no guarantee of a working output at the end. Every client brings a different technology stack, a different analytics setup, a different targeting logic, a different governance process, a different QA standard, and a different set of brand constraints. No packaged tool accounts for all of that simultaneously.

“Every client stack, targeting logic, QA process, analytics setup, and governance layer is different. So instead of building a Vibe code, we built an agentic engineering framework as they provide a reliable approach without compromising with quality.” 

– Kamal Sahni, Co-founder, Optiphoenix

This is the core insight that drove everything: the problem isn’t building AI features for experimentation. The problem is that experimentation itself is structurally varied. You need a system that can adapt to each context, not a product that forces each context to adapt to it.

So we stopped looking for an off-the-shelf solution and built one from first principles – a modular, agent-based development framework designed specifically around the realities of professional CRO delivery.

Key insight: The bottleneck in AI-assisted experimentation isn’t the AI itself. It’s the absence of a structured system around it. Agents don’t make this work – the architecture around the agents does.

2. The framework architecture – a complete breakdown

The Optiphoenix Agentic AB Test Development Framework operates across three phases: Start / Brief, Agentic Task Execution, and Review & QA. Each phase has defined inputs, outputs, and human checkpoints.

What makes this different from simply “using AI to help with coding” is the deliberate assignment of scope. Each agent has a narrow, well-defined responsibility. Agents do not have general access to the whole workflow – they operate on their specific context, pass outputs to the next stage, and are governed by a shared set of standards that applies across the entire pipeline.

Phase 1 – Start / Brief

Every experiment begins with two inputs: a Product Brief and Figma Designs. These are not optional. Requiring both forces clarity before any agent touches code – a discipline that eliminates a significant class of downstream rework.

The brief-agent receives these inputs and performs two functions: analyse the brief for completeness, and produce a structured implementation plan. This plan becomes the shared contract between all agents that follow. It defines scope, complexity classification, audience parameters, and the experiment’s success metrics.

Step 1 → Product Brief + Figma Designs Mandatory dual input.

The brief defines scope and goals. Figma provides the UI reference that design-agent will interpret for pixel-accurate implementation guidance.

Step 2 → brief-agent.md analyses and plans

Reads the brief, classifies experiment complexity, identifies audience parameters, and generates a structured implementation plan that all downstream agents consume. Output: implementation plan · complexity class · audience params

Phase 2 – Agentic Task Execution

This is where the primary AI work happens – and where the framework’s modular design proves its value. Five specialised agents operate in this phase, each with a distinct scope.

After the code and experiment configuration are produced separately, a critical aggregation step occurs: Combine Code & Exp Config. This step merges the implementation code from dev-agent with the targeting and goal configurations from the two Exp-Tool agents, producing a unified experiment package ready for review.

Phase 3 – Review & QA

The Review & QA phase is the deliberate human checkpoint. It consists of two stages: Senior Dev Review (manual code review, step 11) and Build Passed to QA (step 13). If Senior Dev Review surfaces logic or code issues, the test-update-agent (step 12) is activated to correct them before QA.

Critical design decision: The Senior Dev Review is not optional. It is the single point in the workflow where business logic, edge cases, brand nuance, and production judgment are validated. Removing it would undermine the entire quality guarantee the framework provides.

3. Every step in the pipeline

This is the section where most breakdowns get wrong. They describe the agents and skip the stages. But the stages – the planning outputs, the context layers, the aggregation points – are what make the agents reliable. Here is every step, in order:

Step 1 – Product Brief & Figma Designs

Type: Input stage (human-produced)

This is the entry point of the entire pipeline. Before any AI agent is activated, two documents must exist: a Product Brief and Figma Designs.

The Product Brief defines what the experiment is, why it’s being run, what the hypothesis is, who the target audience is, and what success looks like. The Figma Designs provide the visual and interaction reference for what the variant should look like and how it should behave.

Neither is optional. This is a deliberate constraint. Many experimentation workflows begin with a vague brief and no design reference, leaving developers to interpret intent and make assumptions. This framework refuses that starting point – because every assumption made at step 1 compounds into rework at steps 6, 11, and 12.

The dual-input requirement forces the client or strategist to do the thinking before the pipeline begins. The result is that every agent downstream operates on real information, not inferred information.

Step 2 – brief-agent.md (Analyse & Plan)

Type: AI agent

The brief-agent is the first AI in the pipeline and arguably the most consequential. Its output becomes the shared contract for everything that follows.

The agent receives the Product Brief from step 1 and performs a structured analysis. It reads for completeness – flagging underspecified requirements, ambiguous audience definitions, or missing success metrics before implementation begins. It then classifies the experiment’s complexity as simple, medium, or complex, which determines how downstream agents approach the work and how Senior Dev Review allocates scrutiny.

Most critically, it produces a structured implementation plan – a defined-schema document that maps out the scope of the experiment, the technical approach, the audience parameters, and the expected outputs from each subsequent agent. This plan is not a summary of the brief. It is a structured specification written in a format that downstream agents can consume without further human interpretation.

Poor planning is the single largest source of rework in experiment development. By making planning a dedicated agent task with a defined output schema, the framework prevents ambiguity from entering the implementation phase entirely.

What it produces:

• Complexity classification (simple / medium / complex)
• Structured implementation plan
• Audience parameter definitions
• Flagged ambiguities requiring human clarification before proceeding

Step 3 – design-agent.md (Process Figma Mockups)

Type: AI agent · Uses Figma MCP

The design-agent connects to Figma via MCP integration and processes the design mockups referenced in the brief. It does not look at the Figma file as a designer would – it reads it as a structured data source, extracting the information that dev-agent needs to write accurate implementation code.

What it extracts: component hierarchy and nesting structure, layout constraints and spacing values, typography specifications, interaction states (hover, active, focus), animation parameters if specified, and references to reusable components already existing in the Figma library.

It produces a structured design specification – a document that dev-agent can implement directly, without needing a human to interpret the designer’s intent or ask clarifying questions about the mockup.

This eliminates one of the historically highest-friction handoffs in experiment development: the moment where a developer opens a Figma file, makes judgment calls about what the designer meant, and ships code that diverges from the design in ways that only become visible during QA.

What it produces:

• Component structure and nesting specifications
• Layout and spacing values
• Typography and colour tokens
• Interaction and animation parameters
• Structured design spec consumed directly by dev-agent

Step 4 – Create Implementation Plan

Type: Structural stage (output of steps 2 and 3 combined)

Step 4 is not a new agent. It is the point at which the outputs of brief-agent (step 2) and design-agent (step 3) are consolidated into a single, unified implementation plan that dev-agent will work from.

This consolidation step matters because the two agent outputs serve different purposes: brief-agent produces the strategic and functional specification, design-agent produces the visual and interaction specification. Separately, they are both useful. Combined into a single implementation plan, they become the complete technical brief that dev-agent needs to write accurate, scope-appropriate code without asking any clarifying questions.

The implementation plan at this stage contains: the experiment’s functional requirements (from brief-agent), the design specifications (from design-agent), the complexity classification, the audience parameters, and the expected output format for the variation code.

Think of step 4 as the moment where all the planning converges – before a single line of code is written.

Step 5 – Targeting & Audience Definition

Type: Structural stage (strategic input layer)

Step 5 is also not an agent. It is the stage at which the audience and targeting parameters – produced by brief-agent in step 2 – are formally defined and made available as structured input to the exp-tool agents in step 7.

This stage exists as a distinct step because targeting in professional CRO is not trivial. A targeting definition specifies: which URLs or page types the experiment runs on, which user segments are eligible (based on behaviour, device, geography, session history, or other attributes), what the traffic split is between control and variant, and whether the experiment uses cookie-based, session-based, or persistent assignment.

By treating targeting definition as an explicit stage – separate from code implementation – the framework ensures it receives dedicated attention rather than being treated as a secondary concern appended to the implementation task. The targeting parameters produced here feed directly into exp-tool-location.md at step 7.

Step 6 – dev-agent.md (Code Implementation)

Type: AI agent · References Client Folder + Standards + Implementation Plan

The dev-agent is the primary code-generation agent. It receives three inputs: the unified implementation plan from step 4, the targeting context from step 5, and access to the Client Folder – a structured repository of previously shipped experiments for the specific client.

The Client Folder is a significant differentiator. Most AI coding assistants operate without institutional memory – they write code in a vacuum, producing output that’s technically functional but stylistically inconsistent with the client’s existing codebase. This agent has direct access to the client’s actual experiment history, which means it can reference the patterns, component structures, utility functions, and naming conventions that are already in production.

The agent generates the experiment variation code – HTML, CSS, and JavaScript – against the implementation plan specification. As it writes, it enforces the six coding standards (detailed in section 5) inline. It does not write first and check standards later – standards compliance is built into the generation process.

The output is not final. It is a first-pass implementation designed to be accurate, standards-compliant, and scope-appropriate – ready for Senior Dev Review, not for immediate production deployment.

What it produces:

• HTML/CSS/JS variation code
• Standards-compliant implementation throughout
• Code written against client’s existing conventions
• First-pass output ready for step 11 review

Step 7 – exp-tool-location.md (Targeting Setup) + exp-tool-goal.md (Metrics & Goals Setup)

Type: Two parallel AI agents · Both use Exp-Tool MCP (Optimizely)

Step 7 contains two agents running in parallel with each other – and in parallel with dev-agent at step 6. They are separate agents because they handle separate concerns, but they run simultaneously to avoid the pipeline stalling while dev-agent does its work.

exp-tool-location.md handles the targeting configuration. Using the Exp-Tool MCP integration, it connects directly to the client’s Optimizely instance and configures: the experiment’s URL targeting rules (which pages the experiment runs on), the audience segments (which users are eligible), the traffic allocation (what percentage of eligible users enter the experiment), and the variant assignment logic. It works from the targeting parameters defined in step 5.

exp-tool-goal.md handles the metrics and goal configuration. Also using Exp-Tool MCP, it configures: the primary success metric (the single metric the experiment is designed to move), secondary metrics (supporting signals), goal event definitions (the specific user actions being tracked), and statistical significance thresholds. It works from the success criteria defined in brief-agent’s implementation plan.

The separation of these two concerns into distinct agents is deliberate. Targeting configuration and goal configuration have different logical dependencies, different risk profiles if misconfigured, and benefit from independent review. An error in targeting affects who sees the experiment. An error in goals affects what gets measured. Keeping them separate means each can be reviewed and corrected without touching the other.

What exp-tool-location.md produces:

• Configured URL and page-level targeting rules
• Audience segment setup in Optimizely
• Traffic split and variant assignment configuration

What exp-tool-goal.md produces:

• Primary metric configuration
• Secondary metrics and supporting signals
• Goal event definitions and tracking setup
• Statistical significance threshold configuration

Step 8 – Exp-Tool MCP (Convert / Optimizely Example)

Type: Structural stage (MCP context and translation layer)

Step 8 is the MCP integration layer that sits between the exp-tool agents and the Optimizely platform. It deserves explicit attention because it is not simply a technical bridge – it is where Optimizely’s generic configuration examples are translated into client-specific configurations.

The Exp-Tool MCP provides the exp-tool agents with access to Optimizely’s experimentation platform via the Model Context Protocol. But Optimizely’s API and its example configurations are written generically – they describe what’s possible across all Optimizely customers, not what’s correct for a specific client’s setup.

This step is where that translation happens. The MCP layer provides the exp-tool agents with context about the specific client’s Optimizely instance: their project structure, their existing audience definitions, their custom event taxonomy, their environment setup (development, staging, production), and their permission structure.

Without this translation layer, the exp-tool agents would produce technically valid Optimizely configurations that are contextually wrong for the client – configured against the wrong project, referencing non-existent audiences, or using event names that don’t match the client’s analytics implementation.

Step 9 – Exp-Tool MCP Targeting

Type: Structural stage (live MCP execution for targeting)

Step 9 is the live execution stage where exp-tool-location.md’s configuration output is committed to the Optimizely platform via MCP. It is distinct from step 8 (the context/translation layer) because it represents actual writes to the client’s live Optimizely instance – not configuration planning, but configuration execution.

At this point, the experiment’s targeting rules, audience segments, and traffic allocation are created or updated in Optimizely directly through the MCP integration. The Optimizely Remote MCP Server – which authenticates via OAuth and inherits the user’s platform permissions – ensures that agents can only write configurations that the authorised user is permitted to create in the Optimizely UI. No agent can exceed the permission scope of the human operator who authorised it.

This step also includes the Goal Definition execution – where exp-tool-goal.md’s metric and goal configuration is committed to Optimizely, establishing the measurement layer before any code is deployed.

The separation of targeting execution (step 9) from code implementation (step 6) means that if either needs to be revised, it can be corrected independently. A targeting error doesn’t require reopening the code. A code issue doesn’t require reconfiguring the targeting.

Step 10 – Combine Code & Exp Config

Type: Structural stage (aggregation and unified package assembly)

Step 10 is one of the most important structural stages in the pipeline, and one of the most frequently overlooked in descriptions of agentic workflows. This is the aggregation point where the outputs of all parallel workstreams – dev-agent’s variation code (step 6), the targeting configuration (step 7/9), and the goal configuration (step 7/9) – are brought together into a single, unified experiment package.

Before step 10, you have three separate outputs: code that implements the variant, a targeting configuration that defines who sees it, and a goal configuration that defines what it measures. These are independently correct but contextually incomplete – code without targeting is undeployable, targeting without goals is unmeasurable.

At step 10, these are combined and cross-validated. The combination process checks for consistency: does the targeting reference the same pages the code is designed to modify? Do the goal events correspond to interactions the code actually creates? Are the audience parameters consistent between what brief-agent specified and what exp-tool-location configured?

The output of step 10 is the complete experiment package – a unified artefact containing all three components – that passes to Senior Dev Review at step 11.

Step 11 – Senior Dev Review (Manual Code Review)

Type: Human checkpoint – mandatory, cannot be skipped

Step 11 is the most important step in the entire pipeline. It is not a formality. It is the point at which an experienced senior developer reviews the complete experiment package produced in step 10 – the code, the targeting configuration, and the goal setup – against a structured checklist of criteria.

The review is not a general code review. It is a checklist-driven evaluation against five defined criteria:

Standards compliance – Does the code adhere to all six coding standards? Is indentation correct? Are utility functions modular? Is DOM insertion using insertAdjacentHTML? Are null checks explicit? Are waitUntil callbacks single-line? Are existing client utilities being used rather than reimplemented?

Business logic accuracy – Does the code do what the experiment is supposed to do? This is where AI judgment most frequently falls short. The code may be syntactically correct and standards-compliant while implementing the wrong behaviour for the client’s specific business context.

Edge case coverage – Does the code handle the scenarios that the AI didn’t anticipate? This includes device-specific rendering issues, browser compatibility edge cases, race conditions in DOM manipulation, and client-stack-specific conflicts with existing scripts.

Brand alignment – Does the variant look and behave in a way that is consistent with the client’s brand standards? AI agents have no genuine understanding of brand nuance. A component that is visually correct by the design spec may still feel wrong in the context of the client’s actual product experience.

Performance and compatibility – Does the code perform acceptably across the devices and browsers in the client’s user base? Does it conflict with any existing scripts or third-party tools running on the page?

If the review passes all five criteria, the package moves to step 13 (QA). If issues are identified, they are categorised using a structured feedback schema and passed to step 12.

“We intentionally did NOT automate the final review. AI is excellent at boilerplate, pattern reuse, documentation, and first-pass implementations. AI is NOT reliable at edge cases, debugging the right cause, business logic validation, and production judgment. Hence: Senior Dev Review remains mandatory. Standards enforcement remains critical. QA still matters.”

– Kamal Sahni, Co-founder, Optiphoenix

Step 12 – test-update-agent.md (Update Agent Logic)

Type: AI agent · Activated only when step 11 identifies issues

The test-update-agent is not part of the primary execution path. It is a correction loop – activated only when Senior Dev Review (step 11) returns issues that require code or logic changes.

What makes this agent’s design important is the input format it receives. The issues identified in step 11 are not passed as free-form comments like “fix the targeting logic” or “the button animation looks wrong.” They are passed as a structured feedback schema – a defined format that categorises the issue type (standards violation, business logic error, edge case gap, brand misalignment, performance issue), identifies the specific affected code section, and describes the required correction precisely.

This structured input format is critical. Free-form revision instructions given to an AI agent produce unpredictable outputs – the agent interprets vague guidance liberally, often making sweeping changes beyond the intended scope and introducing new issues while correcting the original one. The structured schema constrains the agent to making targeted, auditable corrections only to the specific sections identified by the reviewer.

After the test-update-agent applies its corrections, the updated package loops back to Senior Dev Review (step 11) for re-evaluation. This loop continues until Senior Dev Review passes the package – at which point it moves to step 13.

What it receives:

• Structured feedback schema from Senior Dev Review
• Categorised issue types and affected code sections
• Precise correction requirements

What it produces:

• Targeted corrections to identified issues only
• Updated experiment package for re-review

Step 13 – Build Passed to QA (End)

Type: Human checkpoint and pipeline exit

Step 13 is the final stage of the pipeline. When Senior Dev Review passes the experiment package – either directly from step 11 or after correction via step 12 – it enters formal QA.

QA at this stage is not a re-review of what Senior Dev Review already checked. It is a functional verification: the experiment is deployed to a staging or QA environment and tested against a defined test plan. QA verifies that the experiment runs correctly on the targeted pages, that the targeting rules include and exclude the correct user segments, that the goal events fire correctly on the correct user interactions, that the variant renders correctly across the device and browser matrix specified for this client, and that no existing page functionality is broken by the experiment code.

QA is mandatory. The presence of AI in the development workflow does not reduce the need for functional verification – it changes where in the process errors are most likely to appear, but it does not eliminate the possibility of errors. A build that passes Senior Dev Review is a build that is logically and structurally correct. A build that passes QA is a build that is functionally verified in a real environment.

When QA is passed, the experiment is ready for launch.

4. MCP integrations – how AI talks to Figma and Optimizely

Model Context Protocol (MCP) is the technical backbone that allows the framework’s agents to interact with live external systems – not just generate text output, but actually create feature flags, configure audience rules, and extract real design specifications from Figma.

MCP is an open standard that provides AI agents with structured access to external tools and data sources. Instead of requiring developers to write API scripts or manually transfer information between systems, MCP creates a common protocol layer that allows agents to interact with platform APIs through natural language commands – with the platform enforcing its own permission structure throughout.

Figma MCP Integration

Flow: design-agent → Figma MCP → Component specs → dev-agent input

What the integration does:

• Extracts component hierarchy from Figma frames
• Retrieves layout constraints and spacing tokens
• Reads interaction and state definitions
• Identifies reusable component references

Optimizely Exp-Tool MCP Integration

Flow: exp-tool-location + exp-tool-goal → Optimizely MCP → Live experiment config

What the integration does:

• Creates feature flags and experiment entities
• Configures audience targeting rules and segments
• Sets up primary and secondary goal metrics
• Configures traffic splits and rollout rules
• Queries existing experiment configurations
• Inherits platform permissions via OAuth

The Optimizely Remote MCP Server (launched April 2026) enables exactly this kind of agent-driven experimentation management. What previously required multiple API calls, manual dashboard navigation, and developer involvement can now happen through structured natural language commands from an authorised agent. The server authenticates via OAuth and inherits platform permissions, meaning agents can only execute what the authorised user is permitted to do in the UI.

Why MCP matters for experimentation teams: Historically, experiment configuration bottlenecked at engineering – flag creation, audience setup, and results querying all required technical access or direct developer involvement. MCP effectively removes this bottleneck by giving AI agents – operating under controlled, permission-aware conditions – the ability to perform these tasks programmatically.

5. Standards enforcement – the non-negotiable layer

One of the most important design decisions in the framework is the presence of an explicit, shared coding standards document that governs every agent that produces code. Standards are not suggestions – they are enforced as part of the agent prompt, checked during Senior Dev Review, and validated during QA.

Without standards enforcement, AI-generated code in a multi-agent workflow rapidly diverges in style, convention, and approach. Each agent, operating independently, will make different decisions about indentation, DOM manipulation, null handling, and callback patterns. The result is a codebase that’s technically functional but unmaintainable – which is precisely the wrong outcome for a team building long-term client relationships.

The six non-negotiable standards

1. indentation – 2-space indentation only No tabs. Consistent across all agent-generated code, regardless of which agent produced it. A simple rule with a disproportionate impact on readability during code review.

2. no-inline-util – No inline utility implementations Utility functions must be modular and reusable, not written inline where they’re used. AI agents have a natural tendency to produce self-contained inline code – this standard explicitly forbids it.

3. dom-insertion – insertAdjacentHTML for all DOM insertion This is the performance-correct approach for inserting HTML into the DOM – avoiding the innerHTML reflow that causes layout thrashing in complex experiment variants, particularly on slower devices.

4. null-checks – Explicit null checks, no optional chaining Optional chaining (obj?.property) silently swallows null reference errors rather than surfacing them. Explicit null checks make failure states visible and debuggable – essential in a production experimentation context.

5. waitUntil – Single-line callbacks only The waitUntil utility pattern – waiting for a DOM element to exist before executing code – must use single-line callbacks. Multi-line callbacks inside waitUntil create maintenance nightmares when experiments need post-launch adjustments.

6. client-utils – No reimplementation of existing client utilities If a client’s existing codebase has utility functions (debounce, throttle, storage helpers), the agent must use those rather than producing its own implementation. This requires the Client Folder context to be actively referenced during generation.

The compounding benefit of standards: Standards enforcement compounds over time. Each experiment built to the same standards makes the next one faster to review, easier to debug, and simpler to onboard new developers onto. The efficiency gains are not just in build time – they accrue across the entire experiment lifecycle.

6. Human-in-the-loop – why we intentionally didn’t automate the final review

This is the part of the framework that runs counter to most AI product marketing narratives in 2025. The conversation around AI is dominated by a push toward greater automation, reduced human involvement, faster autonomous execution. We took the opposite view – and we think the data bears it out.

“We intentionally did NOT automate the final review. Not because we couldn’t – but because we understood precisely where AI judgment fails.” – Kamal Sahni, Co-founder, Optiphoenix

The decision to keep Senior Dev Review mandatory was not about distrust of AI in general. It was about a precise understanding of where AI performs reliably and where it does not.

What AI is genuinely good at in this context

• Boilerplate code generation and pattern reuse from existing experiments
• First-pass implementation of well-specified UI components
• Interpreting structured design specs from Figma
• Configuring targeting rules and goal metrics via API
• Applying coding standards consistently at scale
• Documentation generation and comment writing
• Cross-referencing client code conventions from the Client Folder

What humans must validate

• Edge cases specific to a client’s technology stack
• Brand nuance – tone, visual, interaction subtleties AI can’t detect
• Business logic validation – does this code do what the business needs?
• Production-grade judgment calls – performance, security, compatibility
• Detecting hallucinated patterns that look correct but aren’t
• Experiment hypothesis validity and strategic alignment

The industry data

Research from across the industry reinforces this position. McKinsey data shows that processes augmented by both AI and humans are 50–120% more efficient than either working alone. Harvard Business School research found that consultants using AI produced results of more than 40% higher quality compared to a control group – not 40% faster, but 40% better quality. The efficiency comes from the combination, not the substitution.

More starkly: 42% of companies abandoned most of their AI initiatives in 2025 – up from 17% in 2024 – often specifically because they failed to implement appropriate oversight from the start. The pattern is consistent: remove the human review checkpoint, and the AI’s speed advantage turns into a compounding quality debt that eventually requires expensive remediation.

What happens without human review: Without mandatory Senior Dev Review, AI-generated experiment code ships with undetected edge case failures, brand-inconsistent interactions, and logic that’s technically correct but strategically wrong. These failures are expensive to diagnose post-launch and damage client trust in ways that operational efficiency gains cannot offset.

How the review loop is structured

The Senior Dev Review (step 11) is not a general code review – it is a structured checklist-driven review against defined criteria. The reviewer evaluates: standards compliance, business logic accuracy, edge case coverage, brand alignment, and performance characteristics. Issues are categorised and fed as structured input to the test-update-agent (step 12) – not as free-form comments, but as a defined schema that constrains the agent’s corrections to the specific issues identified.

This structure prevents a common failure mode in AI-assisted revision loops: where a vague instruction like “fix the targeting logic” causes the agent to make sweeping changes beyond the intended scope, introducing new issues while correcting the original one.

7. Results – real numbers from 120 experiments

Over a three-month internal pilot, the framework was used to develop 120 experiments across the Optiphoenix client portfolio. Experiments were classified by complexity at the brief-agent stage and tracked for effort reduction against baseline (pre-framework) build times for equivalent experiment types.

The reduction is highest for simple experiments – and this is expected. Simple experiments have the most boilerplate, the most standardised patterns, and the least requirement for human judgment in implementation. The framework eliminates almost all manual work at this tier.

Complex experiments still show a 35% reduction, which is significant given that these are the experiments that require the most Senior Dev Review time. The gains here come primarily from standards enforcement reducing post-review rework, and from the targeting and goal configuration agents eliminating manual Optimizely dashboard work even for complex audience setups.

What the numbers don’t show

Consistency. Experiments built through the framework are significantly more consistent in code quality than those built without it. Pre-framework, consistency depended heavily on which developer was assigned to a given experiment. Post-framework, standards enforcement creates a quality floor that applies regardless of the individual.

Review speed. Senior Dev Review time decreased because the code being reviewed was more predictable. When a reviewer can rely on consistent indentation, naming conventions, null check patterns, and utility usage, they spend less time context-switching and more time on substantive logic evaluation.

Onboarding speed. New developers joining the team could contribute to production experiments significantly faster because the framework abstracts much of the context-building that would otherwise take weeks to develop.

8. Onboarding and scalability benefits

Traditional experimentation teams require weeks to months of ramp-up before a new developer can confidently ship production-grade experiment code for a new client. They need to learn the client’s codebase, understand the team’s coding conventions, internalise the QA standards, and develop intuition for edge cases.

The framework compresses this curve significantly. New developers are not left to absorb conventions organically – the standards are encoded into the agent system itself. The Client Folder provides structured access to historical experiment context. The implementation plan from brief-agent gives new developers a clear scope definition before they write a line of code.

Four pillars of scalability:

Encoded Standards – Coding conventions live in the agent system, not in individual developers’ heads.

Client Folder Context – Historical experiments give new team members – and agents – institutional memory from day one.

Structured Planning – brief-agent produces clear implementation plans that remove ambiguity before implementation begins.

Repeatable Process – The same workflow applies to every experiment. No ad-hoc decisions about how to approach a new brief.

The scalability implication is significant. A team using this framework can take on more experiments per developer without proportional degradation in output quality. The marginal cost of adding an experiment to the pipeline is lower because the framework handles the high-volume, pattern-based work that historically consumed the most time.

9. What happens as LLMs improve

One of the most compelling properties of a well-designed agentic framework is that it compounds in value as the underlying models improve. This is qualitatively different from a product built around a specific model’s capabilities – which becomes obsolete when the model is superseded.

The Optiphoenix framework is model-agnostic at the agent level. The agent prompt documents (.md files) define roles, responsibilities, input/output schemas, and standards – not specific model behaviours. When a better model is deployed, the same framework extracts more value from it immediately, with no structural changes required.

“This system continuously evolves as the underlying LLM models improve. The workflow improves with them.”

– Kamal Sahni, Co-founder, Optiphoenix

Concretely: as code generation models improve, the dev-agent produces better first-pass implementations, reducing Senior Dev Review time further. As vision models improve, the design-agent extracts richer specifications from Figma mockups, reducing interpretation gaps. As reasoning models improve, the brief-agent produces more nuanced implementation plans that better anticipate edge cases.

Each improvement in model capability flows through the framework’s structure into measurable output quality improvements – without requiring the framework itself to be rebuilt. This is the architectural advantage of separating the workflow from the model.

The longer-term trajectory points toward a gradual expansion of what the agents handle autonomously – as model reliability for specific task types increases, the human review checkpoint may shift from reviewing all code to reviewing only specific categories of complexity or client-specific logic. But that evolution is demand-driven by demonstrated model performance, not calendar-driven by hype.

10. Frequently asked questions

What exactly is an agentic A/B testing framework?

An agentic A/B testing framework is a structured system that combines specialised AI agents, MCP integrations, coding standards enforcement, and mandatory human review loops to automate the repeatable parts of experiment development while preserving human judgment for edge cases, brand nuance, and production-level decisions. It’s architecturally different from a single AI tool – it’s a workflow system in which multiple agents operate on defined scopes with structured handoffs between them.

How much time can this type of framework save on A/B test development?

Based on Optiphoenix’s internal pilot across 120 experiments: up to 59% effort reduction for simple experiments, 46% for medium-complexity experiments, and 35% for complex experiments. The overall average across all experiment types was approximately 50% faster setup time compared to pre-framework baselines.

Does this work with any experimentation platform, or just Optimizely?

The framework’s architecture is platform-agnostic at the agent level. The MCP integrations currently implemented connect to Optimizely and Figma. The exp-tool agents could be adapted to work with other platforms that offer MCP servers or accessible APIs – the agent prompt documents would need to reference the target platform’s data model, but the structural logic remains the same.

Why is human review still mandatory if AI handles most of the work?

Because AI is not reliably capable of validating edge cases specific to a client’s technology stack, brand nuance, business logic, and production-grade judgment calls. These are not hypothetical failure modes – they are consistent limitations observed across the 120-experiment pilot. The human review checkpoint is the quality guarantee. Without it, the efficiency gains of AI-assisted development are offset by the cost of post-launch issue remediation and client trust damage.

What is MCP and why does it matter for this framework?

MCP (Model Context Protocol) is an open standard that lets AI agents connect directly to external tools – Figma, Optimizely, analytics platforms – and interact with them programmatically using natural language commands. In this framework, MCP allows the design-agent to read Figma designs directly and the exp-tool agents to configure experiments in Optimizely directly, eliminating the manual data transfer and API scripting that previously created bottlenecks in the workflow.

How does this framework improve onboarding for new CRO developers?

The framework encodes institutional knowledge – coding standards, client conventions, historical experiment patterns – into the agent system itself, rather than leaving it in individual developers’ heads. New team members don’t need to absorb conventions organically over weeks; they work within a structured system that surfaces the right context at each stage. The implementation plans produced by brief-agent give new developers clear scope and expectations before they write a line of code.