AI Multiplied the Experiments, Not the Answers

I The claim

An agent can write forty headlines before your coffee is cold, and it cannot produce one extra visitor to read them. The cost of making a variant fell to roughly nothing while the cost of finding out whether a variant works stayed where it was. Teams now start far more tests than they finish, and an unfinished test leaves you with a page and no information about it.

II What got cheap and what did not

A test has two halves: somebody makes the thing, and then enough people see it for the result to mean something. Agents collapsed the first half, so that the copy, the layout, a second funnel and the tracking code can all be done in an afternoon.

The second half is paid for in traffic, time and statistical power, and no model sells those. A visitor can only be shown one variant and a week still has seven days in it.

You can see the gap in a vendor’s own numbers. Optimizely published a report on its agent platform, drawn from usage data across 47,000 interactions; the same page says nearly 900 companies have adopted the platform. It is vendor data, written to sell the platform, and the page does not state the comparison group. For teams using agents across the whole test lifecycle it lists experiments created up 78.66%, and experiments concluded up 2.38%. Optimizely reads this as good news, points to better win rates, and says the uplift slope for concluded experiments is steeper than for created ones. We hope so. Our own reading of the two figures side by side is that agents sped up the starting of tests a great deal and the finishing of them hardly at all.

III The arithmetic nobody likes

Our own back-of-envelope sum, using the standard rule of thumb for a two-variant test at conventional confidence and power, goes as follows. Say your page converts at 5% and the new variant would lift it to 6%. To see that reliably you need about 8,000 visitors on each side, which is sixteen thousand in total to answer one question.

Now let the agent make ten variants, because it can. You need on the order of 80,000 visitors, and more if you correct for the fact that you are now looking at ten results and will be tempted by the luckiest one. A startup with two thousand visitors a month has just scheduled an experiment that ends in 2030.

What happens in practice is worse than waiting. The team looks after a week, sees variant seven ahead on a handful of signups, and ships it. Variant seven was ahead because something is always ahead. With ten arms and no patience, you are running a lottery and writing up the winner as a finding.

Kromatic’s Real Startup Book sets 200 unique visitors as the minimum for a directional signal on a landing page test, and says of smaller samples that the rate is “directional, not a verdict.” Most early pages live in that zone. Splitting 200 visitors across ten variants gives you twenty people each, a tenth of what Kromatic asks for before it will call a result even directional.

IV What about simulated users

The obvious reply is to skip the traffic: have agents play the customers, run the test on personas, and read the result by lunch.

The serious research on this is more careful than the pitch decks. A September 2026 arXiv paper, “Augmented Hypothesis Testing with Persona-Based LLM Simulations,” starts from the same complaint we have, that A/B testing needs large samples, long timelines and real money. Its answer is to use persona predictions to cut the number of real users a test needs, with methods built to stay statistically valid even when the predictions are wrong. The authors report substantial cost reductions on four real-world datasets.

That design treats the simulation as a forecast of uncertain quality and leaves real people to settle the question. We think that is the right posture, and we would take the saving gladly. A persona saying yes to your pricing page is still an opinion from a model that has never paid for anything, so we would use it to choose which two variants deserve the traffic and leave the choice of winner to the visitors.

V Fewer, larger bets

At small sample sizes, the size of the difference you are testing is the only lever you control. Run the same sum as before with a bigger gap, and a variant that would take conversion from 5% to 10% shows itself with roughly 430 visitors a side, which is under a thousand in total against sixteen thousand for the one-point lift. Big differences are the only kind that small traffic can detect.

So test things that could plausibly double the result: a different customer, a different price, an annual contract against pay-per-use. These are offer changes, and they are where an early product’s uncertainty sits anyway, which is more than we can say for the colour of the button.

This changes the agent’s job. Forty variants are no use to us, so we ask it for the two most different offers that are both defensible, each built out properly with page, funnel, follow-up email and tracking. One agent builds and another checks that the tracking fires before any traffic is spent, and then the agents stop while the traffic does its slow work. In our method this is the landing-page test. It runs before the product is built and it is deliberately narrow, with two arms and sometimes three, and an arm gets no visitors unless we can say why it might win.

VI Write the rule first

On day one, before the first visitor, we write down four things.

• The metric, and it is one metric.
• The threshold that counts as a pass.
• The sample or the date at which we read it, whichever comes first.
• What we will do on a pass, on a fail, and on a muddle.

The last line matters most, because most small tests end in a muddle, and a team with no rule for muddle will run the test “a little longer” until it says something agreeable. Our default for a muddle is that the offer did not clear the bar, so the offer changes. That is harsh, though it costs less than letting an inconclusive test run on.

Agents make this rule more necessary. A person who has to build each variant by hand is rationed by their own laziness, which turns out to have been a useful statistical control. Take that away and what stands between you and a hundred half-read experiments is a sentence written on day one. After the test, show the surviving offer to people you trust and watch someone use it. The number tells you that an arm won and watching tells you why, and the why is what the build, and later operations, will need.