Before you split your ad budget, Runner tells you what the test can prove

Most small ad tests can't tell a better ad from a lucky week. Runner says so before the money moves, and what it would take.

Published · 3 min read

The headline beside a card reading: A 14-day test at today's budget proves a lift of 70% or more. To prove 20%: $850 a day per ad.

Every ad platform has an A/B button. The usual way to use it is the usual way to waste money: split the budget, wait a week, and move everything to whichever ad has the better number.

Here is what that test usually looks like. Fifty dollars a day, split in two, for two weeks. If an order costs you about $25 in ads, each ad gets around fourteen orders. Fourteen orders cannot tell a 20% better ad from a 20% worse one. At that count, only an ad roughly three times better would stand out from the noise. Everything else is a coin toss, read as a verdict and acted on.

What Runner says

Open Ads. Under the change queue there is a short row, one line per connected network: what a two-week test at today's budget can actually prove, and the budget that would prove a 20% difference, rounded up so it never flatters.

One row, three honest answers: a network sized, a test being called on day 13, and a network too thin to size, which Runner says rather than guesses. Shown on Harbor Lane, a demo business; every name is fictional.

One row, three honest answers: a network sized, a test being called on day 13, and a network too thin to size, which Runner says rather than guesses. Shown on Harbor Lane, a demo business; every name is fictional.

The numbers come from your own last 28 days on that network, what you spent and what came back, not from a benchmark. A network with fewer than thirty results in that window gets a different line: needs 30 results in 28 days to size a test. Runner won't size a test on a number that is mostly noise.

Test two ads, one thing different

Two ads that differ in one thing. The ad as it runs, and one ad with one change: the picture. Same audience, equal budgets, same bid, same schedule, placements held fixed. Pick the ad, pick the other picture, pick how long, and Runner drafts both arms as one change for your approval, with the test's metric, duration, budget and stopping rule written into it.

That last part matters more than it sounds. Ad platforms show each ad to the people most likely to respond to it, so two ads in one test reach different people. Hold everything but the creative still, and what's left is the creative.

Calling it

The same row calls a running test from the two ads' own results. Below ten results on either side it says so: test running, day 4 of 14, too early to call. A winner is named only when the whole 95% interval clears zero: the new picture wins, 42% better. When it doesn't clear, Runner says no difference proved yet, and after the days are up, no difference after 14 days. That last line is a result too, and the one most tests end on.

Nothing is stored as a verdict. The judgement runs on every read, from the numbers the network reported, so the row never disagrees with the account.

What to test next

Most ad accounts say the same thing five ways. Runner reads the words of every live ad and asks two plain questions: why would someone buy, and how does the ad say it?

You get one sentence back: your 14 ads give 7 of the 8 reasons to buy, and nothing says it saves time. Then one suggestion, the missing reason told in the form already bringing you the most results, and one button that starts the brief with it filled in.

Every pink square is ads you run. Every plus is an ad nobody has made yet. Shown on Harbor Lane, a demo business; every name is fictional.

Every pink square is ads you run. Every plus is an ad nobody has made yet. Shown on Harbor Lane, a demo business; every name is fictional.

That grid is your list of next tests, already ranked by what works for you.

Give your business a growth team.

Create your account and pick a plan. Your team is built once the plan is paid.

More news

All news