How I Design and Read an A/B Test, With a Simulated Example

Everything below uses simulated data, generated by this page's own code from a fixed seed. It illustrates the method; none of these numbers are real results.

Experiments let product and pricing decisions rest on evidence instead of opinion, but only if the test is designed before it runs. At Chegg I designed the measurement for an A/B test of an ML-driven subscription-targeting initiative: test and control cohorts, primary and guardrail metrics, and the statistical evaluation. This walkthrough applies the same steps to a made-up change, with every number produced by a simulation so the whole process can be shown end to end.

The change and the question

Suppose a team wants to test a new subscription offer page. The question to settle before anything is built: does the new page increase the share of visitors who subscribe, without hurting the quality of the subscribers it brings in?

Writing the question that precisely does most of the design work. It names the unit (visitors), the primary metric (conversion to paid), and the guardrail (subscriber quality).

Test and control cohorts

• Unit of randomization: the visitor, assigned once and kept in the same arm for the whole test, so no one sees both pages.
• Split: 50/50 by a stable hash of the visitor ID, decided at the first exposure to the offer page.
• Who is in: only visitors who reach the offer page. Everyone else would add noise to both arms without being able to react to the change.

Primary and guardrail metrics

• Primary metric: conversion to paid, subscribers ÷ visitors in each arm. One primary metric keeps the decision rule unambiguous.
• Guardrail: cancellation within 7 days among new subscribers. A page that converts more people who regret it within a week is not a win.

The decision rule is written down before launch: ship if conversion rises with a 95% confidence interval above zero and the guardrail shows no meaningful increase.

Sample size and power

The current page converts 4.0% of visitors. The smallest lift worth acting on, the minimum detectable effect (MDE), is 10% relative: from 4.0% to 4.4%, or 0.4 percentage points. For a two-sided test at α = 0.05 with 80% power, the standard two-proportion formula gives:

n per arm = ( z(1−α/2)·√(2·p̄(1−p̄)) + z(power)·√(p₀(1−p₀) + p₁(1−p₁)) )² ÷ (p₁ − p₀)²
          = 39,475 visitors per arm (78,950 in total)

Eighty percent power means that if the true lift is exactly the MDE, the test will call it significant about four times in five and miss it about one time in five. That matters below.

The run length follows from expected traffic, rounded up to whole weeks so weekday and weekend behavior are both covered, with the end date fixed in advance.

Running it

Two rules while the test runs:

• No peeking and stopping early. Checking significance every day and stopping at the first p < 0.05 pushes the false-positive rate far above 5%. If early stopping is a real need, use a sequential method designed for it.
• Check the split. A sample ratio mismatch, arms far from the planned 50/50, usually means a bug in assignment or logging and invalidates the comparison. In the simulation the arms came out at 39,392 and 39,558 visitors; a chi-square test against 50/50 gives p = 0.55, so no mismatch.

Reading the result

Metric	Control	Treatment	Difference (95% CI)	p-value
Conversion to paid	4.06%	4.25%	+0.19 pp (−0.09 pp to +0.47 pp)	0.175
7-day cancellation	12.2%	11.2%	−0.96 pp (−3.16 pp to +1.24 pp)	0.393

The new page converted 4.7% better in relative terms, but the confidence interval runs from −0.09 pp to +0.47 pp: it includes zero. By the rule written before launch, the test is inconclusive. That is not evidence that the page does nothing, since the interval also includes the hoped-for lift, but it is not enough to ship on.

This is where a simulation can show something a real test cannot. The true effect was built in at +0.40 pp, exactly the MDE. The page really does work; this run simply landed low.

Rerunning the identical test 1,000 times, the measured difference ranges from −0.02 pp to +0.84 pp, and 82% of the reruns reach significance: the 80% power the design promised. The other 18% miss a real effect, and the run above is one of them.

The guardrail carries its own lesson about uncertainty. Cancellations moved −0.96 pp, but because only new subscribers count toward it, its sample is small and the interval (−3.16 pp to +1.24 pp) is wide. "No significant change" from an underpowered guardrail is weak reassurance.

What I would do next

• Not ship on this run, and not drop the idea either. The result is consistent with the MDE. If the idea is still a priority, plan a follow-up with more traffic or a bolder change.
• Avoid "just one more week." Extending a test because it is close turns a fixed-horizon test into peeking. A follow-up needs its own sample size and end date.
• Size the guardrail on purpose. Set a non-inferiority margin up front (for example, cancellations may not rise by more than one point) and size the test so "no harm" is a finding rather than a hope.
• Report the interval, not just the p-value. "+0.19 pp, somewhere between −0.09 pp and +0.47 pp" tells a decision-maker far more than "not significant."

Method notes: visitors, conversions, and cancellations are drawn from a seeded random generator, with binomial draws approximated by a normal distribution at these sample sizes. Confidence intervals use unpooled standard errors; p-values come from a pooled two-proportion z-test.