A test can produce a precise p-value and still leave a team unsure what to do. The fix starts before launch: write down the decision, the metric that answers it, the smallest useful effect, and what would count as unacceptable harm.

  • Define the decision first: conversion is primary; subscriber quality needs its own, adequately sized guardrail.
  • Size for a worthwhile change: the minimum detectable effect (MDE) is a planning threshold, not a promise about the result.
  • Treat uncertainty as a result: this seeded example is inconclusive, so it supports neither shipping nor declaring the idea ineffective.
All figures below come from a seeded simulation generated by this page's code. They are illustrative, not results from a real product test.

At Chegg, I designed measurement for an ML-driven subscription-targeting initiative, including cohorts, primary and guardrail metrics, and evaluation. This worked example uses a fictional offer-page change to make those design choices concrete.

Start with the decision

Suppose a team wants to test a new subscription offer page. The question is: does it increase the share of eligible visitors who subscribe, while keeping subscriber outcomes within an acceptable range?

That sentence identifies the population (eligible visitors), the primary outcome (paid conversion), and a secondary concern (what happens after conversion). It also exposes a subtle issue: cancellation among converters conditions on a post-treatment outcome. If the page changes who subscribes, comparing cancellation only among subscribers is a useful diagnostic, but not a clean causal estimate of customer harm. A stronger causal guardrail is retained subscriptions per randomized visitor, or a pre-specified non-inferiority outcome measured across all assigned visitors.

Map the assignment and measurement

For a live test, I would set up the following. The aggregate simulation below illustrates the outcomes; it does not implement a visitor-assignment service.

  • Randomization unit: visitor, assigned once and kept in the same arm.
  • Allocation: 50/50 using a stable visitor-ID hash at first offer-page exposure.
  • Eligible population: visitors who reach that page; record assignment before any outcome is observed.
  • Primary metric: paid subscribers divided by randomized eligible visitors.
  • Guardrail: specify a customer outcome across all assigned visitors where possible. The simulated 7-day cancellation rate among converters is included as a diagnostic, with the selection caveat above.

Write the decision rule before seeing results. For this example, the team would consider shipping if conversion improves convincingly and the pre-defined customer guardrail remains within its non-inferiority margin. “Not statistically significant” on a guardrail does not prove no harm.

Experiment decision path

  1. Define the decision

    Name the eligible population, change, primary outcome, and decision rule.

  2. Size for the MDE

    Choose the smallest worthwhile effect, alpha, power, and fixed horizon.

  3. Validate the run

    Check assignment, exposure, instrumentation, and any sample-ratio mismatch.

  4. Read estimate + interval

    Assess the primary metric and a decision-capable guardrail together.

  5. Apply the prewritten rule

    Use the estimate, interval, and guardrail result to choose an outcome.

    • Rule is metShip
    • Rule is not metDo not ship; stop or plan a new test
Each checkpoint is decided before outcomes are reviewed.

Set the minimum useful effect and sample size

The current page converts 4.0% of visitors. The minimum detectable effect (MDE) is 10% relative: from 4.0% to 4.4%, an absolute change of 0.4 percentage points. It represents the smallest improvement worth detecting at the chosen cost and risk—not an expected outcome.

For a two-sided test at α = 0.05 and 80% power, a standard two-proportion approximation gives:

n per arm = ( z(1−α/2)·√(2·p̄(1−p̄)) + z(power)·√(p₀(1−p₀) + p₁(1−p₁)) )² ÷ (p₁ − p₀)²
          = 39,475 visitors per arm (78,950 total)

At the MDE, 80% power means this design would detect the specified effect in about four of five repeated runs under the model assumptions. It would miss it in about one of five. It does not mean an 80% chance that this particular result is true.

Estimate duration from eligible traffic, then round to complete weeks so weekday and weekend patterns are represented. Fix the end date before launch.

Run the test with checks in place

  • Avoid unplanned peeking. Repeated significance checks with ordinary fixed-horizon p-values inflate false positives. If early decisions matter, choose a valid sequential design in advance.
  • Investigate sample-ratio mismatch (SRM). This run assigned 39,392 control and 39,558 treatment visitors; the 50/50 check gives p = 0.55. No mismatch is detected by this check; that does not prove assignment or logging are valid. In a real test, diagnose an SRM before interpreting treatment effects. See Microsoft Research's SRM troubleshooting guide.
  • Check exposure and logging. Validate assignment persistence, eligibility, missing outcomes, and instrumentation before reading lift.
  • Let outcome windows finish. A 7-day cancellation measure needs seven days of follow-up for every included new subscriber. Stopping enrollment and reading the final outcome are separate dates.

Read the estimate and its uncertainty

MetricControlTreatmentDifference (95% CI)p-value
Paid conversion4.06%4.25%+0.19 pp (−0.09 pp to +0.47 pp)0.175
7-day cancellation among converters (diagnostic)12.2%11.2%−0.96 pp (−3.16 pp to +1.24 pp)0.393

Conversion is 4.7% higher in relative terms in this run, but its interval runs from −0.09 pp to +0.47 pp and includes zero. The result is inconclusive under the stated decision rule. It is compatible both with no lift and with the planned effect; it does not justify shipping on this evidence or conclude the change has no value.

The simulation set the true effect to +0.40 pp, exactly the MDE. That is known only because this is a simulation. The observed estimate is one noisy draw.

075150no effecttrue effect +0.40 ppThis run: +0.19 pp, 95% CI −0.09 pp to +0.47 pp0+0.2+0.4+0.6+0.8Measured difference, treatment − control (percentage points)
Simulated data. 1,000 reruns of the same test (bars), the true simulated effect, and the single run discussed above with its 95% confidence interval.
Measured difference across 1000 simulated reruns
Difference rangeReruns
−0.15 pp to −0.10 pp0
−0.10 pp to −0.05 pp0
−0.05 pp to +0.00 pp1
+0.00 pp to +0.05 pp3
+0.05 pp to +0.10 pp10
+0.10 pp to +0.15 pp18
+0.15 pp to +0.20 pp33
+0.20 pp to +0.25 pp66
+0.25 pp to +0.30 pp82
+0.30 pp to +0.35 pp137
+0.35 pp to +0.40 pp139
+0.40 pp to +0.45 pp142
+0.45 pp to +0.50 pp125
+0.50 pp to +0.55 pp104
+0.55 pp to +0.60 pp54
+0.60 pp to +0.65 pp43
+0.65 pp to +0.70 pp23
+0.70 pp to +0.75 pp13
+0.75 pp to +0.80 pp3
+0.80 pp to +0.85 pp4
+0.85 pp to +0.90 pp0

Across 1,000 identical simulated runs, 82% are significant, close to the design's 80% target. The remaining 18% miss the built-in effect. The guardrail interval (−3.16 pp to +1.24 pp) is wide because the simulated converter-only sample is small; non-significance here is not evidence of safety.

Choose the next step

  1. Do not ship from this run alone. Decide whether the remaining uncertainty is worth resolving.
  2. If continuing, plan a fresh test. Set its sample size, stopping rule, and end date before collecting more data; do not add “one more week” because the current estimate is close.
  3. Make guardrails decision-capable. State an acceptable harm margin and power an all-assigned-user outcome to assess it.
  4. Report absolute and relative change with the interval. “+0.19 pp, with a 95% interval from −0.09 pp to +0.47 pp” is more actionable than a p-value alone.

For a related worked example, see how I compare a seasonal forecast with a baseline, or explore the ad-auction floor-price simulator.

Method: the simulation samples aggregate arm sizes and outcomes with a fixed-seed random generator; it does not model stable-hash assignment, repeated visits, exposure logging, or elapsed follow-up time. Binomial draws are approximated by a normal distribution at these sample sizes. Confidence intervals use unpooled standard errors and p-values use a pooled two-proportion z-test. The figures and histogram are generated from the same computation.