A/B test

A learning project on simulated data: new trial accounts are split 50/50 between the current onboarding and a guided setup checklist. Success is activation, the first proposal sent within 14 days of signup. The checklist's true effect is planted in the simulation (30% to 34%), so every step of the analysis can be checked against the truth. Code and tests: experiments/.

1. Plan the size first

Accounts needed per arm

2,131

Days needed at 85 per arm per day

Power of the planned 28 days

84%

To detect a 4-point lift on a 30% base rate with a two-sided test at alpha 0.05 and 80% power, each arm needs

2,131 accounts. The test was planned for 28 days and analysed once, at the end.

2. The readout

Control activation

31.1%

Checklist activation

33.5%

Difference (points)

2.3%

p-value

0.083

The checklist arm activated 2.3% points more, with a 95% interval from

-0.3% to 5.0% . The interval contains the planted 4-point effect, but it also contains zero, so this run is **not significant** at 0.05. That is what 84% power means: about one run in six misses a real effect of this size. The honest call is "probably positive, not proven", not a win and not a failure.

Running the same design 1,000 times shows the method works as designed: it found the effect in 85.6% of runs, and the 95% interval covered the true lift in 94.0% .

No Results

3. Check the split before trusting the lift

A sample-ratio-mismatch check compares the observed split with the planned 50/50. On this run the split is fine (p = 0.19 ). The simulation also has a tracking bug on purpose: the new onboarding fails to log a fifth of the treatment accounts that never activate. The arms then come out

2,335 vs 2,065 , the check alarms (p = 0 ), and the broken data would have claimed a 8.2% -point lift, about twice the truth.

4. Why not stop early?

The difference swings a lot in the first days. Checking every day and stopping at the first p-value below 0.05 turns a 5% false-positive rate into a much bigger one. In 2,000 simulated A/A tests (both arms identical, so every "win" is false), testing once at day 28 gave 5.1% false positives; testing daily and stopping at the first significant day gave 25.6% .