In this guide
Your new recommendation model scores higher offline. Your teammate's simpler version is faster. Which one ships? Offline metrics — accuracy on a held-out set — measure the model. They do not measure the product: user behavior, latency tolerance, edge cases in the wild. A/B testing is how you compare two model versions on real traffic and let user behavior decide.
This guide covers designing an A/B test for ML models, the statistics you actually need, the ML-specific traps (interference, offline-online mismatch), and what to do when you do not have Google-scale traffic.
What an A/B test is (ML edition)
Split users (or requests) randomly: group A gets the current model (control), group B gets the new model (treatment). Measure a business metric — click-through, conversion, task completion, user retention — not just model accuracy. After enough traffic, compare.
The ML twist: you are testing a system (model + latency + UX + fallbacks), not just weights. A model with 2% better accuracy but 3x the latency can lose on user-facing metrics. That is the test working as intended.
Note: "A/B test" here means a randomized controlled experiment. Showing the new model to your friends and asking which they like is a user study, not an A/B test — useful, but different evidence.
Designing the test: the decisions that matter
1. The metric (decide before you start). Pick one primary metric tied to user value: purchase rate, query success rate, time-to-task-completion. Also track guardrail metrics (latency, error rate, server cost) that can veto a win. Writing the metric down before the test prevents cherry-picking whichever metric happened to improve.
2. The unit of randomization. Randomize by user (not by request) when users see multiple predictions — otherwise one user experiences both models and the experience is inconsistent. Randomize by request only for stateless, one-shot interactions.
3. Sample size (how long to run). This is the question everyone asks and the math everyone skips. The detectable effect shrinks with the square root of sample size: detecting a 1% relative lift needs roughly 100x the traffic of detecting a 10% lift. Use a sample-size calculator (many free ones exist) with your baseline conversion rate and the minimum lift you care about. A test that runs two days "to see" usually sees nothing but noise.
4. Duration. Run for at least one full weekly cycle — user behavior differs on weekends. Do not stop early because the result "looks significant" on day two (peeking inflates false positives; more below).
The statistics you actually need
You need less than you fear:
| Concept | What it means in practice |
|---|---|
| Null hypothesis | "The new model changes nothing." You need evidence to reject it. |
| p-value | Probability of seeing this result if nothing changed. Below 0.05 is the conventional bar. |
| Confidence interval | The range of plausible true effects. "Lift between +0.5% and +3%" beats "p < 0.05." |
| Statistical power | Probability of detecting a real effect. Underpowered tests waste everyone's time. |
| Practical significance | A statistically significant +0.1% lift nobody can feel is not a win. Define your minimum meaningful effect upfront. |
Report confidence intervals, not just p-values. "B beats A by 2.1% (95% CI: 0.8%–3.4%)" tells you both significance and magnitude.
# Proportion comparison sketch (two conversion rates)
from statsmodels.stats.proportion import proportions_ztest
count = [conversions_B, conversions_A]
nobs = [users_B, users_A]
stat, p_value = proportions_ztest(count, nobs)
print(f"p-value: {p_value:.4f}")
ML-specific traps
Offline-online mismatch. The model that wins offline loses online. Causes: training-serving skew, latency differences, the offline metric not matching the online metric, or feedback loops (the old model's outputs shaped the training data). Always treat offline wins as hypotheses, online tests as verdicts.
Interference (network effects). In marketplaces, social feeds, or shared-resource systems, treating one user affects others — violating the independence assumption. A better recommendations model for group B changes what group A sees too. Solutions: cluster randomization (randomize by region or cohort) or accept and bound the bias.
Novelty and primacy effects. Users click new things because they are new, then revert. Short tests overstate wins. Run long enough for novelty to fade, or discount early-period results.
Peeking. Checking results daily and stopping when significant inflates the false-positive rate dramatically. Pre-commit to a runtime (from your sample-size calculation) or use sequential testing methods designed for peeking.
Multiple comparisons. Testing 5 metrics × 3 segments = 15 chances to find a "significant" fluke. Pre-register the primary metric; treat the rest as exploratory.
When you do not have much traffic (the student reality)
Most student projects cannot run a powered A/B test — you need hundreds to thousands of users per group. Honest alternatives:
| Approach | What you get |
|---|---|
| Interleaving | Blend A and B results in one ranking; measure which gets clicked. Needs far less traffic than A/B. Standard for search/recommendation. |
| Offline evaluation + user study | Rigorous offline metrics plus 10–20 observed user sessions. Weaker evidence, but honest about it. |
| Counterfactual / replay evaluation | Log production data under model A; simulate what B would have done. No user impact; limited by logging completeness. |
| Shadow deployment | Run B on live traffic without showing its outputs; compare predictions and latency. Tests the system safely, measures no user behavior. |
A shadow deployment plus a small user study is a credible student-scale evaluation story: "B matched A's outputs within X%, ran Y% faster, and users preferred it in observed sessions."
From test to rollout
A winning test is not the end:
- Sanity-check the segments. Did B win everywhere, or only on one platform/browser/region? Segment wins can hide segment losses.
- Check guardrails. Latency, error rate, cost per request — a metric win with a cost explosion is not a ship.
- Roll out gradually. 10% → 50% → 100%, watching metrics at each step. This is also where you catch the bugs that only appear at scale.
- Keep the old model deployable. Versioned artifacts and a rollback path — the MLOps guide covers the discipline.
- Document the decision. What was tested, the result, the CI, what shipped. Future you will thank present you.
Common mistakes
- No pre-registered metric. Deciding what "won" after seeing the data.
- Stopping early on a peek. Day-two significance is usually noise.
- Testing without enough traffic. An underpowered test that "finds nothing" proves nothing.
- Randomizing by request for returning users. Inconsistent experience contaminates the measurement.
- Ignoring latency as a metric. The slower model loses users before its accuracy matters.
- Shipping on offline metrics alone. Offline is the audition; online is the performance.
- Forgetting seasonality. A test run entirely during exams week measures exams week, not normal behavior.
Where to go from here
A/B testing is part of running ML in production — read MLOps basics for the deployment and monitoring context. For choosing what to measure offline first, the ML evaluation metrics guide covers the metric foundations. If your models are LLMs, RAG evaluation covers testing generative systems. More production-ML topics in the AI & Machine Learning branch hub.