Experiment Report: [Experiment Name]
1. Test Setup
Describe the experiment: what was tested, how visitors were assigned, sample sizes, duration, and allocation method.
2. Metric Definition
Define the primary success metric precisely. How is it computed? What is included and excluded? Why was this definition chosen over alternatives?
3. Results Summary
One paragraph for the stakeholder. What did the experiment show? Be honest about uncertainty. Use confidence intervals, not just point estimates.
4. Detailed Results
For each test, fill in one row:
| Test | Test type | n (A) | n (B) | Metric (A) | Metric (B) | Difference | 95% CI | p-value | Effect size | Interpretation |
|---|---|---|---|---|---|---|---|---|---|---|
5. Confounds and Limitations
List every threat to validity you identified. For each, describe:
- What the confound is
- How it could affect the results
- What you did to account for it (segmented analysis, sensitivity check, etc.)
- Whether it changes the interpretation
6. Recommendation
What should the stakeholder do? Be specific and actionable. Explain the reasoning behind the recommendation, including what the data supports and what it does not.