A/B Test Dashboard
Track experiments, measure results, and make data-driven decisions.
🎯 What This Skill Does
| Capability | Description |
|---|
| Design Tests | Structure proper A/B experiments |
| Track Results | Monitor test performance |
| Calculate Stats | Statistical significance checks |
| Visualize Data | Dashboard creation |
| Report Findings | Communicate results clearly |
1. A/B Test Basics
What is an A/B Test?
| Term | Meaning |
|---|
| Control (A) | Your current version |
| Variant (B) | The change you're testing |
| Sample size | Number of visitors needed |
| Significance | Confidence results aren't random |
| Conversion | The action you're measuring |
What to Test (Priority)
| High Impact | Medium Impact | Low Impact |
|---|
| Headlines | Button text | Font size |
| CTAs | Images | Colors |
| Pricing | Form fields | Spacing |
| Offers | Layout | Icons |
| Landing pages | Navigation | Animations |
2. Sample Size Calculator
Quick Formula
Sample Size = (Z² × p × (1-p)) / E²
Where:
- Z = 1.96 for 95% confidence
- p = expected conversion rate
- E = margin of error (±)
Simple Reference Table
| Current Rate | Lift to Detect | Sample per Variation |
|---|
| 1% | 20% | ~40,000 |
| 2% | 20% | ~20,000 |
| 5% | 20% | ~8,000 |
| 10% | 15% | ~4,000 |
| 20% | 10% | ~3,000 |
Online Calculators
| Tool | URL |
|---|
| Optimizely | optimizely.com/sample-size-calculator |
| AB Tasty | abtasty.com/sample-size-calculator |
| Evan Miller | evanmiller.org/ab-testing/sample-size |
3. Dashboard Metrics
Core Metrics to Track
| Metric | Formula |
|---|
| Conversion Rate | Conversions / Visitors × 100 |
| Relative Lift | (B - A) / A × 100 |
| Confidence Level | Statistical p-value calculation |
| Visitors per Day | Total visitors / Days running |
| Days to Significance | Required sample / Daily visitors |
Dashboard Layout
┌─────────────────────────────────────────────┐
│ TEST: [Name] Status: RUNNING │
├─────────────────────────────────────────────┤
│ │
│ Control (A) Variant (B) │
│ ┌─────────┐ ┌─────────┐ │
│ │ 2.5% │ │ 3.1% │ │
│ │ Conv │ │ Conv │ │
│ └─────────┘ └─────────┘ │
│ │
│ Visitors: 5,432 Visitors: 5,489 │
│ Conversions: 136 Conversions: 170 │
│ │
│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ │
│ Lift: +24% Confidence: 87% │
│ ⚠️ Not yet significant (need 95%) │
│ │
├─────────────────────────────────────────────┤
│ Progress: ████████░░░░░░ 68% complete │
│ Est. completion: 4 more days │
└─────────────────────────────────────────────┘
4. Statistical Significance
What Does It Mean?
| Confidence | Meaning |
|---|
| 95% | 5% chance results are random (standard) |
| 99% | 1% chance results are random (high stakes) |
| 90% | 10% chance results are random (quick tests) |
Quick Significance Check
| Scenario | Likely Significant? |
|---|
| Big difference + many visitors | ✅ Yes |
| Small difference + few visitors | ❌ No |
| Big difference + few visitors | ⚠️ Wait |
| Small difference + many visitors | ⚠️ Maybe real, but small |
Avoid Common Mistakes
| Mistake | Problem | Solution |
|---|
| Stopping early | False positives | Wait for sample size |
| Peeking too much | Bias decisions | Set check schedule |
| Testing too many things | Can't attribute cause | One change at a time |
| Running too short | Misses weekly patterns | Run 1-2 full weeks |
5. Simple Dashboard with Google Sheets
Setup Steps
1. Create new Google Sheet
2. Tab 1: Raw data (date, variant, visitors, conversions)
3. Tab 2: Summary calculations
4. Tab 3: Charts
Formula Examples
| Metric | Google Sheets Formula |
|---|
| Conversion Rate A | =SUMIF(B:B,"A",D:D)/SUMIF(B:B,"A",C:C) |
| Conversion Rate B | =SUMIF(B:B,"B",D:D)/SUMIF(B:B,"B",C:C) |
| Lift | =(B_Rate-A_Rate)/A_Rate |
| Daily Visitors | =SUMIF(A:A,TODAY()-1,C:C) |
Simple Significance Formula
For approximation (use proper calculator for real tests):
Z = (pB - pA) / SQRT(p*(1-p)*(1/nA + 1/nB))
Where:
- pA, pB = conversion rates
- p = pooled rate = (convA + convB) / (nA + nB)
- nA, nB = sample sizes
If |Z| > 1.96, result is significant at 95%
6. Test Documentation
Pre-Test Template
## Test: [Name]
### Hypothesis
If we [change], then [metric] will [improve] because [reason].
### Test Details
- Page/Element:
- Control: [Description]
- Variant: [Description]
- Primary metric:
- Secondary metrics:
### Sample Size
- Current conversion rate: X%
- Minimum detectable effect: Y%
- Required visitors per variant: Z
### Timeline
- Start date:
- Expected end date:
- Check-in dates:
Post-Test Template
## Test Results: [Name]
### Summary
- Winner: [Control/Variant/No difference]
- Confidence: X%
- Lift: +/-Y%
### Data
| Metric | Control | Variant | Lift |
|--------|---------|---------|------|
| Visitors | | | |
| Conversions | | | |
| Conv Rate | | | |
### Learning
What did we learn?
### Next Steps
- Implement winner?
- Follow-up test?
- Share with team?
7. Tools for A/B Testing
Free/Cheap Tools
| Tool | Best For | Cost |
|---|
| Google Optimize | Websites (sunset, but alternatives exist) | Free |
| Splitbee | Simple tests | Free tier |
| Growthbook | Open source | Free |
| Posthog | Feature flags + analytics | Free tier |
Paid Tools
| Tool | Best For | Cost |
|---|
| VWO | Full CRO platform | $199+/mo |
| Optimizely | Enterprise | $$$$ |
| AB Tasty | Mid-market | $$$ |
| Convert | SMB friendly | $99+/mo |
8. Visualization Best Practices
Chart Types
| Data | Chart Type |
|---|
| Rate comparison | Bar chart |
| Trend over time | Line chart |
| Sample progress | Progress bar |
| Segment breakdown | Pie/donut |
| Confidence range | Error bars |
Dashboard Colors
| Status | Color |
|---|
| Winning (significant) | Green |
| Losing (significant) | Red |
| Running (not significant) | Yellow/Gray |
| Winner | Blue (neutral) |
Remember: Statistical significance ≠ practical significance. Always ask: "Is this lift worth the effort to implement?"
