Analysis / Analyst

A/B Test Analysis Framework

Create rigorous A/B test analysis plans with hypothesis formulation, sample size calculations, and statistical decision frameworks.

ChatGPT, Claude · 0 copies · 0 views

Customize this prompt

Fill variables before copying. Required fields stay visible in the preview until completed.

Test Name *Name or description of the A/B test

Primary Success Metric *The main metric you are trying to improve (e.g., conversion rate, click-through rate)

Expected Sample Size *Expected number of users or sessions per variant

Planned Test Duration *How long do you plan to run the test? (e.g., 2 weeks, 30 days)

Rendered Prompt

You are a data scientist specializing in experimentation and statistical analysis. Create a comprehensive A/B test analysis framework for {{test_name}}.

Develop a detailed analysis plan with:

1. **Test Hypothesis**: Formalize the hypothesis:
   - Null hypothesis (H0): No difference between control and treatment
   - Alternative hypothesis (H1): State the expected direction and magnitude
   - Business rationale for the test

2. **Success Metrics**: Define {{success_metric}} with:
   - Primary metric definition and calculation
   - Secondary metrics to monitor
   - Guardrail metrics (metrics that must not degrade)
   - Minimum detectable effect (MDE)

3. **Sample Size Calculation**: Based on {{sample_size}} and {{duration}}:
   - Statistical power analysis (target 80% power)
   - Required sample size per variant
   - Estimated test duration
   - Traffic allocation strategy

4. **Analysis Plan**: Outline the statistical approach:
   - Statistical test selection (t-test, chi-square, etc.)
   - Confidence interval methodology
   - Multiple comparison corrections if applicable
   - Segmentation analysis plan

5. **Decision Framework**: Define criteria for:
   - Statistical significance threshold (p-value)
   - Practical significance threshold
   - Rollback criteria
   - Ship criteria

6. **Reporting Template**: Structure for communicating results to stakeholders.

7. **Potential Pitfalls**: Identify common issues like novelty effects, Simpson's paradox, and sample ratio mismatch.

Ensure the framework follows statistical best practices and accounts for common experimentation biases.

How to use

Provide the test name, your primary success metric, expected sample size, and planned test duration. This generates a complete analysis plan you can use before launching your experiment to ensure statistical rigor.

Example output

A complete A/B test analysis plan for the checkout redesign test. Includes hypothesis formulation, sample size validation (50k users provides 85% power to detect a 2% relative lift), statistical test selection (chi-square test for proportions), and a decision framework requiring p < 0.05 and at least 1% relative improvement to ship.

Related prompt templates

More Analyst prompts to try.

Explore similar templates for ChatGPT workflows and related use cases.

AnalystChatGPT

Metric Dashboard Narrative Writer

Generates a concise executive narrative summarizing dashboard metrics, trends, and recommended actions for non-technical stakeholders.

You are a data storyteller writing for {{audience}}. Given the following metrics from the {{dashboard_name}} dashboard over {{timeframe}}: {{metrics}}. Write a 150-200 word executive narrative that: opens with the single most important takeaway, highlights 2-3 notable trends, calls out any metric that crossed a warning threshold, and closes with one clear recommended action. Use plain language. Structure as: headline, key numbers, trend context, risk flag, recommended action.

AnalystChatGPT

Python Data Cleaning Script Generator

Generates a production-ready Python data cleaning script tailored to specific data quality issues.