Assisted Project: Marketing Campaign A/B Test Analysis

You have results from an email marketing A/B test. Group A received the old subject line; Group B received a personalized subject line. Your job: determine if the personalized subject line performed better, whether the difference is statistically significant, and what the business impact is.

---

Project Goal

Determine whether the personalized email subject line (Group B) has a statistically significantly higher conversion rate than the control (Group A), and estimate the annual revenue impact.

---

Dataset Schema

ab_test.csv — one row per email sent:

| Column | Type | Description |
|--------|------|-------------|
| user_id | int | Unique user identifier |
| group | str | 'control' or 'treatment' |
| opened | int | 1 = opened, 0 = not |
| clicked | int | 1 = clicked, 0 = not (subset of opened) |
| converted | int | 1 = purchased, 0 = not (subset of clicked) |
| revenue | float | Revenue generated (0 if no purchase) |

---

Requirements

Build ab_analysis.py that:

1. Loads and validates the dataset — checks for nulls, confirms group balance, validates binary columns
2. Computes funnel metrics per group: open rate, click rate, conversion rate
3. Runs statistical tests — chi-squared for conversion rate comparison, t-test for revenue per user
4. Computes business impact — absolute lift, relative lift, estimated annual revenue difference
5. Visualizes the funnel — grouped bar chart showing open/click/conversion rates for both groups
6. Writes the recommendation — a 3-sentence executive summary

---

Architecture Hints

Hint 1 — Contingency table for chi-squared:

import pandas as pd
from scipy.stats import chi2_contingency

contingency = pd.crosstab(df['group'], df['converted'])
chi2, p_value, dof, expected = chi2_contingency(contingency)

Hint 2 — Revenue t-test (include zero-revenue rows):

from scipy.stats import ttest_ind

control_rev = df[df['group'] == 'control']['revenue']
treatment_rev = df[df['group'] == 'treatment']['revenue']
t_stat, p_rev = ttest_ind(treatment_rev, control_rev)

Hint 3 — Business impact calculation:

control_cvr = df[df['group']=='control']['converted'].mean()
treatment_cvr = df[df['group']=='treatment']['converted'].mean()
absolute_lift = treatment_cvr - control_cvr
relative_lift = absolute_lift / control_cvr

# Scale to annual
monthly_emails = 100000
avg_order_value = 45
annual_revenue_impact = monthly_emails * absolute_lift * avg_order_value * 12

Hint 4 — Funnel visualization:

import matplotlib.pyplot as plt
import numpy as np

stages = ['Open Rate', 'Click Rate', 'Conversion Rate']
control_rates = [open_rate_c, click_rate_c, cvr_c]
treatment_rates = [open_rate_t, click_rate_t, cvr_t]

x = np.arange(len(stages))
width = 0.35

fig, ax = plt.subplots(figsize=(9, 5))
ax.bar(x - width/2, [r*100 for r in control_rates], width, label='Control', color='#CBD5E1')
ax.bar(x + width/2, [r*100 for r in treatment_rates], width, label='Treatment', color='#2563EB')
ax.set_xticks(x)
ax.set_xticklabels(stages)
ax.set_ylabel('Rate (%)')
ax.set_title('Email Funnel: Control vs Treatment')
ax.legend()
plt.tight_layout()
plt.savefig('ab_funnel.png', dpi=150, bbox_inches='tight')

---

Milestone Checkpoints

Milestone 1: Data loaded and validated — group counts balanced within 5%, no nulls, all binary columns contain only 0/1 Milestone 2: Funnel metrics computed — open rate, click rate, and conversion rate calculated for both groups and printed in a clean table Milestone 3: Statistical tests passed — chi-squared p-value computed for conversion rate, t-test p-value for revenue per user Milestone 4: Business impact and recommendation written — absolute lift, relative lift, estimated annual revenue impact, and a 3-sentence executive summary printed

---

Completion Criteria

---

Executive Summary Template

Fill in the blanks from your analysis:

> "The personalized subject line (Group B) achieved a X% conversion rate vs Y% for the control — a Z% relative improvement that is statistically significant (p=P). At scale, this lift represents approximately $R in additional annual revenue. Recommendation: [ship / do not ship / run longer test] because reason."