Onboarding Checklist A/B Test Protocol: 7-Day Activation, SMB Signups (US and UK), 1,585 Per Arm Over 6 Weeks
Interpreted as: a two-arm randomised A/B test (checklist vs current onboarding) on new SMB signups to a B2B SaaS product in the US and UK, with the primary outcome "activated within 7 days of signup", measured in Amplitude, inside a 6-week cap, no incentives, and UK GDPR (and EU GDPR if any EU-resident users appear) applying. The activation event, the current baseline and weekly signup volume were not supplied, so they are assumptions (see section 9).
Design in one line: individually randomised 1:1 parallel-group trial, 1,585 new SMB accounts per arm (3,170 total, including 5% allowance for exclusions), about 3 weeks of enrolment plus a 7-day follow-up, inside 6 weeks, powered to detect a 5 percentage point absolute lift from a 37.5% baseline.
1. Objective and hypotheses
- Primary hypothesis (directional): Showing new SMB signups an in-product onboarding checklist raises the share who reach the activation event within 7 days of signup.
- Null (H0): The 7-day activation proportion is the same in the checklist and control arms (p_T = p_C).
- Alternative (H1): p_T > p_C. Power is computed for a two-sided test (more conservative); the decision rule is a two-sided test with the lift in the positive direction.
- Secondary hypotheses:
- S1: The checklist shortens median time from signup to activation.
- S2: The checklist raises 7-day activation in both countries (US and UK), tested as a consistency check, not a separate confirmatory test.
- S3: The checklist does not reduce a guardrail metric (7-day return visit in a different session) by more than a pre-set margin.
2. Design and rationale
- Design: Parallel-group randomised controlled experiment, 1:1 allocation, reported against the 2025 CONSORT statement (30-item checklist and flow diagram) and with the protocol written to SPIRIT 2025 (34 items), both published at the SPIRIT-CONSORT site. These are trial guidelines, so I use them as a reporting scaffold, not as a regulatory requirement.
- Unit of randomisation: The new account (one workspace), not the user, so that several teammates in one SMB account never see different experiences. Assignment is made at account creation by a deterministic hash of account ID and an experiment salt. Stratify (block randomise) by country (US, UK).
- Arms: Control = current onboarding. Treatment = current onboarding plus a persistent checklist. Item count, wording and ordering are frozen before launch.
- Blinding: Users cannot be blinded. Analysts are blinded to arm labels (arms coded A/B) until the analysis code is frozen.
- Why this design beats the alternatives: A pre/post comparison is confounded by seasonality and marketing mix. A holdout-by-region design confounds country with treatment. Individual-level randomisation within one product removes both, and the account-level unit prevents contamination between colleagues.
3. Participants
Inclusion: New accounts created during the enrolment window, classified as SMB by the product's existing firmographic rule (confirm the threshold), with a US or UK signup country, and reaching the first post-signup screen where the checklist would appear.
Exclusion: Internal and test accounts, accounts created by sales-assisted or white-glove onboarding (the checklist would overlap with a human), duplicate accounts from the same email domain within the window (keep first), and users under 18 (not expected in B2B; confirm).
Recruitment channel: None. This is a natural-traffic experiment on all eligible organic and paid signups. No incentive is offered, consistent with the constraint.
Sample-size assumptions
| Parameter | Value | Basis |
|---|---|---|
| Baseline 7-day activation | 37.5% | Userpilot 2025 benchmark: 37.5% average (37.04% median) activation rate across 62 B2B SaaS companies; the source does not state a 7-day window or an SMB cut, it is vendor first-party data, and activation definitions differ by product, so treat as a placeholder to replace with your own trailing 8 weeks |
| Absolute MDE | +5 pp (37.5% to 42.5%) | Estimate: no independent randomised study of onboarding checklists on SMB activation was found; vendor case studies in search results were not opened or used. Chosen as the smallest lift I would act on given the cost of the feature (a judgement for the product owner) |
| Alpha | 0.05, two-sided | Convention (estimate) |
| Power | 0.80 | Convention (estimate) |
| Allocation | 1:1 | Maximises power for fixed n (estimate) |
| Attrition/exclusion allowance | 5% | Estimate, covers bot, test and duplicate account removal; no source |
Method: Two-sample proportion test, statsmodels NormalIndPower with Cohen's h from proportion_effectsize (script: data/power.py, output: data/power_output.txt). Result: 1,506 per arm for 37.5% vs 42.5%, 1,585 per arm after the 5% allowance, 3,170 total.
Sensitivity (n per arm before attrition allowance, same method):
| Baseline | +3 pp | +4 pp | +5 pp |
|---|---|---|---|
| 25.0% | 3,395 | 1,932 | 1,250 |
| 30.0% | 3,762 | 2,133 | 1,376 |
| 37.5% | 4,148 | 2,344 | 1,506 |
Feasibility against the 6-week cap: 3,170 accounts over a 3-week enrolment window needs about 1,057 eligible new accounts per week (derived). If real volume is lower, the detectable lift grows. At 30% baseline, 1,000, 1,500, 2,000 and 3,000 accounts per arm detect +5.88, +4.78, +4.13 and +3.36 pp respectively; at 37.5% baseline the same sizes detect +6.15, +5.01, +4.33 and +3.53 pp (computed, data/power_output.txt). If volume is below roughly 1,000 per week, extend enrolment to 4 weeks (leaving 1 week for analysis) or accept an MDE of about +6 pp.
4. Tasks, measures and endpoints
| Endpoint | Exact definition | Window and unit | Source and version |
|---|---|---|---|
| Primary: 7-day activation | Account has at least one user firing the product's single pre-registered activation event (placeholder Activation Event; confirm with product, for example "first core object created and shared") |
Within 7x24 hours of the account's Account Created timestamp, per account |
Amplitude event stream, taxonomy version tag recorded at pre-registration |
| S1: time to activation | Hours from Account Created to first Activation Event, among activated accounts |
Right-censored at 7 days, per account | Amplitude |
| S2: country consistency | Primary endpoint split by US/UK | Per account | Amplitude user property country (derived from billing or signup country, not IP, to limit personal data) |
| S3: guardrail | At least one session on day 2 to day 7 (a return visit) | Per account | Amplitude session events |
| Exposure check | Experiment Exposure event fired when the checklist or its control placeholder first renders |
Per account | Amplitude, deduplicated per account |
| Process: checklist engagement | Checklist items completed and dismissals | Treatment arm only, descriptive | Amplitude |
The primary analysis is intention-to-treat on all randomised, eligible accounts, whether or not they saw the checklist.
5. Procedure and timeline (6 weeks)
| Phase | Duration | Owner | Deliverable |
|---|---|---|---|
| Setup: freeze activation definition, checklist content, event taxonomy; build assignment and exposure logging | Week 1 | Product analyst and engineer | Experiment spec, Amplitude events in staging |
| Approval: privacy review, DPIA screening, notice and consent wording, pre-registration | Weeks 1 to 2 (overlaps setup) | Data protection lead, product analyst | Signed DPIA screening record, time-stamped pre-registration |
| Pilot: 2 to 3 days at 5% traffic, run the SRM check and exposure checks | Week 2 (3 days) | Engineer, analyst | QA report; go or no-go |
| Collection: full enrolment, daily SRM monitoring (no outcome peeking) | Weeks 2 to 5 (about 3 weeks of enrolment plus last cohort's 7-day window) | Product analyst | Raw account-level export |
| Analysis: run the frozen script once on closed cohorts | Week 5 to 6 (3 to 4 days) | Data scientist | Analysis output, CONSORT flow diagram |
| Reporting | Week 6 (2 days) | Product analyst | Decision memo against CONSORT 2025 items, with ship or no-ship recommendation |
6. Bias and validity risks
| Risk | Type | Mitigation |
|---|---|---|
| Unequal arm counts from assignment or logging faults | Selection / measurement | Run a chi-square sample ratio mismatch test on arm counts before looking at outcomes. Microsoft's experimentation team uses p < 0.0005 as a conservative threshold and reports SRM in about 6% of its tests (LinkedIn about 10% of zoomed-in tests); that threshold equals a chi-square above 12.12 with 1 degree of freedom (computed). Do not read results from an SRM-failed test |
| Analysing only users who saw the checklist (triggered analysis) | Selection | Primary analysis uses all randomised eligible accounts. The same Microsoft article describes severe SRM when exposure logging exists in only one arm, so log exposure for control too |
| Teammates in one account see different arms | Contamination | Randomise at account level |
| Novelty effect inflates early lift | Novelty | Report activation by signup week; treat a lift that decays across cohorts as a flag |
| Activation defined after seeing results | Measurement | Freeze the event in the pre-registration |
| Activation event is itself a checklist item, making the lift mechanical | Measurement | Confirm the activation event is a product outcome that users can reach without the checklist; flag in the memo if checklist items equal the activation event |
| Bots, test accounts, duplicates | Selection | Exclusion rules applied by script before unblinding |
| Campaign or seasonality shifts mid-test | Confounding | Randomisation balances them across arms; stratify by country; log major campaigns |
| Opt-out of analytics creates missing outcome data | Attrition / measurement | If a user objects to analytics, event data is unavailable; compare opt-out rate by arm; see section 7 |
| UK and US users differ in baseline | Confounding | Stratified randomisation and country-adjusted estimate as a sensitivity analysis |
7. Analysis plan
- Primary test: Two-sided two-proportion z-test (equivalent chi-square) of 7-day activation, treatment vs control, intention-to-treat, alpha 0.05. Report the absolute difference with a 95% confidence interval and the relative lift.
- Secondary tests: S1 log-rank or Kaplan-Meier on time to activation, censored at day 7. S2: logistic regression with a treatment-by-country interaction (exploratory). S3: one-sided non-inferiority on return visit with a margin to be fixed in the pre-registration (suggest 2 pp; estimate).
- Multiplicity: One confirmatory primary test, so no correction for it. Secondaries are labelled exploratory, with Holm adjustment across S1 and S3 if either is used in the decision.
- Missing data: Account with no event data after exposure counts as not activated only if analytics were not objected to; accounts with analytics objection are reported separately and excluded from the primary denominator (disclose by arm). Sensitivity: worst-case imputation.
- Stopping rule: Fixed horizon. Enrolment ends when 1,585 eligible accounts per arm have been randomised (planned at about 3 weeks), with a hard stop at 4 weeks of enrolment; each account is then followed for its full 7 days before analysis. No peeking: no outcome analysis before then; only SRM and logging health are monitored. If sequential monitoring is later wanted, choose a named method (for example a group-sequential alpha-spending plan) before launch.
- Pre-registration venue: OSF or AsPredicted (neither page was opened in this run; confirm venue format). Register the activation definition, primary test, exclusions, stopping rule and secondary analyses before launch.
8. Ethics and consent checklist
This is a planning aid, not legal advice. A qualified data protection officer or counsel should confirm each row.
| Item | Status | Reason and source |
|---|---|---|
| Research ethics committee (IRB) review | Not applicable (confirm) | Product experiment on a commercial service, not human-subjects health research; confirm with company policy. No health intervention is involved |
| Consent to the experiment itself | Likely required to inform; consent not necessarily the lawful basis | Disclose experimentation in the privacy notice; no source opened on notice wording for experiments, so this is an estimate to confirm |
| Lawful basis for processing event data (UK GDPR) | Required: decide and record | Legitimate interests is the likely candidate; I found the ICO pages (what legitimate interests means) via search but did not open them, so the three-part test is not summarised here |
| Cookie or SDK storage consent (PECR) for Amplitude in the UK | Likely required unless an exception fits | The ICO says storage or access for statistical purposes is exempt only if the sole purpose is aggregate statistics to improve the service, with clear information and a simple free way to object. It lists A/B testing, comparing two groups' interactions with two versions, as likely to meet that exception, but it says consent is needed for tracking or profiling individual visitors, and for connecting a visitor ID to site activity (its example is conversions shared with advertising partners). A logged-in, account-keyed event stream may fall on the consent side, so ask counsel. Third-party analytics providers must act only as a processor |
| DPIA | Likely required to screen; full DPIA only if high-risk criteria are met | Article 35 requires a DPIA for processing likely to result in high risk; the ICO list includes tracking of behaviour online, which requires a DPIA when combined with another European-guideline criterion; the ICO says two criteria together usually indicate a DPIA. Document a screening decision either way. The ICO notes this guidance is under review after the Data (Use and Access) Act, so re-check before launch |
| Personal fields collected | Required: minimise | Account ID (pseudonymous hash), signup timestamp, country (from billing/signup country, not IP), SMB tier, arm, event names and timestamps. Do not collect names, emails or free text in the analysis dataset; keep email out of Amplitude user properties |
| Retention | Required: set and document | Keep account-level experiment data only as long as needed to analyse and archive the result; ICO says individual-level data used under the statistical exception must be kept no longer than needed for aggregation (same ICO exceptions page). Proposed: delete raw account-level export 90 days after decision (estimate; confirm) |
| Vulnerable groups | Not applicable (confirm) | B2B business users; the ICO treats children and people with power imbalances as vulnerable; exclude under-18s |
| Incentives | Not applicable | None offered |
| International transfers (UK data stored in US Amplitude or US data in UK) | Likely required | Confirm Amplitude data region and transfer mechanism; the ICO notes international transfers must be considered for third-party analytics (ICO exceptions page above) |
| US privacy law | Likely required to check | State privacy laws may apply to US users; no primary source was opened, so confirm with counsel |
9. Open assumptions to confirm
- Baseline is 37.5%. If the product's real trailing baseline is lower (for example 30%), n per arm rises to 1,376 at +5 pp (computed). Confirm: product analyst, from 8 weeks of Amplitude data.
- Activation event and the 7-day clock. Changes endpoint validity and baseline. Confirm: product owner.
- +5 pp is the smallest lift worth acting on. If the product owner needs +3 pp, n rises to about 4,148 per arm at 37.5% baseline, which is unlikely to fit in 6 weeks. Confirm: product owner.
- Weekly eligible signup volume of about 1,057 or more. Below that, the MDE exceeds 5 pp. Confirm: growth analyst.
- Account-level randomisation is technically possible with the existing experiment tool. Confirm: engineering.
- Legal position on Amplitude storage and consent (UK PECR, UK GDPR, EU GDPR if EU users appear), DPIA outcome and transfer mechanism. Gates launch. Confirm: data protection lead or counsel.
- Checklist is the only change. Any concurrent onboarding change breaks attribution. Confirm: product manager.
- No credible randomised effect-size benchmark for checklists was found. The +5 pp MDE is a design choice, not an evidence-based expected effect. Confirm: product owner and data scientist.
Do next
- Pull 8 weeks of Amplitude data to replace the 37.5% benchmark baseline and measure weekly eligible signups, then re-run
data/power.py. - Fix and freeze the activation event, checklist content and exclusion rules in a written spec.
- Send the Amplitude storage, lawful basis, DPIA screening and transfer questions to the data protection lead or counsel.
- Pre-register the protocol (OSF or AsPredicted) before any traffic is assigned; run the 5% pilot with the SRM check.
- Treat the design, sample size and regulatory status here as recommendations for the responsible people to confirm; they are not legal or regulatory advice.
Coverage
Not reached or not used: the Userpilot 2024 benchmark blog post (404); ICO legitimate interests pages and EU ePrivacy/EDPB sources (found in search, not opened); OSF and AsPredicted pages; Amplitude documentation; vendor case studies on checklist lift (appeared in search, not opened, not relied on); EQUATOR Network page (not opened).
Sources
- Userpilot SaaS Product Metrics Benchmark Report 2025
- ICO: What are the exceptions? (storage and access technologies, PECR)
- ICO: When do we need to do a DPIA?
- Microsoft Research: Diagnosing Sample Ratio Mismatch in A/B Testing (14 September 2020)
- SPIRIT-CONSORT 2025 website
- ICO: What is the legitimate interests basis? (found in search, not opened)
- Local files:
data/power.py,data/power_output.txt(statsmodels power calculations, run 6 October 2026)