Stronger evidence before you move budget.
Where traffic and volume allow a test, a holdout or geo experiment estimates what the spend changed, with its range, so a budget decision rests on a measured effect. Feasibility is checked before any test budget is committed.
Problems it solves
Did marketing cause the outcome, or merely receive credit for it?
The most direct way to see what spend changed is to compare against a similar group that did not receive it. The right design depends on your traffic, conversion volume, geography, channels and constraints.
Credit is not causation
Platform-reported ROAS can include customers who would have bought anyway.
Tests without enough power
Experiments launched without checking whether they can detect the effect that matters.
Results nobody trusts
Tests analysed after the fact, with changing definitions, rarely settle a decision.
Evidence that is never reused
Experiment results that never feed back into everyday reporting.
Attribution shows which campaign received credit. An experiment estimates what the campaign caused, by comparing a group that received it with a comparable group that did not.
Choosing a design
The right design depends on what can be randomized. Each step down this path gives up some control over the design: a platform lift study is still randomized, but the platform runs and reports it.
Can individual users or accounts be randomized?
Yes
Individual holdout
No
Are there comparable geographic regions?
Yes
Geo experiment
Regions are matched on their history before assignment.
No
Does the platform offer a lift study?
Yes
Platform experiment
Measured inside one platform, by that platform.
No
Otherwise
Quasi-experimental or observational analysis
Useful, with weaker causal identification. The report says so explicitly.
What an experiment plan specifies
- Hypothesis
- The change being tested, the outcome it should move and the expected direction, written before launch.
Unit of randomization Users, accounts or regions: whatever can be assigned without one group influencing the other.
Treatment group
Receives the campaign or change being tested.
Control group
Does not receive it during the same period.
- Primary metric
- One business outcome defined in advance, preferably taken from the CRM.
- Minimum detectable effect
- The smallest effect worth acting on, set against the baseline rate and its variability.
- Power
- Chosen before launch, so that a null result is informative rather than inconclusive.
- Duration
- Long enough to cover the sales cycle and weekly patterns.
- Feasibility
- Checked against traffic, conversion volume, budget and operational constraints before anything runs.
- What the result can establish
- An estimate with an interval, valid for the population, period and design tested.
Decision use and limits
A suitable test can inform whether activity created additional outcomes. Feasibility depends on the decision, baseline, volume, variance, timing and ability to form a valid comparison. Results can remain inconclusive.
Deliverables
Experiment design
Hypothesis, metric, design and decision rule agreed before launch.
Feasibility analysis
Pre-test analysis of whether the available volume can detect a meaningful effect.
Holdout and geo tests
Designs matched to your channels and constraints, from audience holdouts to regional tests.
Monitoring and analysis
Checks during the test and an analysis that states uncertainty plainly.
Calibration
Using results to adjust how everyday measurement is read.
Process
Frame the decision
Which budget or channel decision should the result change?
Check feasibility
Estimate what the available data can detect before committing spend.
Run the test
Launch, monitor and document the experiment.
Analyse and calibrate
Report the effect with its range and update everyday measurement accordingly.
Tools and systems
- Geo experiments
- Holdout groups
- Platform lift studies
- A/B testing
- Python
- SQL
Questions
Can every business run experiments?
No. Feasibility depends on traffic, conversion volume, geography, channels and business constraints. The first step is an honest check of what can be detected; sometimes the answer is that a test would not be informative yet.
Will a randomized experiment always be possible?
Not always. When full randomization is operationally impossible, other designs such as geo tests or matched comparisons may be appropriate, with their limitations stated.
What is calibration?
Using experiment results to check and adjust everyday measurement, so the numbers you read every week reflect what the tests showed.