Experimentation & incrementality
Move from attribution toward evidence.
Holdouts, geo experiments and platform tests designed around your traffic, volume and constraints, to find out what marketing actually caused.
Problems it solves
Did marketing cause the outcome, or merely receive credit for it?
Attribution distributes credit for conversions that may have happened anyway. The only direct way to see what spend changed is to compare against a group that did not receive it, and the right design depends on your traffic, conversion volume, geography, channels and constraints.
Credit is not causation
Platform-reported ROAS includes customers who would have bought anyway.
Tests without enough power
Experiments launched without checking whether they can detect the effect that matters.
Results nobody trusts
Tests analysed after the fact, with changing definitions, rarely settle a decision.
Evidence that is never reused
Experiment results that never feed back into everyday reporting.
Deliverables
Experiment design
Hypothesis, metric, design and decision rule agreed before launch.
Feasibility analysis
Pre-test analysis of whether the available volume can detect a meaningful effect.
Holdout and geo tests
Designs matched to your channels and constraints, from audience holdouts to regional tests.
Monitoring and analysis
Checks during the test and an analysis that states uncertainty plainly.
Calibration
Using results to adjust how everyday measurement is read.
Process
Frame the decision
Which budget or channel decision should the result change?
Check feasibility
Estimate what the available data can detect before committing spend.
Run the test
Launch, monitor and document the experiment.
Analyse and calibrate
Report the effect with its range and update everyday measurement accordingly.
Tools and systems
- Geo experiments
- Holdout groups
- Platform lift studies
- A/B testing
- Python
- SQL
Questions
Can every business run experiments?
No. Feasibility depends on traffic, conversion volume, geography, channels and business constraints. The first step is an honest check of what can be detected; sometimes the answer is that a test would not be informative yet.
Will a randomized experiment always be possible?
Not always. When full randomization is operationally impossible, other designs such as geo tests or matched comparisons may be appropriate, with their limitations stated.
What is calibration?
Using experiment results to check and adjust everyday measurement, so the numbers you read every week reflect what the tests showed.