Loyalty glossary · 7. Data and analytics (31)
Incrementality testing
Incrementality testing measures the outcome caused by an intervention by comparing an exposed group with a comparable control or holdout group.
Incrementality testing asks whether an offer changed behaviour that would otherwise have happened. The test compares members who receive an intervention with a control group or holdout group that remains eligible for measurement but does not receive it.
The control group supplies the counterfactual. Without that comparison, a rise in purchase, redemption or activity may simply reflect members who were already going to act. The test needs a defined outcome, a clear assignment rule and a period in which the groups can be compared.
A holdout group is a deliberate part of the design, not a collection of members who happen to miss a message. Random assignment protects the comparison from the programme team's selection choices. The groups should be comparable on the behaviour and eligibility conditions that matter to the test.
Uplift modeling extends the question beyond an average result. It estimates which members are more likely to change because of the intervention, which helps a programme avoid spending an incentive on members who would have acted without it. The model still needs test results to anchor its predictions.
Churn rate can be an outcome when the intervention is intended to retain members. Zero-party data and first-party data may support segmentation and measurement, but data ownership does not prove causation. An upgrade award can be tested in the same way as a message or incentive, and distributor loyalty can supply a partner context for the intervention.
The result should be read as an incremental effect, with the treatment, control or holdout definition preserved alongside it. A campaign total without that comparison is an observed outcome, not incrementality.
If 120 of 1,000 treated members purchase and 90 of 1,000 holdout members purchase, the treated rate is 12 percent and the holdout rate is 9 percent. The difference is 3 percentage points, or 30 additional purchases per 1,000 members, which is the incremental result in this illustration.