Learn · Module detail
running your first holdout test
Keep a comparable group from receiving the loyalty treatment, measure the same outcome for both groups, and use the difference to estimate incremental behaviour rather than rewarding activity that would have happened anyway.
## The mechanism
Define the qualifying event, value change, owner, correction path, and member explanation. Keep eligibility, liability, fulfilment, consent, and reversal as separate rules. The operating control must be testable from the ledger.
## Worked example
Illustrative arithmetic: 40 treatment members make 3 purchases each, or 120 purchases. 40 holdout members make 2 purchases each, or 80 purchases.
## Do it
State the rule in one sentence. Map the event that starts it, the event that changes value, the exception path, and the review action. Test ordinary, cancelled, duplicated, and disputed cases before release.
## Common mistakes
- Measuring activity instead of the target behaviour. Name the counterfactual. - Treating face value as operator cost. Price fulfilment and correction. - Hiding exclusions or expiry. Put the boundary beside the decision. - Automating an undefined rule. Resolve ownership before implementation.