Loyalty glossary · 7. Data and analytics (31)
Holdout Group
A holdout group is a segment of loyalty programme members who are deliberately excluded from a promotion, campaign, or product change, so their activity can be compared against members who received it.
A loyalty programme cannot prove that any offer changed member behaviour without a holdout group. The holdout receives nothing while the treated receives the offer, and the difference in their subsequent activity is the only true measure of impact. Without that comparison, every reported win is indistinguishable from a coincidence.
Operators often reject holdouts because they imagine the foregone points as lost revenue. This is a budgeting mistake, not a measurement one. The points not given to the holdout are the price of knowing whether the points given to everyone else did anything at all. A programme that refuses to pay that price is paying a larger one in ignorance.
Work the arithmetic because the argument only lands with numbers. A programme sends a bonus points offer to 80 percent of active members and withholds it from 20 percent. After one month, treated members accrue an average of 120 points each, while the holdout accrues 90 points. The incremental lift is 30 points per member. A naive comparison to a zero baseline would credit 120 points, overstating the effect by 300 percent.
The accrual data from a holdout is cleaner than the data from a treated group. Treated member earning includes both the base rate and the campaign lift, so any accrual model fitted to that series will overstate future liability. The holdout provides the base rate alone, which is what an actuarial model needs to forecast redemption cost without the campaign's temporary inflation.
The holdout logic is not limited to promotions. When a programme tests a higher tier threshold for activity-based qualification, it must hold out a group on the old threshold to see whether members change behaviour to keep the new status. Without that group, a retention rate of 90 percent after one year could mean the threshold worked or it could mean the programme simply kept its most loyal members and lost everyone else.
A holdout group is a cost only if the programme measures success by how much it spends. The real cost is running campaigns year after year with no idea which ones worked. The programmes that refuse holdouts are not avoiding risk; they are avoiding the evidence that would force them to stop spending on things that do nothing.