Loyalty glossary · 7. Data and analytics (31)
Control Group
A control group is a subset of members deliberately excluded from a test or change, used as the baseline against which the behaviour of the treated group is measured. It isolates the incremental effect of a loyalty programme action from unrelated shifts in member behaviour.
The point of a control group is to separate what the programme did from what would have happened anyway. Without it, a rise in redemptions after a new earn rate could be the programme working, or it could be a holiday season, a competitor's devaluation, or a change in accrual rules that affected everyone. The control group answers the question by showing the counterfactual.
Most loyalty programmes do not run proper control groups. They compare the test period to the prior period and call the difference the programme effect. That method is worthless because member behaviour drifts with calendar events, economic conditions, and marketing campaigns. A before-and-after comparison cannot tell you whether the lift came from the change or from the month you chose to run it.
A legitimate control group is built before the test starts, not after. Members are split randomly or matched on historical behaviour, and the control group receives the old terms for the full test window. Changes to activity-based qualification are particularly prone to false results without this, because the rule change itself selects which members become active, and the control group reveals whether that selection is real or just noise.
Take a simple test: a programme offers 2 points per dollar to a test group and 1 point per dollar to a control group over 6 months. At the end, the test group has earned an average of 6,000 points per member and the control group has earned 4,000 points. The difference is 2,000 points, which is a 50 percent lift. Without the control group, a naive comparison to the prior period might attribute all 6,000 points to the new rate, but the control group shows that half of that would have happened anyway.
Even when a control group exists, it can be ruined by contamination. If test and control members are in the same tier, share the same customer service line, or see the same targeted offers, the treatment leaks across groups and the measured effect shrinks. The actuarial models that later use this data then inherit the bias, which is why a clean holdout matters more than a large one.
A programme that cannot produce a control group has not tested anything. It has run an anecdote. Any result reported without a counterfactual is not evidence of what the change does, it is evidence of what happened while the change was in place. That distinction is the difference between a loyalty programme that learns and one that just spends money guessing.