Guide 12 · Measurement
Incrementality: how to measure what a loyalty program adds
Incrementality is the extra spending or behavior a loyalty program causes, beyond what members would have done without it. You measure it by comparing members who get the program, or a change to it, with a similar group that does not: a holdout group. The gap between the two groups over the same period is the program's lift, and only that gap should count as its return.
Most loyalty reports show what members do: how much they spend, how often they visit, how many points they redeem. Few show what the program changed. Incrementality is the measure that does, and it is the number that decides whether rewards are paying for new behavior or for loyalty you already had. This guide explains how to measure it with a holdout test, what to do when you cannot run one, and how to keep the result honest.
Define incrementality
What is incrementality?
Incrementality is the difference between what members did and what they would have done without the program. That second number cannot be observed directly, so you estimate it with a comparison group that is like the members in every way except exposure to the program or to a specific change. The gap between the two groups is the lift, and the lift is what the program earns. The same idea applies at every scale: the whole program, a tier, a bonus offer or a reminder email. See incremental revenue for how the result feeds the money side.
Why member spend misleads
Why is member spend not proof the program works?
Because your best customers join first. Starbucks said Rewards members drove nearly 60% of its US company-operated revenue in fiscal 2025, and McDonald’s CEO said loyalty members drive about 30% of its sales and eat at its restaurants about 2.5 times more often. Figures like these show how much spending runs through a program. They cannot show how much of it the program caused, because frequent customers are the most likely to sign up.
The same problem affects other popular measures. Tier attainment, redemption rate and revenue per member describe activity. A member who reaches a tier may have spent the same amount without it. Unless a measure compares members with a counterfactual, it describes the program’s audience rather than its effect.
Run a holdout
How does a holdout test work?
- Pick one thing to test. The whole program, or a specific change such as double points, a new earn rate or a welcome offer.
- Choose one primary metric in advance. Spend per member, visits, or retention over the test period.
- Split eligible customers at random. The treated group gets the program or change; the holdout group does not. Random assignment, not region or tier, is what makes the groups comparable.
- Keep the holdout clean. No program emails, bonus points or member prices for the holdout during the test.
- Run the test for the full period you planned, then compare the two groups on the same dates.
- Subtract the cost. Lift minus the cost of the rewards and the running cost is the incremental contribution.
Size and length
How big should the holdout be and how long should the test run?
A holdout of 5 to 10% of eligible members is a reasonable starting point, with a few hundred members as a practical minimum so random noise does not swamp the result. The smaller the effect you expect, the larger the groups or the longer the test needs to be.
Length matters as much as size. Run at least one full purchase cycle: a monthly cycle for frequent purchases, a full season or longer for occasional ones. Short tests are distorted by paydays, holidays and one-off promotions, and early results tend to look better than final ones.
Write the plan down before the test starts: the metric, the group sizes, the end date and the result that would lead you to roll the change out or drop it. Share it with the finance team. A plan agreed in advance stops anyone from extending a disappointing test until it looks better, or ending a good one early to claim the win.
Calculate the lift
How do you calculate incremental lift?
Compare the groups per member, then scale up. In a hypothetical program that enrolls 9,000 members and holds out 1,000, members spend $520 each over six months and holdout customers spend $460.
| Line | Members | Holdout | Difference |
|---|---|---|---|
| Spend per member, six months | $520 | $460 | $60 (13% lift) |
| Incremental spend, 9,000 members | $540,000 | ||
| Margin on incremental spend at 40% | $216,000 | ||
| Rewards and running cost | $190,000 | ||
| Incremental contribution | $26,000 |
Total member spend in this example is $4.68 million, but only $540,000 of it is incremental, and after margin and cost the program earns $26,000. A report built on total member spend would overstate the program’s effect many times over. Feed the incremental figure into the ROI calculation, not the total.
When you cannot hold out
What if you cannot hold out members?
Sometimes a holdout is not practical: the program is already open to everyone, the member base is small, or the business will not withhold benefits. There are weaker but useful alternatives:
- Test changes, not the program. Hold out one offer or one rule change from a random group. This is the most common form of incrementality testing.
- Staggered rollout. Launch by region or store in waves and compare areas that have the change with areas that do not yet.
- Matched comparison. Compare members with non-members who looked similar before joining, for example in visit frequency and spend, and follow both over time.
- Segment the result. McDonald’s describes its members in three groups by how often they visit. A change often moves frequent, occasional and lapsed members differently, and an average hides it.
Each of these is more open to bias than a random holdout, so state the method next to the number.
Avoid contamination
Which mistakes contaminate the result?
- Choosing the holdout from a different group, such as another region or tier, so any gap reflects the groups rather than the program.
- Leaking the treatment. A partner offer, an app notification or a statement reaches holdout members and narrows the gap.
- Stopping early because the first weeks look strong.
- Counting a new member’s first purchase as incremental when many sign up at a checkout they were already making.
- Changing two things at once, so neither effect can be separated.
- Treating zero lift as a failure instead of a finding.
Incrementality is harder to produce than member revenue, but it is the only measure that tells you where reward spending changes behavior. Start with one test on one offer, report the method with the result, and build from there. For the first numbers to track before you test, see which loyalty program metrics to track first.
More answers
Frequently asked questions
What is incrementality in loyalty marketing?
It is the part of member behavior the program causes. If members spend $500 a year and similar customers without the program spend $460, the program's incremental effect is $40 per member, not $500.
What is a holdout group?
A random set of eligible customers or members who are deliberately kept out of the program or out of a specific change. Because they are chosen at random, they show what would have happened without it.
How large should a holdout group be?
Large enough to detect the effect you care about. Holding out 5 to 10% of eligible members is a reasonable starting point, with a few hundred members as a practical minimum. Smaller programs need longer tests to separate a real effect from noise.
How long should an incrementality test run?
At least one full purchase cycle, and longer if purchases are infrequent or seasonal. For many retail and restaurant programs that means several months. Stopping early because the first weeks look good is a common mistake.
Is it unfair to hold some customers out of a program?
It can feel that way, so most programs test changes (a bonus, an offer, a new rule) rather than the whole program, and keep holdouts small and temporary. Holding out one offer from a random 5% of members is usually invisible to them.
What if a test shows no lift?
That is a result, not a failed test. It means the change did not alter behavior enough to measure, so the money behind it can go somewhere else.