The Importance of Net Lift
How shifting a marketing program's measure from response to net lift revealed that most of its budget was barely working — and cut spend by well over half with no loss in results.
Summary
A large retail bank ran a high-performing direct-marketing program measured by net response rate. That metric, while standard, could not distinguish customer responses caused by a campaign from responses that would have happened anyway. By using an existing random holdout group as a control, the program shifted its primary measure from response to net lift — the true causal effect of each campaign. Re-measured this way, the bulk of spend in the most expensive channel was shown to be generating little incremental lift, while specific lower-volume activity was generating most of it. Reallocating against net lift reduced the budget for this activity by well over half with no loss in measured effect. The program was, and remains, a genuine centre of excellence; this work added to its strength rather than correcting a failure.
Context
The program ran customer-acquisition and cross-sell campaigns at scale across five channels, each with a very different cost-per-contact:
- Direct mail — physical mail. By orders of magnitude the most expensive channel.
- Email — negligible cost per contact.
- SMS — very low cost per contact.
- In-app messaging — effectively zero marginal cost.
- Outbound telemarketing (OBTM) — agent-driven calls; moderate-to-high cost per contact, between the digital channels and mail.
The majority of the program's budget was allocated to direct mail. This was a defensible position, not an oversight. Direct mail remains a genuinely effective channel — arguably more so today than a decade ago — precisely because it cuts through the saturation of digital messaging that consumers are now subjected to. A physical, well-produced piece of mail stands out in a way an email no longer can. The strategic catch is cost: because direct mail is so expensive per contact, the margin for waste is small, and it has to be aimed with precision to justify its place. Getting direct mail right — sending it only where it earns its cost — matters more than for any other channel, simply because the downside of sending it wrong is so much larger.
The program's targeting and modelling were sophisticated, developed over many years by a highly capable team. It consistently helped the business punch above its weight and continues to operate as a centre of excellence today.
The Problem
The program measured channel performance by net response rate: of the customers contacted in a given channel, what proportion took the desired action.
This is the industry-standard metric and a reasonable one. It has a structural limitation, however, that is invisible within the metric itself.
Because direct mail was the most expensive channel, it was correctly reserved for the customers judged most likely to respond — the highest-value, most engaged segments. The population receiving mail was therefore pre-selected for responsiveness before any mail was sent.
As a result, direct mail reported the highest net response rate of any channel, and appeared in reporting to be the most effective use of budget. Every figure was accurate. The interpretation, however, conflated two different things: the quality of the targeting (which was excellent) and the effect of the channel (which the metric could not isolate). A channel aimed at customers who would have acted regardless will report a high response rate while contributing little incremental effect. This is a selection-bias effect, and standard response reporting cannot detect it, because the bias is determined by group composition before measurement begins.
The Method
The relevant question is not how many contacted customers responded, but how many additional customers responded because they were contacted, relative to comparable customers who were not.
The first quantity is response. The second is net lift — the incremental, causal effect of the campaign. Measuring net lift requires a counterfactual: a comparable group of customers who were not contacted, whose behaviour indicates what the contacted group would have done absent the campaign.
That counterfactual already existed in the program's data. Standard practice was to withhold a random share of each campaign's target population — on the order of 10% — and contact no one in it, as a control for model development and validation. Because this holdout was selected at random from the same target population, the contacted and held-out groups were statistically equivalent in every respect except contact. This is, structurally, a randomised controlled trial: a treatment group, a randomly assigned control group, and a measurable difference in outcome between them.
The method change was to read both groups rather than one — comparing the response rate of contacted customers against the response rate of the held-out control, campaign by campaign, and treating the difference (net lift) as the measure of effect.
Findings
Measured by net lift rather than response, the allocation of effect across channels differed substantially from what response reporting had indicated:
- Direct mail, broad volume: Across the majority of the mailed population, net lift was low. Contacted customers and equivalent held-out customers responded at similar rates, indicating that most of the response was not caused by the mail.
- Direct mail, top segment: Meaningful net lift from mail was concentrated in a narrow band of the highest-propensity customers — approximately the top 15–20% of those targeted. Within this band, mail was genuinely effective and justified its cost.
- Outbound telemarketing: A disproportionate share of total net lift came from lower-volume OBTM, despite its smaller budget.
- Digital channels (email, SMS, in-app): Low cost per contact meant that even modest lift was efficient on a cost-per-incremental-response basis.
The pattern was consistent: the largest portion of spend (broad direct mail) was producing a small portion of the causal effect, while smaller-budget activity (targeted mail and OBTM) was producing the majority of it.
Action & Result
Reallocation was carried out incrementally and tested at each stage, specifically to avoid removing the portion of direct-mail volume that was generating real lift. Over a period of roughly seven to eight months, mail volume was reduced in steps, with net lift re-measured against the holdout after each reduction to confirm that incremental effect was being preserved rather than eroded.
The budget for this activity was reduced by well over half, while net lift was held flat. The same measured effect on customer behaviour was achieved at a fraction of the previous spend, with the difference falling to the bottom line. No new technology was acquired and no headcount was added; the change came from re-measuring existing campaigns against a control already present in the data, and acting on the result.
Why This Generalises
The specifics are banking, but the measurement problem is general. Any business that evaluates an activity — marketing, promotions, discounts, retention offers, outbound sales — by measuring only the people who received it is exposed to the same effect: the activity is aimed at those most likely to act, those people act, and the activity is credited with behaviour that may have occurred regardless.
The correction is not more data or new software. It is a change in the measured quantity — from response to net lift — and the discipline to hold out a randomised control so that the counterfactual is real. Most businesses already hold the raw material for this in their existing data. The difficulty, and the reason it is frequently done wrong, lies in constructing the control correctly and not drawing causal conclusions from a biased comparison. It is straightforward to produce a number that looks like lift but is not, which is why the result is valuable when the method is sound.