How to Test Whether Something Works (Without a Data Team)
You don't need a statistician to know if your marketing works. The holdout method, explained in plain terms, with a simple worked example any small business can run.
The only honest way to know if your marketing works is to compare what happened with what would have happened if you'd done nothing. That sounds like something requiring a statistician and a big budget. It isn't. A small business can run a genuinely valid test of almost any marketing activity, cheaply, with no data team. Here's how.
The core idea: hold a group back
The whole method rests on one move: don't do the thing to everyone.
Before you run a campaign — an email, an offer, a mailing — set aside a portion of the people you would have sent it to, at random, and deliberately don't contact them. Run your campaign on everyone else. Then compare what the two groups did.
The group you contacted is your test group. The group you held back is your comparison group — your "what would have happened anyway." Because the only difference between them is that one got the campaign and one didn't, any difference in their behaviour is the real effect of the campaign. That difference is your lift.
If the contacted group bought at 12% and the held-back group bought at 10% anyway, your campaign's real effect is the 2-point difference — not the full 12%. (Why that distinction matters so much is covered in would it have happened anyway?)
Why the "hold back" group has to be random
This is the one part you can't skip. The group you hold back must be chosen randomly from the same pool as the group you contact.
If you hold back a group that's different in some way — your least engaged customers, say, or a different region — then any difference you see might be because the groups were different to begin with, not because of the campaign. Randomness is what makes the comparison fair. It ensures both groups have the same mix of keen and lukewarm customers, so the only thing separating them is your marketing.
Most spreadsheet tools and email platforms can pick a random subset for you. That's all the sophistication you need.
A simple worked example
Say you have 1,000 customers and you want to test whether a discount email actually drives sales.
- Randomly set aside 200 of them. These get no email.
- Send the discount email to the other 800.
- After a couple of weeks, look at what fraction of each group bought.
- If the 800 bought at a higher rate than the 200, the difference is your lift. If they bought at the same rate, your email did nothing — and you just learned something worth knowing.
That's it. No special software, no analyst. Just the discipline to hold a group back and the patience to compare.
What this protects you from
Running even rough tests like this guards against the most expensive mistake in marketing: pouring money into something that feels productive but isn't actually causing results. A campaign can generate plenty of response and zero lift. Without a comparison group, you'd never know — you'd just see the response and keep paying for it.
A few practical notes:
- Keep the held-back group big enough to be meaningful. A handful of people won't tell you much; a couple of hundred starts to.
- Give it time. Look at results after customers have had a realistic window to act, not the next morning.
- Don't over-think the maths. For most small business decisions, "the contacted group clearly did better" or "there's no real difference" is all you need. Precision matters more as the stakes rise.
This is exactly the method that, applied rigorously at large scale, can reveal that most of a marketing budget is barely working — a real example is laid out in the importance of net lift. The principle is identical whether you're testing on a thousand customers or a million.
If you'd like help setting up honest tests of what's actually working in your business — and reading the results properly — that's something I do with Canadian small businesses. You can get in touch here.
