The hidden problem behind campaigns that look profitable

How to measure incremental impact

To correctly evaluate the impact of a campaign, you need to split users into two groups at random:

• Test group: randomly assigned to see the ads
• Control group: randomly assigned to be held out from the ads

Random assignment is what makes the two groups comparable, including on the things you cannot measure. Where users cannot be split one by one, a designed geo experiment (whole regions held out) is the usual alternative.

The performance difference between these two groups shows your incremental lift.

But for this difference to matter, it must be statistically significant.

1. Incrementality testing

This method relies on causal inference. It allows you to verify the direct effect of advertising by building a controlled experiment that isolates whether ad exposure actually changes user behavior.

2. Statistical significance

A difference between the test and control group can sometimes happen by chance.

Ad effects are small next to the normal swings in sales, so a test needs a large sample before a real effect stands out from the noise (Lewis & Rao, 2015).

To check whether a gap is more than chance, we use a significance test (for conversion rates, a two-proportion test), which compares the two groups and asks: if the ads had no effect at all, how often would chance alone produce a difference this large?

That answer is the p-value. Below 0.05 means a gap this large would turn up less than 5% of the time if the ads did nothing, so the result is called statistically significant. It does not mean there is a 95% chance the effect is real, and it says nothing about how big the effect is. That is what the next step is for.

3. Confidence interval

Every estimate carries uncertainty. A confidence interval is the range of effect sizes consistent with the data. If you ran the same test many times, 95% of the 95% intervals built this way would contain the true effect.

For example:

• a lift of 18% with a range of –5% to +40% is too uncertain
• a range of +12% to +24% sits entirely above zero, so the data point to a real, positive effect of roughly that size

4. Multiple linear regression

In the real world, not everything can be controlled. Seasonality, discounts, parallel channels, or even the day of the week can affect results.

Multiple linear regression (OLS) helps separate the campaign's contribution from seasonality and other measured factors. It cannot account for what it does not measure, so it is no substitute for an experiment. When researchers compared observational methods against large randomised tests run at Facebook, the observational estimates often missed the experimental lift by a wide margin (Gordon et al., 2019).

Why ROAS alone isn't enough

Advertising platforms attribute conversions according to their own algorithms.

A customer who would have purchased anyway, but happened to see an ad somewhere in the journey, is often counted as an "ad-driven conversion."

eBay tested this on its own brand keyword ads. When it switched them off, almost all of that traffic arrived through the free organic listings instead, so the paid clicks had added very little (Blake et al., 2015).

This leads to misleading conclusions: campaigns that look profitable but do not actually create new value.

Without an incremental framework, these attribution distortions can lead to poor budget and strategic decisions.

And when causality is missing, companies often:

• cut truly effective campaigns
• keep spending on campaigns that only appear profitable

The result? Money moved on appearances, with no evidence behind the move.

Why incrementality matters for your business

Incrementality is the foundation of a marketing strategy built on real results.

It lets you estimate which part of your sales is caused by advertising and which would have occurred naturally.

This distinction is essential for:

• optimizing budgets
• making strategic decisions
• achieving sustainable growth

Only by understanding incremental impact can you accurately evaluate the return on your advertising investment.

In essence, incrementality moves budget decisions from intuition towards evidence about cause.

How to apply incrementality in your campaigns

You don't have to be a statistician to apply these principles. You only need to adopt an evidence-based mindset.

The goal is simple: identify which campaigns truly drive growth and eliminate the spend that brings no value.

In short

Incrementality starts as a math formula and ends as a shift in perspective.

It forces you to move:

• from assumptions to evidence
• from superficial reporting to causal understanding

In our industry, everyone talks about performance, but the real difference is made by those who can prove, with data, that their advertising creates real value.

Because what you cannot prove, you cannot optimize.

And what you cannot optimize, you cannot scale.

References

• Blake, T., Nosko, C., & Tadelis, S. (2015). Consumer heterogeneity and paid search effectiveness: A large-scale field experiment. Econometrica, 83(1), 155–174.
• Gordon, B. R., Zettelmeyer, F., Bhargava, N., & Chapsky, D. (2019). A comparison of approaches to advertising measurement: Evidence from big field experiments at Facebook. Marketing Science, 38(2), 193–225.
• Lewis, R. A., & Rao, J. M. (2015). The unfavorable economics of measuring the returns to advertising. Quarterly Journal of Economics, 130(4), 1941–1973.