Theory Roadnewbusiness@theoryroad.com(866) 3-THEORYGet in touch

Incrementality Testing: Methods, Tradeoffs and How to Run a First Test.

Attribution tells you which ads were near a sale. Incrementality tells you which sales would not have happened without the ads. Here is how the main test designs work, where each one fits, and how to run your first one without fooling yourself.

By November 18, 202612 min read

Want this done for you?See what Theory Road runsget a free plan

Incrementality testing is the practice of measuring the causal effect of marketing by comparing a group that was exposed to an ad or campaign with a comparable control group that was not. The difference in outcomes between the two groups is the incremental lift: the sales, sign-ups or visits that happened because of the marketing and would not have happened without it. The IAB and MRC Retail Media Measurement Guidelines describe incrementality as the potential causal impact of marketing, measured as value above a baseline and isolated from other business influences.

This reference is for marketing, analytics and finance leaders who need to know whether paid media is producing new revenue or taking credit for sales that would have happened anyway. It covers the main test designs, how they compare, how incrementality relates to marketing mix modeling and multi-touch attribution, what drives test size and length, and how to run a first test.

Why attribution is not incrementality.

Attribution assigns credit for a conversion to the ads a person saw or clicked before converting. It answers which touchpoints were present. It does not answer whether the conversion needed them. A loyal customer who searches your brand name and clicks a paid ad would often have bought anyway. A retargeting ad shown to someone with a full cart may get credit for a purchase already in motion.

Platform-reported results inherit this problem, because each platform counts conversions that follow its own ads. Incrementality testing fixes it by building a counterfactual: a group that represents what would have happened with no ads, so the difference can be measured directly.

Incrementality testing methods.

Every method works by comparing a treated group to a control group. They differ in how the groups are formed and what can go wrong.

  • Randomized holdout (user-level). People or households are randomly assigned to a test group that can see the ads and a control group that cannot. The IAB and MRC guidelines say MRC standards highly recommend randomized controlled trials because random assignment helps make the groups statistically identical except for the treatment.
  • Geo lift or matched markets. Regions are split into test and control. Ads run, stop or change in test regions while control regions continue as normal, and sales are compared over time. The IAB and MRC guidelines recommend difference-in-differences and time-based regression approaches for matched-market tests, and note that matching alone cannot control for local price cuts, displays or coupons.
  • Platform conversion lift. Studies run inside an ad platform using its own holdout. Google's Conversion Lift splits audiences into a treatment group that sees ads and a control group that does not. Meta's Conversion Lift randomly assigns people to a test group that sees ads and a control group that does not.
  • Ghost ads and PSA tests. Ways of identifying who in the control group would have been shown your ad. In a PSA test, the control group is shown an unrelated public service ad in the same slot. In a ghost ad design, the platform logs when it would have served your ad to a control user without serving it. The IAB and MRC guidelines name both as options for handling users who only had an opportunity to be exposed.
  • Synthetic control. When a holdout is not possible, a control group is built from unexposed people or markets that closely match the exposed group on past behavior. The IAB and MRC guidelines recommend matching on purchase and digital behavior, using methods such as propensity score matching, and validating with A/A tests before the real test.

Comparing incrementality methods.

Incrementality testing methods compared
MethodHow groups are formedStrengthsLimitationsBest for
Randomized user holdoutRandom assignment of people or householdsStrongest causal evidence when randomization holdsNeeds identity and a platform or partner that can hold people out; walled gardens limit cross-channel useSingle-channel tests on platforms that support it
Geo lift / matched marketsRegions split into test and controlWorks across channels, online and offline sales, no user tracking neededFewer units means noisier results; local events and promotions can distort marketsChannel-level questions, offline sales, TV and CTV, cross-channel budgets
Google Conversion LiftUser-based or geography-based holdout inside Google AdsBuilt into the platform; geo version supports offline dataNot available to all accounts; access through a Google representativeMeasuring Google campaigns, including Search, YouTube and Performance Max
Meta Conversion LiftRandomized test and control groups inside MetaRandomized design run by the platformMeasures Meta only; results reported by the platform being measuredMeasuring Meta campaigns and checking attribution against lift
Ghost ads / PSAControl users identified as would-have-been-exposedCompares like with like, reducing dilution from unexposed usersPSA tests pay for placeholder impressions; ghost ads require platform supportPrecise exposure-level lift where supported
Synthetic controlMatched unexposed group built from historical dataUsable when no holdout is allowedOnly as good as the matching variables; hidden differences bias resultsRetail media and situations where holding out is not possible

Google's comparison of lift types sets out the platform tradeoff clearly. User-based studies report results by conversion action, age, gender and country, and are available in Google Ads for Video, Discovery and Demand Gen campaigns, with other campaign types through an account representative. Geography-based studies cover Search, Shopping, Performance Max, Display, App and video, support online and offline conversions, and tend to need a higher budget, with noisier results that are not sliced by demographics.

The control group mistake that breaks most tests.

Most bad incrementality results come from a control group that is not comparable. The IAB and MRC guidelines warn against confusing the treatment group with the intention-to-treat group. If two million shoppers are split evenly, the one million in the test group only had the opportunity to see the ad. Not all of them will be exposed. Comparing exposed people to the whole control group mixes the ad's effect with the differences between people who get reached and people who do not.

The guidelines' fix is to make sure the same targeting and bidding logic that decides who gets the ad also decides who goes into the control group. Ghost ad and PSA designs, or intent-to-treat analysis, are the standard ways to do that. When you review a vendor's lift study, ask exactly how the control group was formed.

How incrementality relates to MMM and MTA.

Incrementality tests, marketing mix modeling (MMM) and multi-touch attribution (MTA) answer different questions and work best together.

  • Multi-touch attribution splits credit for each conversion across the touchpoints a person was tracked through. It is fast and granular, but it is correlational and blind to channels that cannot be tracked at the person level.
  • Marketing mix modeling uses weekly or daily sales and spend history to estimate each channel's contribution, including offline media and price. It covers the whole budget but needs years of data and can confuse channels that move together. Our guide to marketing mix modeling explains how the models work.
  • Incrementality tests give the most direct causal answer for one channel or campaign at one point in time, and are used to check and calibrate the other two.

Both major open-source MMM tools are built to take experiment results as inputs. Google's Meridian offers GeoX for running geo experiments and says it can convert experiment results into priors for the model. Meta's Robyn says it calibrates models against ground-truth methods, including geo-based tests and Facebook lift studies. The practical loop is test, calibrate the model, reallocate budget, then test again.

Sample size and test length.

There is no universal minimum budget or duration. What a test needs depends on a few factors, and a power analysis before launch tells you whether the test can detect the effect you care about.

  • Baseline conversion volume. The fewer conversions per day, the longer the test or the larger the groups needed to separate signal from noise.
  • Expected lift. Small effects need much more data to detect than large ones. If the plausible lift is small, the test has to be bigger.
  • Unit of randomization. User-level tests have many units. Geo tests have few, which is why Google says geo studies tend to need more budget and are noisier. Meta's GeoLift documentation notes that people-based experiments have higher statistical power than geo-based ones and recommends them where possible.
  • Purchase cycle. The test and its measurement window should cover enough of your consideration cycle for the effect to show. The IAB and MRC guidelines say pre-campaign hold-out periods and test duration should be disclosed.
  • Seasonality and other activity. Avoid launching during promotions, price changes or major seasonal swings that hit test and control differently.

How to run a first incrementality test.

Pick one decision.
Choose a budget question you will act on, such as whether branded search, retargeting or a new channel is producing new sales. Write down what result would make you cut, hold or grow spend.
Choose the outcome and source of truth.
Use business data you trust: orders and revenue from your commerce platform, qualified leads from your CRM, or retail sales. Avoid judging the test with the same platform's attributed conversions.
Pick the design.
Use a platform conversion lift study if the question is about one platform and you have access. Use a geo test if you need offline sales, cross-channel effects or independence from the platform being measured.
Run a power check.
Estimate whether the planned groups, budget and duration can detect a lift large enough to change your decision. If not, extend the test, enlarge the groups or test a bigger change.
Freeze everything else.
Hold creative, pricing, promotions and other channel spend steady in test and control. Log any change that cannot be avoided.
Read incremental cost, not just lift.
Report incremental conversions, incremental revenue, cost per incremental conversion and incremental ROAS. Compare them with the platform-reported figures to see how far attribution was off.
Act and repeat.
Move budget based on the result, feed it into your mix model if you have one, and schedule the next test. Effects change as spend, creative and competition change.

Affiliate and partner programs raise the same question in a different form, since coupon and loyalty sites often claim sales that were already happening. Our guide to affiliate marketing incrementality covers how to test those programs.

What enterprise buyers should require.

If an agency or measurement vendor runs your tests, put these in writing:

  • Procurement fit covering vendor onboarding, insurance and payment terms, and an MSA plus SOW that names each test, its hypothesis, design, outcome metric and decision rule.
  • Data access and ownership. Raw test data, group assignments and results stay with you, and every ad and analytics account is in your company's name.
  • Method disclosure. How groups were formed, the power analysis, duration, hold-out period and limitations, which the IAB and MRC guidelines say should be disclosed.
  • Security and privacy review of any customer data uploaded for matching or clean room analysis.
  • Reporting cadence that includes a pre-registered plan, a mid-test health check and a final readout.
  • A named senior analyst on the work, and exit terms that hand over code, models and test history.

Where Theory Road fits.

Theory Road is an operator-led growth firm based in Liberty Hill, in the Austin area, working with companies across the US. We build and run our own portfolio of brands with our own money, so we test whether spend is incremental before we scale it. Senior people do the work. Engagements run on an MSA with a statement of work, and the client owns every account and data set. If you want incrementality testing built into how your paid media is run and budgeted, tell us what you need.

We are not a fit if you need a standalone econometrics vendor to build a large custom mix model without running media, or a certified third-party measurement provider for retailer or publisher audits.

What is incrementality testing in marketing?

It is measuring the causal effect of marketing by comparing outcomes for a group exposed to ads with a comparable control group that was not. The difference is the incremental lift, the outcomes the marketing actually caused.

What is the difference between incrementality and attribution?

Attribution assigns credit to the ads a person saw before converting. Incrementality measures whether the conversion would have happened without the ads. Attribution can credit ads for sales that would have happened anyway.

What is a geo lift test?

A geo lift test splits regions into test and control, changes advertising in the test regions, and compares sales over time. It works across channels and with offline sales, but has fewer units than a user-level test, so results are noisier.

How long should an incrementality test run?

Long enough to detect the lift you care about given your conversion volume and purchase cycle. A power analysis before launch sets the duration. Google notes geo studies tend to need more budget than user-based ones.

What are ghost ads?

A ghost ad design records when an ad would have been shown to someone in the control group without showing it, so exposed users can be compared with control users who would have been exposed. PSA tests do something similar by showing the control group an unrelated ad.

How does incrementality testing relate to marketing mix modeling?

Tests give a direct causal read on one channel at one time. Mix models estimate every channel's contribution from historical data. Tests are used to calibrate mix models, and both Google's Meridian and Meta's Robyn accept experiment results as inputs.

Get in touch

Let’s talk.

Tell us what you sell and what you want more of. Email newbusiness@theoryroad.com or use the form, and a senior strategist will be in touch.

Tell us about your business.

A senior strategist reads your inquiry personally.

Rather email? newbusiness@theoryroad.com or call (866) 3-THEORY.

Thanks. We have it.

A senior strategist will be in touch by email or phone.