Comparing exposed and unexposed markets carefully

To estimate whether a CTV change affected sales or enquiries across markets, choose treatment and comparison areas before launch and measure the same business outcome in both. Under the test design, “unexposed” means the planned CTV treatment was withheld or reduced. It does not mean no resident saw any advertising.

Select markets the test can use

Choose boundaries that the media buyer can target and the business can use to assign outcomes. A postcode cluster is unsuitable if delivery cannot be controlled or sales cannot be assigned to it consistently. States may be easy to name but can leave too few independent areas for a useful comparison.

Inspect a pretest series of the same outcome for each area. Compare scale, trends and volatility, then check stock, service coverage, pricing and planned local promotions. Similar sales on one date do not establish comparable trends.

Where feasible, group broadly comparable markets and randomise treatment assignment within those groups. If markets are selected without randomisation, record the selection reasons and the assumptions needed for the comparison to be credible.

Estimate whether the available markets and outcome volume can detect a change large enough to affect the decision. A design that cannot separate a useful effect from ordinary variation may still produce a number, but that number will not settle the spending question.

Planning a Valid CTV Market Test

Protect the treatment contrast

Write down what differs between groups: the CTV inventory, dates, spend or delivery rule. For a new campaign, treatment markets might receive CTV while controls continue their existing media.

For a spend increase, both groups might continue existing activity while treatment markets receive the additional CTV spend. These designs answer different questions.

Check delivered impressions by market during the flight. Material delivery in controls or little delivery in treatment areas weakens the planned contrast. Keep other controllable activity as steady as possible and record changes such as a local promotion or supply interruption. Travel, purchases outside a home area and imperfect location signals can also blur the comparison.

Do not replace a control market midway because its result looks inconvenient. If a serious operational change occurs, document it and assess whether the original analysis remains credible.

Treatment vs Control Markets: Key Differences in CTV Testing

Interpret the tested change

Analyse the outcome using the plan agreed before results were inspected. Compare treatment outcomes with the counterfactual estimated from the controls and pretest data, then report the effect estimate, uncertainty range, markets, dates and exclusions. A positive estimate with a wide range may be inconclusive.

A raw post-campaign sales gap is not CTV lift: the areas may have started at different levels or faced different conditions. Even a credible market test estimates the effect of the tested change across its included areas. It does not identify which household responded, and its result may not carry unchanged to another inventory mix, offer or period.

Key Metrics for Evaluating CTV Test Validity