Incrementality
Part of CTV creative experimentation
Deciding when a CTV creative test needs more evidence
Diagnose whether a CTV creative test needs more responses, more comparable delivery or a better measure before choosing a version.
A CTV creative test needs more evidence when its result cannot support the choice it was designed to make. Identify the gap before extending delivery: too little outcome data, unequal exposure, partial measurement or a difference too small to change the decision. More impressions will not repair the wrong measure or a confounded comparison.
Return to the test brief
What changed between versions? Which outcome was primary, and what difference would justify switching? A completion-rate lead cannot settle a message-recall question. Scans alone cannot settle whether a QR treatment increased completed enquiries.
| Working state | What the evidence supports | Next step |
|---|---|---|
| Decision-ready | The chosen outcome and uncertainty support the pre-agreed choice within comparable delivery | Use the result within its tested scope |
| Directional | A pattern exists, but imbalance or uncertainty could change the choice | Make only a limited, reversible choice or improve the test |
| Unresolved | The primary outcome or credible comparison is missing | Repair the design before naming a winner |
These are practical decision labels, not statistical grades.
Diagnose the gap
Too little outcome data. Count usable survey responses or business events in each arm and examine uncertainty against the difference that would matter. A larger usable sample can help distinguish smaller differences, though the amount needed depends on the study and outcome.
Unequal delivery. Compare dates, inventory, audience, device mix, spend and exposure frequency. Check whether creative optimisation favoured one version. Address the imbalance identified here before deciding whether more delivery is useful.
Partial measurement. Show which inventory and outcomes entered the analysis, and whether each reported slice has enough responses to interpret. Limit conclusions to the inventory and outcomes actually measured.
Little decision value. A precise difference may still be too small to justify changing an edit or buy. A wide uncertainty range may include both a worthwhile gain and a harmful loss. Report that range beside the estimate.
Choose the next evidence worth collecting
If the design is sound but the outcome sample is thin, more comparable collection may help within the study's rules. If assignment or inventory differs, address the imbalance identified above first. If the outcome was not recorded or the survey asked the wrong question, fix measurement. If several creative elements changed, isolate the decision still worth testing.
Set the next analysis point and decision rule before looking for a favourable result. If the campaign must proceed, choose a reversible version and state the limited evidence behind it. Close the review with what was learned, what remains unknown and the specific evidence that could change the choice.
