
Incrementality
Part of CTV creative experimentation
Deciding when a CTV creative test needs more evidence: signs
Diagnose whether a CTV creative test needs more responses, more comparable delivery or a better measure before choosing a version.
undefined
A CTV creative test needs more evidence when its confidence interval still crosses the pre-agreed threshold for changing the edit or buy, or when its outcome or comparison is incomplete. Set the minimum difference worth detecting, the confidence or significance target, and statistical power before delivery; use these alongside the baseline outcome and its variability to plan sample size (Statsig; Kameleoon). The signs below distinguish uncertainty that more comparable outcomes may reduce from design or measurement gaps that more impressions cannot fix.
Set the decision threshold in the test brief
What changed between versions? Which outcome was primary, and what difference would justify switching? Set that smallest difference as the minimum detectable effect (MDE), the smallest difference you want to be able to consistently detect (Statsig). A completion-rate lead cannot settle a message-recall question, and scans alone cannot settle whether a QR treatment increased completed enquiries.
Use the pre-agreed action threshold, based on the MDE and chosen confidence level, rather than the largest point estimate (Statsig; Kameleoon). Decision-ready means the confidence interval sits wholly on one side of that threshold, so the same choice follows across plausible values; directional means the estimate points one way but the interval crosses the threshold. Unresolved means the primary outcome or a credible comparison is missing; if the interval crosses the threshold, collect more outcomes only when the design and measure are sound.
These are practical decision labels, not statistical grades. A result does not become decision-ready just because one version leads; compare its confidence interval with the pre-agreed action threshold.
Signs a CTV creative test needs more evidence
Count usable survey responses or business events in each arm, then compare the estimated difference and its confidence interval with the MDE and pre-set confidence or significance target. If the interval crosses the action threshold, use a sample-size calculation based on the baseline outcome, variability, MDE, target and power to estimate the additional evidence needed, rather than choosing an arbitrary increment (Statsig; Kameleoon).
Compare dates, inventory, audience, device mix, spend and exposure frequency. Check whether creative optimisation favoured one version. Address the imbalance identified here before deciding whether more delivery is useful.
Show which inventory and outcomes entered the analysis, and whether each reported slice has enough responses to interpret. Limit conclusions to the inventory and outcomes actually measured. IAB Australia notes that CTV is still traded like a 1:1 digital impression and discusses co-viewing in relation to measurement; state what the test counts as an exposure before interpreting audience outcomes (IAB Australia).
A precise difference may still be too small to justify changing an edit or buy. Compare the confidence interval with the MDE: if it sits entirely below the threshold, the version has not shown the worthwhile gain set in the brief; if it spans the threshold, the choice remains uncertain (Statsig; Kameleoon). A wide interval may include both a worthwhile gain and a harmful loss. Report the interval beside the estimate.
Decision readiness in CTV creative tests: What to look for
- Decision-readyConfidence interval lies entirely on one side of the action threshold; choice is consistent across plausible values.
- DirectionalPoint estimate suggests a direction, but confidence interval crosses the action threshold; uncertainty remains.
- UnresolvedPrimary outcome missing or comparison not credible; more evidence may be needed only if design and measurement are sound.
Key metrics for evaluating CTV creative test validity
- Minimum Detectable Effect (MDE)
- Smallest difference worth detecting; set in test brief before delivery.
- Baseline Variability
- Measured variation in primary outcome; informs sample size planning.
Choose the next evidence worth collecting
If the design is sound but the outcome sample is thin, compare usable responses with the planned sample size and extend comparable collection only if more data is still needed. If assignment or inventory differs, address the imbalance first. If the outcome was not recorded or the survey asked the wrong question, fix measurement. If several creative elements changed, isolate the decision still worth testing.
Set the next analysis point and decision rule before looking for a favourable result. Do not stop early because a current estimate crosses the threshold; wait for the planned analysis point. If the campaign must proceed, choose a reversible version and state the limited evidence behind it. Close the review with what was learned, what remains unknown and the specific evidence that could change the choice.
Before deciding to collect more evidence in a CTV test
- Is the test design sound?Confirm random assignment, comparable inventory, audience, device mix, and spend.
- Are outcomes measured correctly?Ensure survey questions align with primary objectives; verify data capture.
- Is there enough usable data?Compare actual responses to planned sample size; use sample-size calculation to estimate need.
- Has the creative been isolated?If multiple elements changed, identify which one still needs testing.
- Is the analysis point planned?Do not stop early—wait for pre-defined analysis timing unless reversible action is taken.



