# What should a product experiment establish before the team acts on it?

By Sean Everett · October 7, 2026

Match the experimental design to the decision, define the comparison in advance and preserve the difference between a useful result and an explanation of why it happened.

Topics: Product experiments · Decision quality · Measurement

## Write the decision before choosing the variants

Specify what the team will decide after the experiment: release a change, reject it, narrow its audience or investigate further. Name the intended outcome and the conditions that must remain acceptable. Define who is eligible and how long the relevant behavior needs to be observed.

Kohavi and colleagues’ practical guide to web experiments emphasizes a predetermined evaluation criterion and explains how sample size and variability affect the ability to detect differences. Have an analyst connect the intended decision to an appropriate design, observation plan and treatment of uncertainty.

Choose the unit of assignment deliberately. A shared account or collaborative workflow may expose people to one another’s treatment. Record which experience each unit should receive and test that assignment and measurement work before interpreting differences as a product effect.

## Make the conclusion match what actually changed

Consider a hypothetical onboarding test. Version A uses the current headline and a long form; version B changes the headline and shortens the form. A well-run comparison can estimate the effect of the combined experience for the tested audience. It cannot identify which individual change produced the difference.

If the next decision depends on separating those contributions, choose a design that varies them appropriately. NIST’s experimental-design guidance explains that confounding can prevent separate estimates of effects and interactions. More variants require enough evidence for the comparisons being attempted; adding combinations does not create that evidence.

Keep the baseline credible and the implementation differences inspectable. Include technical failures, missing events and unequal exposure in the review. If the intended groups did not receive the intended experiences, resolve that problem before turning the observed outcome into a recommendation.

## Decide with uncertainty and operating consequences visible

Agree in advance how the team will handle an encouraging result, an unacceptable side effect and an inconclusive comparison. Research by Johari and colleagues addresses why continuously monitoring ordinary tests and choosing when to stop can invalidate their inference. Have an analyst establish a suitable stopping and analysis plan.

Inspect the result for the audience and observation period actually tested. Separate planned subgroup comparisons from patterns discovered afterward. A promising pattern can motivate a new experiment; it should not silently acquire the same status as the original planned comparison.

Record the design, resulting uncertainty, observed operating effects and decision owner. If the change proceeds, retain a way to detect unexpected consequences during rollout. Feed unresolved questions into the next test so the experiment portfolio accumulates usable learning instead of a collection of winning labels.

## Related reading

- [Kohavi and colleagues: Practical guide to controlled web experiments](https://ai.stanford.edu/~ronnyk/GuideControlledExperiments.pdf)
- [NIST: Confounding in experimental designs](https://www.itl.nist.gov/div898/handbook/pri/section3/pri3343.htm)
- [Johari and colleagues: Sequential analysis for A/B tests](https://arxiv.org/abs/1512.04922)
- [Connecting operating measures to decisions](https://evergences.com/insights/operating-metrics-decisions/)

Source: https://evergences.com/insights/experiment-decision-governance/