Research methods · Synthetic data · Product decisions
When can synthetic research support a product decision?
Choose a bounded use, compare generated responses with relevant human evidence, and check the differences that could change the decision before expanding the method.
Define the claim the research needs to support
A simulated customer response can suggest a question to investigate. Estimating what a population wants asks much more of the method. Start by writing the decision, the people it concerns and the evidence that would change the choice. This makes it possible to judge whether simulation has a useful role.
For an exploratory use, try generating alternative explanations or exercising a questionnaire's branches. Have researchers examine those outputs before they shape fieldwork. Describe them as generated material so a plausible response cannot quietly become a customer finding.
Aher and colleagues' 2023 experiments reproduced some established human-study patterns while revealing distortions in another task. That mixed result supports checking a specific application. It does not establish that a simulated audience represents customers in a new market.
Compare the features that matter to the decision
Select human observations that fit the intended audience, question and period. Assess their quality too: a flawed reference cannot provide a reliable test. Decide which comparisons matter before seeing the generated results, including response variation, relationships between answers and relevant subgroup differences.
In their 2024 public-opinion study, Bisbee and colleagues found that similar average responses could coexist with different variation and statistical relationships. A convincing headline average therefore leaves important validation work unfinished. Check whether the method would lead to the same practical choice, especially for a smaller group whose needs could change the product.
Keep evaluation observations separate from material used to tune the simulation where feasible, and disclose any overlap or unknown training exposure. More generated responses do not add independently observed people to the study.
Make the boundary visible and revisit it
Keep a short method record: the source and dates of human inputs, intended population, model configuration, prompts, comparisons and unresolved weaknesses. Show readers where the evidence is observed and where it is simulated. A useful idea from a simulation can earn a human test without earning a population estimate.
The 2026 AAPOR task-force report recommends application-specific evaluation and continued checks as models, integrations and populations change. Recheck a method when the underlying audience or context changes, even if the software version stays the same. Updating inputs addresses their age; representativeness and validity still need examination.
Set a stopping rule tied to the decision. If a material subgroup or relationship cannot be validated, narrow the claim, collect further human evidence or leave the question unresolved.