ESRA 2027 sessions by theme
Back to Overview of Sessions
Evaluating Synthetic Survey Responses: Validity, Robustness, and Downstream Consequences |
| Coordinator 1 | Mr Georg Ahnert (University of Mannheim) |
| Coordinator 2 | Dr Anna-Carolina Haensch (LMU Munich; MCML; University of Maryland, College Park) |
| Coordinator 3 | Dr Oliver Wieczorek (INCHER, University of Kassel) |
| Coordinator 4 | Professor Betting Langfeldt (University of Kassel) |
Session Details
Generative AI is increasingly used to produce synthetic survey responses, most visibly through persona-prompted large language models ("silicon sampling"). Commercial offerings are growing, yet quality standards are far from established. The generation pipeline involves a wide range of design choices (model, persona construction, prompt wording, decoding parameters), opening a large space of forking paths. Downstream consequences are equally uncertain: training on synthetic data is known to induce model collapse in machine learning, but little is known about how this plays out in survey research.
This session focuses on evaluation: how to establish whether fully synthetic or hybrid datasets are fit for a given purpose, such as pre-testing, imputation, or substantive analysis. It aims to reconcile quality standards across AI research and survey methodology. Topics include:
Bias and downstream consequences
- Biases occurring across different measurement levels (e.g., individual items, dimensions, or composite scales) in synthetic responses
- Effects of increasing proportions of synthetic data on the reliability and validity of classical statistical analyses or the training of machine- and deep learning algorithms
- Model collapse and accumulating error in hybrid survey panels
Evaluation approaches
- Established evaluation procedures from survey methodology (e.g. for imputation) applied to synthetic responses
- Lessons from the evaluation and adjustment of non-probability samples for synthetic responses
- Prediction-powered inference (PPI) and related approaches for combining human and synthetic responses while retaining valid statistical inference
- Validating synthetic responses when comprehensive ground-truth data are unavailable
- Multiverse analyses of design choices in synthetic survey responses (e.g., persona construction, response generation)
Standards and reporting
- Locating synthetic responses within a Total Survey Error framework
- Detecting and quantifying training-data contamination in AI models used to generate synthetic survey responses
- Proposals for process-based quality and reporting standards.
