ESRA 2027 sessions by theme
Back to Overview of Sessions
Beyond AI-Generated Questions: Predicting and Validating Measurement Quality |
| Coordinator 1 | Dr Lydia Repke (GESIS - Leibniz Institute for the Social Sciences) |
| Coordinator 2 | Dr Marek Muszyński (Institute of Philosophy and Sociology of the Polish Academy of Sciences) |
| Coordinator 3 | Dr Piotr Koc (GESIS - Leibniz Institute for the Social Sciences) |
| Coordinator 4 | Mr Leon Martinez (GESIS - Leibniz Institute for the Social Sciences) |
Session Details
The rapid development of large language models (LLMs), machine learning (ML), and synthetic data is transforming the way survey questions and measurement instruments can be developed. AI can now generate, translate, adapt, and evaluate large numbers of survey items at minimal cost. Synthetic respondents and silicon sampling are increasingly explored as tools for examining questionnaire designs and anticipated response patterns before costly data collection. However, a fundamental methodological question remains: How can we determine whether AI-generated or AI-evaluated instruments actually provide valid and reliable measurement?
This session addresses the emerging intersection of survey measurement, psychometrics, ML, LLMs, and synthetic data. It focuses on methods for predicting, evaluating, and validating the quality of measurement instruments, including their reliability, validity, dimensionality, measurement error, and comparability across populations, countries, and time.
We invite contributions addressing questions such as: Can ML and LLMs predict measurement quality before data collection? How can AI-assisted questionnaire development be benchmarked against empirical evidence? To what extent can synthetic respondents reproduce latent structures, measurement error, and heterogeneity observed in human survey data? Which aspects of instrument development and evaluation can be already automated, and where does empirical validation remain indispensable?
The session welcomes methodological, empirical, and conceptual contributions, including research on predictive models of measurement quality, AI-assisted item development and evaluation, synthetic or silicon sampling, benchmarking and auditing AI tools, reproducibility and transparency of LLM-based approaches, and the integration of established measurement evidence with new AI-based methods.
By bringing together researchers across survey methodology, psychometrics, statistics, and computational social science, the session aims to identify evidence requirements, promising validation designs, and research priorities for developing methodological standards for ensuring that technological innovation advances not only the efficiency of survey research, but also the scientific quality of its measurements.
