ESRA 2027 sessions by theme

Back to Overview of Sessions

Session Details

Pretesting remains a cornerstone of questionnaire design, yet established methods—cognitive interviewing, expert reviews, behavior coding—are resource-intensive and difficult to scale across the growing number of languages, modes, and survey programs that require quality assurance before fielding. Recent advances in large language models (LLMs) have opened new possibilities for automating or augmenting parts of the pretesting process: detecting ambiguous wording, flagging comprehension problems, simulating respondent behavior, generating probes for cognitive interviews, or predicting items likely to produce measurement error.

This session invites contributions that explore how AI and LLMs are being used—or could be used—to assess and improve questionnaire quality prior to fielding. We welcome papers addressing, among others:

- Applications of LLMs to identify problematic question wording, double-barreled items, leading questions, or comprehension difficulties
- LLM-based simulation of respondent cognitive processes (e.g., emulating the four-stage response process) as a substitute or complement to think-aloud interviews
- Automated generation or analysis of cognitive interview probes and verbal protocols
- Comparative studies benchmarking AI-based pretesting against traditional methods (expert review, cognitive interviewing, behavior coding) in terms of validity, error detection, cost, and scalability
- Methodological and epistemological questions: what kinds of question problems LLMs can and cannot detect, biases introduced by AI-based tools, and how findings from AI pretesting relate to actual respondent behavior in the field
- Practical workflows integrating AI tools into existing pretesting pipelines, including validation strategies and quality control