ESRA 2027 sessions by theme

Back to Overview of Sessions

Session Details

Pretesting is crucial for ensuring high-quality survey data. To assess the validity of responses, we need to know how respondents understand the questions and whether that understanding aligns with the researcher’s intended meaning. Willis’ (2005, 2015) original approach of cognitive interviewing has merged with other survey methodological and qualitative research approaches into a range of open forms of pretesting, all of which let respondents articulate their understanding in their own terms.

These approaches are often arranged on a single axis from efficient to rich, but their differences are better described in terms of what each makes visible and what each requires. Self-administered probing reaches large, dispersed samples, removes interviewer effects, and allows quantitative as well as qualitative analysis, but probes must be anticipated, leaving little room to pursue the unexpected. Interviewer-administered forms - from classical cognitive interviewing with standardised and emergent probes, through more focused qualitative methodologies such as problem-centred interviews, to fully interpretive designs - retain that flexibility and can reconstruct how an answer came about, at a higher cost in training, fieldwork, and analysis. The choice of method depends on the test objectives, the stage of instrument development, and practical constraints; the most interesting methodological questions often arise where these have to be weighed against one another.

This session brings together researchers developing and applying approaches from across this range. We are particularly interested in the added complexity of different cultures and languages across countries; in the use of pretest data for substantive research beyond their original purpose; in pretesting techniques for emerging data collection modes, such as smartphone surveys or conversational agents; and in how computational and AI-assisted tools can support open pretesting, not least where analytical demands are highest. Practical experiences, including approaches that did not work as intended, are especially welcome.