ESRA 2027 sessions by theme

Back to Overview of Sessions

Session Details

Respondents have been shown to produce shorter and less rich open-ended answers in text-based online surveys than in interviewer-administered surveys. For example, the average number of words across several open-ended questions in the American National Election Study (ANES) was less than half when respondents answered in a web survey than in person, and for some questions the respondents produced fewer codeable ideas per answer (Antoun & Presser, 2024). Shorter open-ended answers with fewer ideas are also harder for researchers to code. The risk is that web surveys, as currently deployed, miss out on the potential to collect rich and high quality data in open ended answers.

This session focuses on whether and how AI tools and LLM-assisted “interviewing” in online surveys may assist respondents and potentially improve not only the length but also the quality of responses to open-ended questions. The session brings together new evidence on how LLMs can be deployed to help online survey respondents to provide richer and higher quality answers to open-ended questions. This could include work in which, for example, an LLM probes an initial answer in real time to elicit further information, an LLM initiates dialogue to clarify what the respondent’s open-ended answer meant, or other new LLM interventions.

The session is intended to address the following research questions:
How and when should LLMs probe open ended answers?
What is the impact of different kinds of probing on data quality?
Are LLMs following survey best practices in terms of probing?
Are there effects on breakoffs when an LLM probes open-ended answers versus not, in experimental settings?
What tools are available to survey researchers and how well are they documented?
What are the ethical issues when probing open-ended answers in real time?