ESRA 2027 sessions by theme

Back to Overview of Sessions

Session Details

Among the attempts to use large language models (LLMs) for survey research, open-text analysis has attracted relatively more attention (von der Heyde et al., 2026). While recent studies have evaluated LLM effectiveness (e.g., Mellon et al., 2024; Schonlau et al., 2023; Sparkman et al., 2026; von der Heyde et al., 2025), the stochastic nature of LLMs and the lack of predictive theory necessitate further evidence regarding their behaviors and boundary conditions. This session organizes empirical studies testing LLM efficacy in analyzing open-ended survey responses, focusing on the following areas:

First, we consider open-text analysis as a workflow involving multiple tasks——data pre-processing, code frame creation, code assignment, example extraction, and performance evaluation and calibration——and welcome studies examining LLM roles across all of them. Second, we seek work evaluating how human expertise and LLM capabilities work in concert to enhance output quality and workflow efficiency.

Third, we encourage investigations into the effects of (1) topic areas, (2) task characteristics (e.g., single- vs. multi-label coding, binary vs. multi-point scale), (3) data pre-processing (e.g., pre-tiering data by quality), (4) prompt designs, (5) language, (6) LLM self-reflection/evaluation, (7) collaborative or adversarial LLMs, (8) model mix (e.g., LLMs with different capabilities and compute demands), and (9) quality control mechanisms.

Finally, we invite submissions discussing the utility of LLMs, balancing performance gains against costs and, where applicable, relative to other computational methods.

Submissions should demonstrate clarity in LLM implementation (e.g., models, hyperparameters, compute demand). Where applicable, consider distinguishing between consistency and accuracy analysis when evaluating coding performance and utilizing multiple metrics (e.g., code prevalence, F-scores, precision, recall, and Krippendorff’s Alpha) rather than solely relying on percent-agreement. We encourage designs that include a temporal component that addresses