ESRA 2027 sessions by theme

Back to Overview of Sessions

Session Details

Modern AI systems depend extensively on data generated by people. Humans label training datasets, provide preference data for reinforcement learning, evaluate model outputs, red-team systems, assess fairness and safety, and generate the behavioural and conversational data used to study deployed models. Yet these human inputs are often treated as neutral observations rather than measurements produced through specific recruitment, task-design and data-collection processes.

Survey methodology offers a mature framework for understanding representation, sampling, measurement and inference in human-data collection, as well as how biases introduced through recruitment, task design or response processes can propagate into downstream model training, evaluation, fairness and performance.

This session invites research on how survey methodology can contribute to better AI development, evaluation and deployment. We welcome empirical, methodological and conceptual work applying survey methods and frameworks across the AI lifecycle. Topics may include:

• Sampling and recruitment of annotators, raters, evaluators and users, including representativeness, expertise, oversampling, self-selection and weighting.
• Measurement and task design for human feedback, including construct validity, instructions, wording, response scales, order effects, satisficing, burden and interface effects.
• Sampling of prompts, images, conversations, model outputs and other stimuli used in training or evaluation.
• Modelling disagreement and heterogeneity in human judgments, including rater effects, latent-variable models, uncertainty and pluralistic preferences.
• Assessing how errors in human-generated data propagate into downstream model accuracy, fairness, safety and alignment.
• Evaluating real-world AI use, including selective feedback, data donation, post-deployment monitoring and the representativeness of behavioural data.
• Standards for transparency, documentation and auditing of human-data collection in AI pipelines.

The aim is to bring survey researchers and AI researchers together around a shared problem: when AI systems depend on human data, who provides those data, how they are measured, what they measure, and what inferences they can credibly support.