ESRA 2027 sessions by theme

Back to Overview of Sessions

Session Details

Large language models (LLMs) are now a prominent, multi-purpose tool in social and behavioral science. One increasingly popular method is "LLM-as-a-Judge" (LLMJ), wherein LLMs classify and evaluate stimuli. The potential of LLMJ cannot be overstated; tasks that once took teams of humans weeks to complete can now be completed in minutes with the same accuracy and nuance achieved by experts. Given such impressive performance, it is no surprise that LLMJ has been widely adopted across fields.

This enthusiasm, however, has outpaced careful, systematic investigation of best practices. Studies that employ LLMJ tend to do so in highly specific ways, and they examine validity for those specific purposes without considering generalizability. Perhaps as a result, although LLMJ ratings tend to correspond to human ratings, estimates vary greatly both between and within studies. In addition, variation in how prompts are written, such as the use of binary vs. continuous scoring rubrics, or even spacing within the prompt, can meaningfully impact the estimates and conclusions drawn from LLMJ. These observations highlight the need for programmatic methodological research, especially for audiences who are less familiar with the computational architecture of these models, and thus unlikely to anticipate these biases or pinpoint their causes.

This session aims to showcase the optimal ways to use LLMJ across different contexts. Broadly, topics of interest may include:
- Biases of LLMJ (e.g., position bias, length bias) and their impact across different contexts
- Effects of structural aspects of prompts (e.g., spacing, number of survey items) on LLMJ ratings
- Effects of linguistic differences (e.g., positive vs. negative framing, promotion vs. prevention language) on LLMJ ratings
- Reliability of results across models, model specifications (e.g., temperature), or time
- Variation in LLMJ ratings across data sources (e.g., news articles, social media posts)