Podcast Series on Survey Research

Series with Prof. Dr. Barry Schouten

Episode 5: Surveys and AI-ML

In this short film, Barry Schouten discusses connection between surveys and AI-ML models with Stephanie Eckman, Christoph Kern and Bolei Ma. The following discussion points are covered in this short movie:

– What is Social Data Science

– Differences and similarities between Social Science and Social Data Science

– Similarity between annotators labeling cases for Data Science datasets and survey respondents or interviewers

– Sources for biases of data annotators

– How to account for biases while working with models in Data Science

– Annotation sensitivity

– Conceptual shifts in Data Science

– Robustness of models in Data Science

– Future of representation error research in annotation sensitivity research

– Potential of using ML models in cross-cultural research

– Two cases of using ML models in surveys (Supermarket receipts, Hate-speech)

– Potential of using AI for data annotation

References:

Kern, C., Eckman, S., Beck, J., Chew, R., Ma, B., & Kreuter, F. (2023). Annotation sensitivity: Training data collection methods affect model performance. arXiv preprint arXiv:2311.14212.

Beck, J., Eckman, S., Ma, B., Chew, R., & Kreuter, F. (2024, March). Order effects in annotation tasks: Further evidence of annotation sensitivity. In Proceedings of the 1st Workshop on Uncertainty-Aware NLP (UncertaiNLP 2024) (pp. 81-86).

Eckman, S., Ma, B., Kern, C., Chew, R., Plank, B., & Kreuter, F. (2025). Correcting Annotator Bias in Training Data: Population-Aligned Instance Replication (PAIR). arXiv preprint arXiv:2501.06826.

Affiliations:

Dr. Stephanie Eckman – Data Scientist at Social Data Science Center, University of Maryland

Prof. Dr. Christoph Kern – Junior Professor of Social Data Science and Statistical Learning, Chair of Statistics and Data Science in Social Sciences and the Humanities (SODA) at the Ludwig-Maximilians-University of Munich; Project Director at the Mannheim Centre for European Social Research (MZES)

Bolei Ma – Researcher and PhD student at Social Data Science and AI Lab at the Ludwig-Maximilians-University of Munich

Prof. Dr. Barry Schouten – Senior Methodologist at Centraal Bureau voor de Statistiek (CBS), the Netherlands.

Episode 4: Survey Quality Prediction

In this episode, Barry Schouten discusses Survey Quality Predictor (SQP) with Lydia Repke and Daniel Oberski. SQP is a survey quality prediction system for continuous questions used in survey research. The following discussion points are covered in this episode:

– What is SQP and how can it be used?
– History of the SQP.
– How was the list of item characteristics identified for the SQP?
– How is SQP coding organised?
– Case study of using the SQP.
– How important is item positioning on the item quality prediction?
– Language importance in predicting item quality.
– What is the future of SQP?
– Potential of predicting quality for new types of data.
– Concept drifting and potential of using SQP for tracking or tackling that.

References:

Scherpenzeel, A. C., & Saris, W. E. (1997). The validity and reliability of survey questions: A meta-analysis of MTMM studies. Sociological Methods & Research, 25(3), 341-383.

Felderer, B., Repke, L., Weber, W., Schweisthal, J., & Bothmann, L. (2024). Predicting the Validity and Reliability of Survey Questions (No. hkngd_v1). Center for Open Science.

Affiliations:

Dr. Lydia Repke is the manager of the Survey Quality Predictor 3.0 (SQP), acting head of the Scale Development and Documentation team at GESIS, Germany and PI of the project LoneCovid.

Prof. Dr. Daniel Oberski is the scientific director of ODISSEI (Open Data Infrastructure for Social Science and Economic Innovations) and holds the chair in Social and Health Data Science at the department of methodology and statistics at Utrecht University, the Netherlands.

Prof. Dr. Barry Schouten – Senior Methodologist at Centraal Bureau voor de Statistiek (CBS), the Netherlands.

Contact details of the SQP team and Dr. Lydia Repke for collaborations:
sqp@gesis.org
lydia.repke@gesis.org
SQP website

Episode 3: Survey monitoring

Survey monitoring

In this episode, Barry Schouten discusses survey monitoring and the use of dashboards with Brady West and James Wagner. Paul Burton also presents how such dashboards and indicators are used and interpreted in practice. Survey monitoring is becoming more and more important, especially considering rising costs, new data collection modes, and responsive and adaptive survey designs. The following topics are covered:

History and experience of using dashboards for survey monitoring
Evolution of dashboards
Dashboards content in the responsive design framework
The process of interventions based on the dashboards
Dashboards in the context of different modes of data collection and mix-mode
Transition from micro- to macro-level of survey monitoring
Data protection with regard to dashboards data
Tailoring interviewers’ reactions to respondents’ concerns regarding survey participation
Recent changes in the survey environment
Future of dashboards

References:

Groves, R. M., & Heeringa, S. G. (2006). Responsive design for household surveys: tools for actively controlling survey errors and costs. Journal of the Royal Statistical Society Series A: Statistics in Society, 169(3), 439-457.

Kreuter, F. (2013), Improving Surveys with Paradata. Analytic Uses of Process Information. Edited Book. Wiley Series in Survey Methodlogy

Wagner, J., West, B.T., Kim, B., Suolang, D., Engstrom, C., Sinibaldi, J. (2024), Using a Stopping Rule to Optimize Cost-Quality Tradeoffs in a Large, Mixed-Mode Survey: A Simulation Study, JOS, https://doi.org/10.1177/0282423X241287452

Schouten, B., Calinescu, M., Luiten, A. (2013), Optimizing quality of response through adaptive survey designs, Survey Methodology, 39 (1), 29 – 58.

Affiliations:

Prof. Dr. Brady T. West – Research Professor, Survey Research Center, Institute for Social Research, University of Michigan-Ann Arbor

Prof. Dr. James Wagner – Research Professor, Survey Research Center, Institute for Social Research, University of Michigan-Ann Arbor; Adjunct Lecturer in Quant Methods and Social Science Program, College of Literature, Science, and the Arts, University of Michigan-Ann Arbor

Paul Burton – Lead Statistician, Survey Research Center, Institute for Social Research, University of Michigan-Ann Arbor

Prof. Dr. Barry Schouten – Senior Methodologist at Centraal Bureau voor de Statistiek (CBS)

Episode 2: Web tracking

In this short film, Barry Schouten discusses web tracking research with Melanie Revilla, Oriol Bosch and Carlos Ochoa. The film also showcases Barry’s example of participating in the web tracking study and the experts’ real-time approach to analysing this kind of data produced during a short web surfing session. The following topics concerning web tracking research are discussed during the conversation:

How web tracking research is done
Technological challenges (webs and apps; PC vs. mobile devices; different OS issues)
Respondent’s role in web tracking study
Survey and web tracking data linkage
Concept drifting during web tracking panel research
Web tracking from participants’ and researchers’ side
Web tracking data cleaning
Data protection
Future of web tracking research

Affiliations:

Prof. Dr. Barry Schouten – Professor at the Department of Methodology & Statistics at Utrecht University; Senior Methodologist at Centraal Bureau voor de Statistiek (CBS)

Dr. Melanie Revilla – survey methodologist researcher at Institut Barcelona d’Estudis Internacionals (IBEI), Universitat Pompeu Fabra; Principal Investigator of the WEB DATA OPP project

Dr. Oriol Bosch – Postdoctoral Researcher at Leverhulme centre for Demographic science, University of Oxford; Research Fellow at the Research and Expertise Centre for Survey Methodology (RECSM), Universitat Pompeu Fabra

Carlos Ochoa – Researcher at the Research and Expertise Centre for Survey Methodology (RECSM), Universitat Pompeu Fabra

Episode 1: Postcorona face-to-face interviewing

In this short film, Prof. Dr. Barry Schouten is conversing with two interviewers, Maria van Beers and Fred Stoffer, who have varying amounts of experience in fieldwork. The following topics through the lens of an interviewer are covered:

Engaging respondents in survey participation
Respondents’ motivations for survey participation
Tailoring contact strategy and invitation process
Being an interviewer in modern times
Respondents trust in surveys and interviewers
Adaptive interviewing techniques

Affiliations:

Maria van Beers – Interviewer at Centraal Bureau voor de Statistiek (CBS)

Fred Stoffer – Interviewer at Centraal Bureau voor de Statistiek (CBS)

Prof. Dr. Barry Schouten – Professor at the Department of Methodology & Statistics at Utrecht University; Senior Methodologist at Centraal Bureau voor de Statistiek (CBS)

Series with Dr. Tom Emery

Episode 4: The introduction of large language models (LLMs) and AI to social science research

Listen to the episode here.

In this episode, Tom Emery and Ethan Busby discuss the implications of the introduction of large language models (LLMs) and AI to social science research. Surveys on extremism, conflict and polarisation are hard to study using surveys due to high sensitivity and sample specificity, but can we use large language models (LLMs) to address these challenges? The discussion involves the following main points:

What can LLMs and AI, in general, tell us about society?
How easy was the collaboration of social and computational scientists?
What is algorithmic fidelity? Why do social scientists using LLMs and AI in research need this?
How can we work with such a rapidly growing tool?
What are the potential uses of LLMs and AI within social science research? And how can it be used within survey research?

Affiliations:

Dr. Ethan Busby, Assistant Professor, Department of Political Science, Brigham Young University, Provo, UT, USA

Dr. Tom Emery – Director of ODISSEI, the Dutch National Infrastructure for Social Science; Associate Professor, Department of Public Administration and Sociology of Erasmus University, Rotterdam

Useful links:

Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023). Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3), 337-351.

Episode 3: FAIR principles implementation in survey research with Steve McEachern

Listen to the episode here.

In this episode, Dr. Tom Emery discusses the WorldFAIR project and FAIR principles implementation with regard to survey research methods with Dr. Steve McEachern, focusing on several key points:

What are the FAIR principles, and why are they important?
What challenges arise when implementing FAIR principles in survey research?
What is machine actionability, and how does it relate to survey research?
How do FAIR principles apply to cross-cultural surveys?
How can we make data accessible for machines?
What potential advancements will the full implementation of FAIR principles bring to future research?

Affiliations:

Dr. Steve McEachern – Director and Manager of ADA (Australian Data Archive)

Dr. Tom Emery – Director of ODISSEI, the Dutch National Infrastructure for Social Science; Associate Professor, Department of Public Administration and Sociology of Erasmus University, Rotterdam

Useful links:

The WorldFAIR Project

Episode 2: Crafting a Comparative Longitudinal Study on Child Well-being: Methodological Insights of GUIDE project

Listen to the episode here.

In this podcast episode of our newly introduced series, Dr. Tom Emery, Director of ODISSEI and Associate Professor at Erasmus University Rotterdam engages in a comprehensive exploration of an ambitious birth cohort study spanning across Europe. Joined by esteemed guest Professor Gary Pollock from Manchester Metropolitan University, the discussion focuses on examining the intricate process of setting up a longitudinal study aimed at the dynamics of child well-being in the evolving landscape of digital technologies.

The conversation delves into the methodological challenges embedded in setting up such a study, touching upon the formulation of research questions, overcoming methodological challenges, and pioneering innovative approaches in questionnaire development. Through this dialogue, listeners gain valuable insights into the complexities inherent in longitudinal research design and the nuanced considerations essential for navigating the contemporary terrain of child well-being research.

Affiliations:

Prof. Gary Pollock – Manchester Metropolitan University, Co-Director of Survey Infrastructure for Growing Up In Digital Europe (GUIDE)

Dr. Tom Emery – Director of ODISSEI, the Dutch National Infrastructure for Social Science; Associate Professor, Department of Public Administration and Sociology of Erasmus University, Rotterdam

Useful links:

Growing up in Digital Europe (GUIDE)

Cohort Community Research and Development Infrastructure Network for Access Throughout Europe (COORDINATE)

Episode 1: Sensors and Apps in Surveys with Bella Struminskaya

Listen to the episode here.

In the ESRA (European Survey Research Association) Podcast Series, dr. Tom Emery, the director of ODISSEI (Open Data Infrastructure for Social Science and Economic Innovations in the Netherlands) and associate professor at the Erasmus University Rotterdam, connects with the leading experts in survey research to get an understanding of the field development.

In the first episode, dr. Tom Emery is joined by Bella Struminskaya, Associate Professor at the Department of Methodology & Statistics at Utrecht University. Together they discuss the usage of apps and devices in survey research.