Podcast Series on Survey Research

Survey Quality Prediction

In this episode, Barry Schouten discusses Survey Quality Predictor (SQP) with Lydia Repke and Daniel Oberski. SQP is a survey quality prediction system for continuous questions used in survey research. The following discussion points are covered in this episode:

– What is SQP and how can it be used?
– History of the SQP.
– How was the list of item characteristics identified for the SQP?
– How is SQP coding organised?
– Case study of using the SQP.
– How important is item positioning on the item quality prediction?
– Language importance in predicting item quality.
– What is the future of SQP?
– Potential of predicting quality for new types of data.
– Concept drifting and potential of using SQP for tracking or tackling that.

References:

Scherpenzeel, A. C., & Saris, W. E. (1997). The validity and reliability of survey questions: A meta-analysis of MTMM studies. Sociological Methods & Research, 25(3), 341-383.

Felderer, B., Repke, L., Weber, W., Schweisthal, J., & Bothmann, L. (2024). Predicting the Validity and Reliability of Survey Questions (No. hkngd_v1). Center for Open Science.

Affiliations:

Dr. Lydia Repke is the manager of the Survey Quality Predictor 3.0 (SQP), acting head of the Scale Development and Documentation team at GESIS, Germany and PI of the project LoneCovid.

Prof. Dr. Daniel Oberski is the scientific director of ODISSEI (Open Data Infrastructure for Social Science and Economic Innovations) and holds the chair in Social and Health Data Science at the department of methodology and statistics at Utrecht University, the Netherlands.

Prof. Dr. Barry Schouten – Senior Methodologist at Centraal Bureau voor de Statistiek (CBS), the Netherlands.

Contact details of  the SQP team and Dr. Lydia Repke for collaborations:
sqp@gesis.org
lydia.repke@gesis.org
SQP website

Survey monitoring

In this episode, Barry Schouten discusses survey monitoring and the use of dashboards with Brady West and James Wagner. Paul Burton also presents how such dashboards and indicators are used and interpreted in practice. Survey monitoring is becoming more and more important, especially considering rising costs, new data collection modes, and responsive and adaptive survey designs. The following topics are covered:

• History and experience of using dashboards for survey monitoring
• Evolution of dashboards
• Dashboards content in the responsive design framework
• The process of interventions based on the dashboards
• Dashboards in the context of different modes of data collection and mix-mode
• Transition from micro- to macro-level of survey monitoring
• Data protection with regard to dashboards data
• Tailoring interviewers’ reactions to respondents’ concerns regarding survey participation
• Recent changes in the survey environment
• Future of dashboards

References:

Groves, R. M., & Heeringa, S. G. (2006). Responsive design for household surveys: tools for actively controlling survey errors and costs. Journal of the Royal Statistical Society Series A: Statistics in Society, 169(3), 439-457.

Kreuter, F. (2013), Improving Surveys with Paradata. Analytic Uses of Process Information. Edited Book. Wiley Series in Survey Methodlogy

Wagner, J., West, B.T., Kim, B., Suolang, D., Engstrom, C., Sinibaldi, J. (2024), Using a Stopping Rule to Optimize Cost-Quality Tradeoffs in a Large, Mixed-Mode Survey: A Simulation Study, JOS, https://doi.org/10.1177/0282423X241287452

Schouten, B., Calinescu, M., Luiten, A. (2013), Optimizing quality of response through adaptive survey designs, Survey Methodology, 39 (1), 29 – 58.

Affiliations:

Prof. Dr. Brady T. West – Research Professor, Survey Research Center, Institute for Social Research, University of Michigan-Ann Arbor

Prof. Dr. James Wagner – Research Professor, Survey Research Center, Institute for Social Research, University of Michigan-Ann Arbor; Adjunct Lecturer in Quant Methods and Social Science Program, College of Literature, Science, and the Arts, University of Michigan-Ann Arbor

Paul Burton – Lead Statistician, Survey Research Center, Institute for Social Research, University of Michigan-Ann Arbor

Prof. Dr. Barry Schouten – Senior Methodologist at Centraal Bureau voor de Statistiek (CBS)

Web tracking

In this short film, Barry Schouten discusses web tracking research with Melanie Revilla, Oriol Bosch and Carlos Ochoa. The film also showcases Barry’s example of participating in the web tracking study and the experts’ real-time approach to analysing this kind of data produced during a short web surfing session. The following topics concerning web tracking research are discussed during the conversation:

• How web tracking research is done
• Technological challenges (webs and apps; PC vs. mobile devices; different OS issues)
• Respondent’s role in web tracking study
• Survey and web tracking data linkage
• Concept drifting during web tracking panel research
• Web tracking from participants’ and researchers’ side
• Web tracking data cleaning
• Data protection
• Future of web tracking research

Affiliations:

Prof. Dr. Barry Schouten – Professor at the Department of Methodology & Statistics at Utrecht University; Senior Methodologist at Centraal Bureau voor de Statistiek (CBS)

Dr. Melanie Revilla – survey methodologist researcher at Institut Barcelona d’Estudis Internacionals (IBEI), Universitat Pompeu Fabra; Principal Investigator of the WEB DATA OPP project

Dr. Oriol Bosch – Postdoctoral Researcher at Leverhulme centre for Demographic science, University of Oxford; Research Fellow at the Research and Expertise Centre for Survey Methodology (RECSM), Universitat Pompeu Fabra

Carlos Ochoa – Researcher at the Research and Expertise Centre for Survey Methodology (RECSM), Universitat Pompeu Fabra

Postcorona face-to-face interviewing

In this short film, Prof. Dr. Barry Schouten is conversing with two interviewers, Maria van Beers and Fred Stoffer, who have varying amounts of experience in fieldwork. The following topics through the lens of an interviewer are covered:

• Engaging respondents in survey participation
• Respondents’ motivations for survey participation
• Tailoring contact strategy and invitation process
• Being an interviewer in modern times
• Respondents trust in surveys and interviewers
• Adaptive interviewing techniques

Affiliations:

Maria van Beers – Interviewer at Centraal Bureau voor de Statistiek (CBS)

Fred Stoffer – Interviewer at Centraal Bureau voor de Statistiek (CBS)

Prof. Dr. Barry Schouten – Professor at the Department of Methodology & Statistics at Utrecht University; Senior Methodologist at Centraal Bureau voor de Statistiek (CBS)

The introduction of large language models (LLMs) and AI to social science research

In this episode, Tom Emery and Ethan Busby discuss the implications of the introduction of large language models (LLMs) and AI to social science research. Surveys on extremism, conflict and polarisation are hard to study using surveys due to high sensitivity and sample specificity, but can we use large language models (LLMs) to address these challenges? The discussion involves the following main points:

• What can LLMs and AI, in general, tell us about society?
• How easy was the collaboration of social and computational scientists?
• What is algorithmic fidelity? Why do social scientists using LLMs and AI in research need this?
• How can we work with such a rapidly growing tool?
• What are the potential uses of LLMs and AI within social science research? And how can it be used within survey research?

Affiliations:

Dr. Ethan Busby, Assistant Professor, Department of Political Science, Brigham Young University, Provo, UT, USA

Dr. Tom Emery – Director of ODISSEI, the Dutch National Infrastructure for Social Science; Associate Professor, Department of Public Administration and Sociology of Erasmus University, Rotterdam

Useful links:

Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023). Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3), 337-351.

FAIR principles implementation in survey research with Steve McEachern

In this episode, Dr. Tom Emery discusses the WorldFAIR project and FAIR principles implementation with regard to survey research methods with Dr. Steve McEachern, focusing on several key points:

• What are the FAIR principles, and why are they important?
• What challenges arise when implementing FAIR principles in survey research?
• What is machine actionability, and how does it relate to survey research?
• How do FAIR principles apply to cross-cultural surveys?
• How can we make data accessible for machines?
• What potential advancements will the full implementation of FAIR principles bring to future research?

Affiliations:

Dr. Steve McEachern – Director and Manager of ADA (Australian Data Archive)

Dr. Tom Emery – Director of ODISSEI, the Dutch National Infrastructure for Social Science; Associate Professor, Department of Public Administration and Sociology of Erasmus University, Rotterdam

Useful links:

The WorldFAIR Project

Crafting a Comparative Longitudinal Study on Child Well-being: Methodological Insights of GUIDE project

In this podcast episode of our newly introduced series, Dr. Tom Emery, Director of ODISSEI and Associate Professor at Erasmus University Rotterdam engages in a comprehensive exploration of an ambitious birth cohort study spanning across Europe. Joined by esteemed guest Professor Gary Pollock from Manchester Metropolitan University, the discussion focuses on examining the intricate process of setting up a longitudinal study aimed at the dynamics of child well-being in the evolving landscape of digital technologies.

The conversation delves into the methodological challenges embedded in setting up such a study, touching upon the formulation of research questions, overcoming methodological challenges, and pioneering innovative approaches in questionnaire development. Through this dialogue, listeners gain valuable insights into the complexities inherent in longitudinal research design and the nuanced considerations essential for navigating the contemporary terrain of child well-being research.

Affiliations:

Prof. Gary Pollock – Manchester Metropolitan University, Co-Director of Survey Infrastructure for Growing Up In Digital Europe (GUIDE)

Dr. Tom Emery – Director of ODISSEI, the Dutch National Infrastructure for Social Science; Associate Professor, Department of Public Administration and Sociology of Erasmus University, Rotterdam

Useful links:

Growing up in Digital Europe (GUIDE)

Cohort Community Research and Development Infrastructure Network for Access Throughout Europe (COORDINATE)

Sensors and Apps in Surveys with Bella Struminskaya

In the ESRA (European Survey Research Association) Podcast Series, dr. Tom Emery, the director of ODISSEI (Open Data Infrastructure for Social Science and Economic Innovations in the Netherlands) and associate professor at the Erasmus University Rotterdam, connects with the leading experts in survey research to get an understanding of the field development. In the first episode, dr. Tom Emery is joined by Bella Struminskaya, Associate Professor at the Department of Methodology & Statistics at Utrecht University. Together they discuss the usage of apps and devices in survey research.

——————————————————————————————————————–

First Podcast Season (Archive)

Initiated at the ESRA21 Online Conference, two scholars from survey research are interviewed by a podcast host for approx. 45 minutes to discuss one recent topic in the research area. The current series is published bi-monthly and hosted by the ESRA President Vera Toepoel.

March 13, 2023
Smart Surveys: Collection Digital Trace Data
Vera Toepoel, Mark Trappmann and Barry Schouten

Download the audio file:

November 25, 2022
How to Make Surveys More Enjoyable
Vera Toepoel, Jon Puleston and Niels Schillewaert

Download the audio file:

August 29, 2022
How to Measure a Difficult Concept: Survey or Wearables? An Investigation of Physical Activity
Vera Toepoel, Arie Kapteyn and Hidde van der Ploeg

Download the audio file:

June 9, 2022
The Future of Survey Costs
Vera Toepoel, Joost Kappelhof and Gerry Nicolaas

Download the audio file:

April 25, 2022
Use of Probability and Non-Probability Samples
Vera Toepoel, Kea Tijdens and Carina Cornesse

Download the audio file:

July 23, 2021
Surveys: Past, Present, Future
Annette Scherpenzeel, Caroline Roberts and Eldad Davidov

Download the audio file:

July 19, 2021
Polling
Patrick Sturgis, Courtney Kennedy and Peter Lugtig

Download the audio file:

July 9, 2021
Data Linkage and Sharing
Emanuela Sala, Florian Keusch and Annette Jäckle

Download the audio file:

July 2, 2021
Surveys during the Pandemic
Tom Emery, Tim Hanson and Daphne Ahrendt

Download the audio file: