ESRA 2025 Program

Monday 14 July    Tuesday 15 July    Wednesday 16 July    Thursday 17 July    Friday 18 July   

Get the ESRA 2025 App on iOS or Android:
    

All time references are in CEST

Adapting survey mode in a changing survey landscape: Experiences from repeat cross-national, cross-sectional, and general social surveys 2
Session Organisers	Dr Gijs van Houten (Eurofound) Dr René Bautista (NORC at the University of Chicago) Professor Rory Fitzgerald (ESS HQ; City St Georges, University of London) Mr Tim Hanson (ESS HQ; City St Georges, University of London) Mr Nathan Reece (ESS HQ; City St Georges, University of London) Ms Daphne Ahrendt (Eurofound) Ms Jodie Smylie (NORC at the University of Chicago)
Time	Tuesday 15 July, 15:30 - 17:00
Room	Ruppert Blauw - 0.53

Studies to measure attitudes, opinions, and behaviors are critical to understanding societies around the world. In the face of social developments, changing trends in respondent recruitment methods, budget constraints, national infrastructure disruptions, and public health concerns, many repeat cross-sectional social surveys are experimenting with self-completion and mixed-mode approaches. The European Social Survey (launched 2001), United States’ General Social Survey (launched 1972), and the European Quality of Life Surveys (launched 2003) are examples of longstanding studies collecting data to inform research on changes over time and now exploring and transitioning to new modes. This session brings together cross-sectional social surveys to share experiences in survey mode transition.

The session's aims include: (1) Share results and lessons from recent mode experiments and mixed-mode applications by general social studies, and potential ways to improve methods. (2) Highlight how different cross-sectional studies have recently modified survey protocols to adapt to changing public conditions. (3) Provide space for data creators, data users, and survey practitioners to discuss methodological and statistical challenges for cross-sectional studies considering such moves. (4) Discuss integrity and comparability of data collected using new data collection methods with the existing time-series. (5) Explore applications of emergent technologies to new modes.

We invite submissions from those involved in transitioning repeat, cross-sectional, and cross-national social surveys to new data collection approaches. Topics of interest include: results from pilots or feasibility studies based on self-completion or mixed-mode approaches; findings from experimental research testing aspects of self-completion/mixed-mode designs (e.g., incentive and mailing strategies, survey length adaptations, sequential vs. concurrent designs); impacts of mode switches on measurement and survey time series; and discussions of experiences and challenges with adapting cross-sectional surveys to new modes across different cultural/national contexts.

Keywords: general surveys, survey methodology, data collection, data collection modes, mixed mode, self administration

Papers

Surveys at the “cross-modes”: Leveraging ESS Round 10 natural experiment to compare response behaviours between face-to-face and self-completion modes

Dr Marek Muszyński (Institute of Philosophy and Sociology, Polish Academy of Sciences) - Presenting Author
Professor Piotr Jabkowski (Faculty of Sociology, Adam Mickiewicz University, Poznan)

Rising costs and challenges in interviewer-administered surveys necessitate the development of mixed-mode studies. Cross-mode comparisons are essential to maintain consistent measurement quality across modes. Research indicates that mode differences affect response behaviours, including attentiveness (Daikeler et al., 2024; Kim et al., 2019) and response styles (Aichholzer, 2013; Hope et al., 2022; Liu, 2015).

The COVID-19 pandemic disrupted the European Social Survey (ESS) Round 10, planned for late 2020 and early 2021, leading some countries to adopt self-completion methods instead of face-to-face interviews. This shift created a natural experiment, enabling the comparison of response behaviours across different survey modes. Our study capitalises on this opportunity to analyse response patterns such as inattentive responding and response styles in ESS Rounds 9 and 10, contrasting the primarily face-to-face mode in Round 9 with the mixed modes of Round 10, where about a quarter of the countries used self-completion modes (postal and web surveys). This is further compared to Round 11, which returned to face-to-face mode.

We employed multilevel regression models to examine cross-mode response behaviours. Outcome variables included straightlining indices and measures of response styles (extreme, midpoint, and acquiescent), with survey mode and socio-demographics (age, gender, education, etc.) as moderators.

Preliminary findings indicate that respondents in self-completion modes exhibit less straightlining and fewer acquiescent and midpoint responses than in face-to-face modes, but yield more extreme responses and outliers identified by Mahalanobis distance. These differences are more pronounced with 11- compared to 5-point rating scales. Mode effects were held when controlled for participant socio-demographic characteristics. Interaction effects between mode and socio-demographics were mainly non-significant.

The findings are significant for designing mixed-mode international surveys and theoretical understanding of mode differences in response behaviours.

Exploring the impact of Mixed-Mode Effects in the Generation and Gender Survey (GGS)

Miss Ilaria Francesca Lunardelli (NIDI - KNAW) - Presenting Author

The second round of the Generations and Gender Survey (GGS) introduced Computer-Assisted Web Interviewing (CAWI) as the primary data collection mode, marking a significant shift towards digital methods in sensitive survey research.
While most countries implemented CAWI as the sole mode for GGS-II Wave 1, a subset opted for mixed-mode data collection, incorporating CAWI alongside other modes such as Computer-Assisted Telephone Interviewing (CATI), Paper and Pencil Interviewing (PAPI), or Computer-Assisted Personal Interviewing (CAPI).
The choice to use multiple modes may enhance sample diversity and inclusion, yet it also introduces potential mixed-mode effects. These effects are particularly critical when examining sensitive indicators, such as fertility and well-being metrics, in a longitudinal, cross-country framework.
To examine the impact of mixed-mode effects, this study will employ a combination of descriptive analyses, regression models and statistical tests within a cross-country comparative framework. The analysis will investigate how mixed-mode effects manifest across different geographical and cultural contexts, such as the French and Uruguayan one, to understand whether these settings influence the extent and nature of mode-related biases. Ultimately, special attention will be given to sensitive indicators, as they are particularly susceptible to mixed-mode effects and hold central importance in in GGS.
Preliminary results suggest that, overall, mixed-mode data collection has contributed to higher response rates and greater diversity among respondents. However, mode effects can still be detected, with variations depending on the country and greater impact on sensitive indicators.
The GGP Central Hub Team aims to provide insights into the implications of mixed-mode effects on accuracy and comparability, as well as contributing to the optimization of future data collection.

The Impact of Switching to Self-Completion Protocols on Item Nonresponse in Complex and Sensitive Survey Questions. Lesson learned from the ESS rounds 9, 10, and 11

Dr Piotr Jabkowski (Adam Mickiewicz University, Poznan) - Presenting Author
Dr Piotr Cichocki (Adam Mickiewicz University, Poznan)
Dr Aneta Piekut (The University of Sheffield)

The COVID-19 pandemic forced a shift in data collection methods, with many surveys adopting self-completion modes to overcome social distance constraints. This study examines the consequences of the transition toward self-completion protocols in the European Social Survey (ESS) and aims to answer whether the transition impacts survey quality. Notably, we focus on item nonresponse rates in complex and sensitive questions. Our analysis works on data from 23 countries participating in the ESS round 9 (2018, face-to-face modes), round 10 (2020, self-completion and face-to-face protocols), and round 11 (2022, face-to-face modes).

We focus on item nonresponse in two types of questions: complex items, such as household composition (gender, age and the relationship with respondent), and sensitive items, such as income and several measures of political trust. Using multilevel regression models, we predicted the probability of item nonresponse occurrence across self-completion (web, paper) and face-to-face modes of data collection (PAPI, CAPI, Video: web), controlling for individual-level predictors (e.g. age, education, gender) and several country-level characteristics.

Our results show differences in nonresponse patterns between face-to-face and self-completion modes. Compared to rounds 9 and 11, when countries utilised self-completion protocols in Round 10, they exhibited lower nonresponse for sensitive items but higher nonresponse rates in complex questions. In contrast, when countries constantly used face-to-face modes for data collection across three rounds, they did not demonstrate any significant differences in item nonresponse rates over time.

Our findings highlight the need for tailored methodological adaptations when moving to self-completion protocols, particularly addressing potential biases introduced by item nonresponse. Our analysis contributes to survey methodology by providing insights for optimising mixed-mode designs, ensuring data quality, and maintaining data comparability over time.

The effect of questionnaire duration on yield, sample composition and data quality: findings from the 2024 European Working Conditions Survey

Dr Gijs van Houten (Eurofound) - Presenting Author
Dr Tanja Kimova (Verian)
Ms Hajar Gad (Verian)
Mr Christopher White (Eurofound)
Mr Jamie Burnett (Verian)

The European Foundation for the Improvement of Living and Working Conditions (Eurofound) commissioned Verian to conduct the eighth edition of the European Working Conditions Survey (EWCS) in Spring 2024 in 37 countries. As part of Eurofound’s strategy for future-proofing its surveys, the EWCS 2024 was conducted both face-to-face and online in all EU Member States (using a telephone push-to-web or a postal push-to-web approach), and the implementation of the online component included a range of test elements.
A key test element is the duration of the online questionnaire. The questionnaire for the face-to-face segment of the EWCS takes around 45 minutes to complete. Ideally, respondents completing the survey online would complete the same questionnaire. However, there are concerns about the impact of questionnaire duration on survey yield, sample performance, and data quality.
To assess these impacts, respondents in the online segment of the survey were randomly allocated to one of two conditions in terms of questionnaire duration: full (ca 40-45 minutes) or abridged (ca 25-30 minutes). The sample split is informed by the results of a pilot test carried out in all countries, with a view to maximise the likelihood of achieving an even split in the net sample.
In this paper we will discuss the results of this test, assessing the differences between the questionnaire duration conditions in terms of yield, sample composition, and data quality. The pilot test suggested a negative impact of questionnaire duration on yield in most countries. Analysis of the mainstage data will allow assessing this overall reduction of yield, and the impact on the cost per item, against the impact on response bias and data quality.