All time references are in CEST
Adapting survey mode in a changing survey landscape: Experiences from repeat cross-national, cross-sectional, and general social surveys 1 |
|
| Session Organisers | Ms Siobhan O'Muircheartaigh (European Social Survey ERIC (City, University of London)) Mr Tim Hanson (European Social Survey ERIC (City, University of London)) Dr Oshrat Hochman (GESIS) Dr Rene Bautista (NORC at the University of Chicago) Professor Rory Fitzgerald (European Social Survey ERIC (City, University of London)) |
| Time | Tuesday 18 July, 11:00 - 12:30 |
| Room | U6-07 |
Studies to measure attitudes, opinions and behaviours have been and remain critical to understanding societies around the world. In the face of the COVID19 pandemic and changing trends in the interviewer workforce, many repeat cross-sectional social surveys have been experimenting with self-completion and mixed-mode approaches. The European Social Survey (launched in 2001) and United States’ General Social Survey (launched in 1972) are key examples of long-standing studies that collect data to inform research on changes over time, and are now exploring new modes for the future. This session brings together ESS, GSS, and other cross-sectional social surveys to share experiences in survey mode transition.
The session's aims are threefold: (1) Share strategies and lessons-learned from recent mode experiments by ESS, GSS, and other studies, and potential ways to improve methods in future. (2) Highlight how different cross-sectional studies have modified survey protocols in recent years to adapt to changing conditions in the public (e.g., public health crisis, shifting communication modes, public’s willingness to respond to surveys, trends in interviewer workforce). (3) Provide a space for data creators, data users, and survey practitioners to discuss methodological and statistical challenges for cross-sectional studies considering such a move.
We invite submissions from those involved in transitioning repeat, cross-sectional, and cross-national social surveys to new data collection approaches. Topics of interest include: results from pilots or feasibility studies based on self-completion or mixed-mode approaches; findings from experimental research testing aspects of self-completion/mixed-mode designs (e.g., incentive and mailing strategies, survey length adaptations, sequential vs. concurrent designs); impacts of mode switches on measurement and survey time series; and discussions of experiences and challenges associated with adapting cross-sectional surveys to new modes across different cultural/national contexts.
Dr Mengyao Hu (University of Michigan)
Professor Vicki Freedman (University of Michigan )
Mr Justin Kamens (Westat)
Mr Shawn McCloskey (Westat)
Professor Mick Couper (University of Michigan )
Dr Mary Beth Ofstedal (University of Michigan) - Presenting Author
Limited experimental research has explored in a national context how the transition from telephone to a mix of web and telephone modes affects response rates (RR), field effort, and potential bias in estimates. We extend this research by experimentally evaluating a change from a phone-only to mixed mode design for a national follow-back study of caregivers to older adults. The National Study of Caregiving (NSOC) interviews family and unpaid individuals assisting older adults in the National Health and Aging Trends Study (NHATS). Prior NSOC rounds (e.g., 2015, 2017) were by telephone and most NSOC interviews were completed within one month of NHATS. In 2021, NHATS embedded an experiment so that half of NSOC-eligible caregivers were randomized to the usual telephone-only design and the other half to a sequential mixed-mode design offering web and telephone options. Contact information has been obtained for over 3,000 NSOC-eligible individuals. Results show a higher predicted RR at completion for the mixed-mode design (62.3% RR; about half online) relative to phone-only in 2021 (58.3%). Among cases randomly assigned to sequential mixed modes, RR and % by web varied by available contact information. Those with a mailing and email address (30%) had the highest RR and % by web (79% RR; 79% by web) and those with no mailing address (15%) had the lowest rates (41% RR; 9% by web). Average days since NHATS was only slightly longer for the mixed mode sample (20.7 days) relative to the phone-only sample (19.1 days). We also explored effects of changing to a mixed mode design on NSOC estimates and whether these effects differed by question types (subjective vs. objective) and response scales (e.g., ordinal, nominal or open-ended). Finally, implications for trend analyses are discussed.
Dr Irit Adler (B.I. Cohen Institute, Tel Aviv University) - Presenting Author
Mr Erez Garnai (B.I. Cohen Institute, Tel Aviv University)
Professor Sigal Alon (B.I. Cohen Institute, Tel Aviv University)
Preparation for ESS round 10 in Israel had begun while Covid’s second wave was at its peak, and restrictions did not allow for safe enough conditions for Face-to-Face (F2F) interviews. The imminent risk of a lockdown and a limited budget made conducting F2F interviews too big. Therefore, we decided to field this round as Self-Completion. Unfortunately, postal services in Israel are, de facto, non-existent. We could have used commercial postal services, but there was no way for respondents to send their envelopes back (for paper questionnaires). As a result, we decided to use a Fieldworker-Assisted data collection mode. It had been an uncharted terrain, but we gained experience and knowledge that enabled us to develop this method further.
Self-Completion Fieldworker-Assisted data collection has important advantages over Postal recruitment and F2F. Compared with Postal recruitment, the ability to contact respondents can enhance the response rate; fieldworkers (FWs) can help with general questions about the survey, assist with respondents’ reservations, provide a link for the web questionnaire/ scan the QR code, and hand out paper-questionnaires when needed. Furthermore, using FWs allows for more control over fieldwork; offers the possibility of managing difficulties in the field; helps locate addresses, and encourages respondents to complete partial questionnaires. The response rate of the Self-Completion Fieldworker-Assisted method in Israel was similar to that obtained by the Postal Self-Completion method in other countries and lower than the F2F method in Israel, yet, the basic sample composition shows good sample representativeness.
Analyzing ESS10 fieldwork data enables us to present key findings: the effect of contact made by FWs on response rate; the likelihood that households with a phone will participate in the survey; the effect of the reminder on response rate, the interviewers’ effect, and more.
Dr Michal Kotnarowski (Institute of Philosophy and Sociology, Polish Academy of Sciences) - Presenting Author
In this paper, we will present the experiences of the Polish European Social Survey (ESS) team with the implementation of the self-completion mode in ESS round 10. In particular, we will describe practices related to the optimisation of the survey to increase the response rate. Our activities included testing various incentives (conditional and unconditional) and various approaches in offering web-questionnaire and paper questionnaires (concurrent vs sequential mode). In addition, we will present the steps involved in sending a fieldworker to convince respondents who did not respond to the postal invitation to participate in the survey. The analyses will also show how the sociodemographic structure of the data obtained differs depending on the approach used. The presentation will also show the results of a short survey about the self-completion mode conducted among respondents who did not respond to the postal invitation. The results of our analysis can indicate which solutions worked well in the implementation of the self-completion mode. We can also identify the main reasons for non-response to a postal invitation.
Dr Anke Erdmann (Federal Criminal Police Office) - Presenting Author
Mr Christoffer Glaubitz (State Criminal Police Office Schleswig-Holstein)
In 2012 and 2017, a nationwide study on victimization and crime-related attitudes in the German population was conducted using CATI (“German Victimization Survey”, DVS). Both waves had moderate response rates of 19% (2012) and 14% (2017), which emphasizes the population’s decreasing willingness to participate in telephone surveys. In 2020, a new nationwide and periodical victim survey was launched in Germany (“Safety and Crime in Germany”, SKiD). Regarding the increase of web-based surveys (and the concurrent decline in CATI-surveys) and because a PAPI-mode has achieved remarkable response rates in comparable studies on the federal state level, SKiD was designed as a mixed-mode study combining PAPI and CAWI simultaneously. Around 123.000 randomly chosen residents received a postal questionnaire containing also a QR-Code and a weblink. Thus, the respondents could choose whether they wanted to participate using PAPI or CAWI mode. Although this design reached a satisfying response rate of 38%, it is necessarily tied to high financial costs. In search of a way to reduce costs without decreasing data quality, an experiment was implemented that tested a sequential mixed-mode design: For the experimental group (n=1000), the study was initially announced and carried out as a push-to-web survey. A paper questionnaire was sent for the first time with the second reminder, thus, PAPI-mode was only available for people that did not respond after being contacted twice. The main goal of this sequential design was to realize as many interviews as possible using CAWI-mode in order to reduce the amount of cost-intensive paper-and-pencil-interviews. In this presentation, we evaluate this experiment regarding response rate, sample composition, and cost effectiveness. Consequences for future research – particularly for the forthcoming waves of the large-scale survey SKiD – will be discussed.
Mr Benjamin Schapiro (NORC at the University of Chicago) - Presenting Author
Mr Eyob Moges (NORC at the University of Chicago)
Dr Rene Bautista (NORC at the University of Chicago)
As cross-sectional longitudinal surveys transition from single mode to multi-mode designs it is important to understand the implications of design and measurement changes on the resulting data. The General Social Survey (GSS) and the European Social Survey (GSS) are both undergoing such transitions and are focused on maintaining the integrity and comparability of the data collected. The introduction of design changes may introduce changes in the estimates that may confound comparisons with historical data. Experiments provide a mechanism to understand such measurement implications.
This presentation aims to leverage ESS survey data as point of reference to further understand survey item experiments conducted in the 2021 GSS. The 2021 GSS was administered as a web-only survey mode and included web-specific experiments. One experiment manipulated response categories to replicate volunteered survey responses commonly captured in face-to-face mode Neither condition perfectly replicated the previous GSS face-to-face question, posing an issue with comparisons to historic GSS time-series. Other surveys --such as the European Social Survey-- may represent an opportunity to add insight and supplement our understanding of new GSS experimental items. The ESS contains many questions with wording very similar to experimental GSS items, and response scales which better reflect the measurement properties of the experimental items than existing GSS items.
In this presentation, we analyze data from the 2021 GSS voluntary response experiment, and Rounds 5-10 of the ESS, analyzing whether these GSS experiments produce distributions comparable to the ESS, specifically with regard to more neutral response categories. We will examine several non-experimental overlaps between GSS and ESS items, to understand any comparisons between the two data sources. This analysis will illustrate how, in the absence of within-study trends with modified response categories, surveys of a similar scale and length can