Program at a glance 2021

Survey Participation and Breakoffs

Session Organiser Mr Julian B. Axenfeld (University of Mannheim)
TimeFriday 16 July, 13:15 - 14:45

Increasing and securing high survey response rates are core issues in survey research striving to secure high data quality and limit costs. This only gets more significant the more response rates continue to globally decline and pressures to save costs intensify. The increasing use of online data collection does its part in limiting costs, but also brings new challenges such as issues of higher breakoff rates, noncoverage, or multi-device use.
This session explores new developments in recent research regarding the prediction and monitoring of survey participation and breakoff as well as interventions aiming especially to increase and improve survey participation. Research particularly revolves around when to contact respondents, the effectiveness of push-to-web strategies, and how to present the questionnaire regarding mobile optimisation, survey topics and filter questions.

Keywords: response rates, nonresponse, data quality, mixed-mode surveys, web surveys

Impact of Question Topics and Filter Question Formats on Web-survey Breakoffs

Mr Zeming Chen (University of Manchester) - Presenting Author
Mr Alexandru Cernat (University of Manchester)
Mrs Natalie Shlomo (University of Manchester)
Ms Stephanie Eckman (RTI International)

Web surveys have become increasingly popular over the last decade, but they tend to suffer from more breakoffs, which take place when respondents start the survey but do not complete it. Many studies have investigated the factors impacting breakoffs, but they often ignored the breakoff timing and gave scant attention to two factors: question topics and filter question formats (grouped vs. interleafed as defined by whether filter questions are presented upfront or not). This study aims to research the effect of these two factors on the breakoff event and its timing with the help of survival analysis. By using a web survey that experimentally manipulates the filter question format and randomly orders the question topic, this study finds that presenting the filter questions in the grouped format leads to lower breakoffs at the beginning in comparison to the interleafed counterpart, but the breakoff risk in the grouped format catches up quickly when respondents realise their previous answers will trigger more questions. This study also identifies that the insurance topic has more breakoffs while demographic and income topics have fewer breakoffs despite their perceived sensitivity level. The present study demonstrates the importance of taking into account the time dimension in the study of breakoff and gives practical guidance on questionnaire design and survey breakoff mitigation.

Effects of Single- and Mixed-Mode Designs on Establishment Survey Participation and Costs

Mr Benjamin Küfner (Institute for Employment Research (IAB)) - Presenting Author
Professor Joseph W. Sakshaug (Institute for Employment Research (IAB))
Dr Stefan Zins (Institute for Employment Research (IAB))

The IAB-Job Vacancy Survey is a voluntary nationally-representative establishment survey that quantifies the size and structure of job vacancies and other worker flows in Germany. Since 2011, it has been carried out using a concurrent mixed-mode design, with establishments receiving paper questionnaires and the option of online completion. This mode design is facing increasing costs and declining response rates. To counteract these trends, a more pronounced push-to-web strategy offers a promising alternative. However, a change of mode design might affect nonresponse bias and data quality. To test an implementation of a mode design switch, a large-scale experiment comparing four self-administered mode designs was conducted with 155,000 establishments in the 4th quarter of 2020: Just Online, Just Paper, Sequential Mixed Mode with online invitation followed by paper questionnaire for nonrespondents, and the standard concurrent mixed-mode design. Further, we experimented with a pre-due-date reminder as an additional response enhancement measure motivating establishments to respond earlier. In this paper, we present first results of these experiments on response rates and costs.

The impact of day of mailing on web survey response rate and response speed

Professor Peter Lynn (University of Essex)
Professor Annamaria Bianchi (University of Bergamo)
Dr Alessandra Gaia (University of Milano-Bicocca) - Presenting Author

Survey researchers using online data collection methods continue to invest in efforts to identify ways of improving response rates. Response speed is also of importance, particularly for surveys that use methods other than email to send reminders or to seek participation from initial nonrespondents (i.e. push-to-web mixed mode surveys), due to the additional costs associated with slow response. Meanwhile, the tools used to improve response rates and response speed have become more sophisticated, particularly various types of adaptive designs. Researchers no longer focus on the average effect of survey design features but are instead interested in the effect on subgroups of particular interest, namely those with otherwise low response rates or a propensity for slow response. This reflects a recognition that both outcomes of interest (response rate, response speed) and the effectiveness of design features that influence the outcomes may vary substantially over sample subgroups.

Panel surveys provide a particularly rich environment for the application of targeted or static adaptive designs as the wealth of prior information available can be used to identify subgroups with (likely) variation in the outcomes of interest and to inform the choice of design features that might provide improved outcomes. The design feature of interest in this article is the day of week on which an invitation to participate is mailed. Using experimental data from the Understanding Society Innovation panel we study the interaction between the day of week on which an invitation to participate is mailed and socio-demographic and survey participation characteristics of sample members. The socio-demographic characteristic in which we are interested is, specifically, economic activity status as we hypothesise ways in which the association between mailing day and outcomes might vary depending on economic activity status. We also examine the role of prior participation in the panel and previous provision of an email address as moderators of the interaction between mailing day and economic activity status. The latter affects the frequency and volume of reminders that can be sent and could well therefore interact with mailing day. Our study therefore provides evidence of whether any interaction between mailing day and economic activity status depends on whether an email address is available, allowing invitations and reminders to be sent by email.

Predicting survey breakoff in Web surveys

Dr Anke Metzler (Darmstadt University of Technology)
Professor Marek Fuchs (Darmstadt University of Technology) - Presenting Author

Web surveys suffer from substantial breakoff which has the potential to induce breakoff bias, decrease data quality and cause inaccurate survey estimates. Interactive features have successfully been incorporated in Web surveys to compensate for potential increases of item missing, non-differentiation or speeding. To adopt this approach for survey breakoff, it is essential to identify these respondents prior to their actual dropping out of the survey.

For analyses reported in this paper, we used five Web surveys among university applicants conducted in 2013 (n=7,395), 2014 (n=5,996), 2015 (n=4,034), 2016 (n=944) and 2017 (n=545). We assess previous response behavior such as response time and item nonresponse, as well as page characteristics of previous pages and the breakoff page such as number of characters, number of questions, number of answer categories and question type to predict subsequent survey breakoff on question level.

Previous findings suggest that the item nonresponse and response time alone do not reliably predict breakoff, in part because the effect of item nonresponse and response time on breakoff is non-linear: “careless respondents” (very fast and no item missing), “failing optimizer” (very slow and very high item missing rates) and “lurking dropouts” (very fast and very high item missing rates) are more likely to break off than respondents with intermediate response time and moderate levels of item nonresponse.

In this presentation, we present more sophisticated multilevel analyses modelling the nonlinear effects and also interaction effects of response time and item nonresponse. In addition we include indicators of page characteristics to allow a more reliable prediction of breakoff. Results suggest that item nonresponse and response time can be used to more reliably predict survey breakoff when indicators for response burden of the particular survey page are considered in parallel.