Adaptive and Responsive Designs in Complex Surveys: Recent Advancements and Challenges 1 |
|
| Chair | Mr Jason Fields (U.S. Census Bureau ) |
| Coordinator 1 | Dr Asaph Young Chun (U.S. Census Bureau) |
| Coordinator 2 | Professor James Wagner (University of Michigan) |
| Coordinator 3 | Dr Barry Schouten (Statistics Netherlands) |
| Coordinator 4 | Ms Nicole Watson (University of Melbourne) |
Adaptive designs have been used for almost a decade in survey research, providing researchers with methods to balance survey quality and survey cost during data collection (Groves and Heeringa, 2006). This article evaluates the effectiveness of such a design aiming at increasing response rates and reducing the potential for nonresponse bias in the 2016/17 Baccalaureate and Beyond Longitudinal Study (B&B, field test) of college graduates. We use prior wave response behavior to assign cases to different data collection protocols, varying contact frequency, mode, incentive amount, and interview length.
More specifically, prior wave nonrespondents were randomly assigned to either a default (used in the prior data collection), or an aggressive data collection protocol (aimed at converting nonrespondents through higher incentives, more frequent contacts, and an abbreviated interview). Prior study early respondents were assigned to a relaxed protocol, and all other respondents to the study default protocol. Data collection consisted of three phases, where different design components (incentive, survey length, or mode) were changed. We investigate response rates, sample representativeness and nonresponse bias using frame data for each group and each phase. The availability of a rich set of auxiliary frame data for all sample members allows us to compare the effectiveness of the different data collection protocols.
Preliminary results indicate the response rate is significantly higher in the aggressive (RR2=34.3%) compared to the default (RR2=19.8%, p<0.001) data collection protocols for the base-year nonrespondents, and significantly higher in the relaxed (RR2=71.0%) compared to the default (RR2=63.5%, p<0.01) data collection protocols for the base-year respondents. Furthermore, while response rates significantly increase with each data collection phase, nonresponse bias assessed using Mahalanobis distance measures decreased significantly in the first data collection phase, but not in later phases. We discuss implications of our findings for data quality and efficiency gains for data collection.
The UK Household Longitudinal Study, also known as Understanding Society, started in 2009 and for the first 7 waves almost all interviewing was completed face to face. At wave 8 in 2016 the study became mixed mode with 40% of households being invited to take part online initially (with non-responders being followed up face to face). To make the study as cost effective as possible, it was important to maximise the number of people and, in particular, whole households that completed the survey online before the start of face to face interviewing. Sample was issued in monthly batches so, to test the most effective design for web fieldwork, an adaptive design approach was used to vary the length of web fieldwork, the incentives used, and the reminder strategy for different batches. Most of these variations were implemented experimentally. This paper will cover the practicalities of using an adaptive design approach, as well as the details of the different web fieldwork designs used, how they affected response rates, and whether or not they were cost effective.
The Survey of Income and Program Participation (SIPP) is a longitudinal household survey conducted by the U.S. Census Bureau. Government policy formulators depend heavily upon SIPP for information on employment dynamics, the utilization of government assistance programs, the family context, and their interactions. The SIPP program has engaged a test of adaptive design procedures to increase the representation of movers as well as an effort to increase sample balance of program participants relative to their distribution in the initial sample. The movers are prioritized before and during data collection using administrative data. The program participation prioritization, is based on work the SIPP research team has undertaken to use R indicators as a tool to target to under represented households and balance on variables related to program participation. During the course of the first test, the research team found that balancing on some of the 17 variables could drive imbalance in other variables. This research describes the overall planning for the development of adaptive procedures in the SIPP and the research to develop a methodological approach for case prioritization using a composite R indicator consisting of all variable level and categorical level R indicators and their propensity to respond.
Responsive or adaptive designs involve making adjustments to survey design protocols during data collection in order to actively manage in real-time the trade-off between fieldwork costs, response bias and response rates (Groves and Heeringa, 2006; Couper and Wagner, 2011). These approaches are facilitated by technological developments such as electronic sample management systems allowing the real-time collection and analysis of para-data. Schouten et al. (2011) recommend the use of the ‘R’ indicator to evaluate representivity in real-time. Longitudinal studies are generally better-able than cross-sectional surveys to evaluate response bias as they are able to use prior wave data.
Although there has been a growing interest in this area, relatively few large-scale surveys have implemented these approaches. This paper will report experiences from the eighth wave of Next Steps (previously known as the Longitudinal Study of Young People in England) in managing the trade-offs between fieldwork cost, response bias and response rate in a live survey setting.
Next Steps is a cohort study following around 16,000 young people in England born in 1989-1990. The sample was recruited in 2004 when the study members were aged 13/14. By wave 7 in 2010, the sample size had fallen to around 8,600, and this attrition was strongly related to baseline socio-demographic characteristics. The study was re-started in 2013 and another wave of the survey conducted in 2015/2016, for which all participants who have ever taken part will be approached again.
The survey design for Wave 8 consists of a sequential mixed mode approach involving web, telephone and face-to-face. Due to length of time, and variability in time, since the study members were last interviewed there was considerable uncertainty about what the response rates will be, both overall and to each of the different survey modes, and therefore what the cost of the fieldwork would be.
In order to help manage this uncertainty, we conducted a ‘soft-launch’ of the survey involving a random sub-sample of 2,000 cases which provided more reliable estimates of response rates and fieldwork costs. The intention was that this would also allow us to evaluate the impact of additional fieldwork effort, particularly re-issuing and ‘enhanced tracing’ on response rates and response bias. Overall, our aim was to maximise the overall response rate, and hence the achieved sample size, and minimise non-response bias, within the context of a fixed fieldwork budget.
We set out to use response models in order to compute a response probability for each participant and use R-indictors to monitor the representativeness of the achieved sample at key points during fieldwork, to evaluate the impact of fieldwork effort on sample representivity and, in the event that it was not possible to issue all of the cases to face-to-face fieldwork due to the budget constraint, use the R-indictors to help prioritise cases to be issued.