All time references are in CEST
Mixing Modes in Longitudinal Surveys 1 |
|
| Session Organisers | Professor Mark Trappmann (Institute for Employment Research, University of Bamberg) Dr Mary Beth Ofstedal (Institute for Social Research, University of Michigan) |
| Time | Tuesday 18 July, 11:00 - 12:30 |
| Room | U6-01a |
The Covid-19 pandemic has forced many panel and cohort surveys to replace personal interviews by telephone or self-administered modes and thereby accelerated a trend of mixing modes in longitudinal surveys. While introducing new data collection modes helped prevent attrition or even the loss of entire survey waves during the pandemic, it also created new challenges for longitudinal surveys related to mode effects on survey measurement.
Many of the challenges presented by mode effects and the methodological tools for investigating and adjusting them differ between longitudinal and cross-sectional surveys. On the one hand, in longitudinal surveys the potential for harm is substantial. Even small mode-effects can dramatically affect estimates of change if the trait under investigation is relatively stable over time. On the other hand, longitudinal surveys allow exploiting within-subject variation and thus the application of more stringent methods to separate (self-)selection into mode from mode effects on measurement.
We invite submissions of research that investigate mode effects in a longitudinal setting. This may include mode experiments, analyses that separate selection effects from measurement effects, or approaches to separate mode effects from time trends and particularly from effects of the pandemic. We also invite contributions that address the impact of mode effects for longitudinal estimates and that offer solutions for communicating to users the importance of recognizing the potential for mode effects and how to deal with them in their research.
Keywords: data collection mode, mixed-mode, longitudinal surveys, mode effects
Mr Phil Schumm (The University of Chicago) - Presenting Author
Mr Ned English (NORC)
Dr Colm O'Muircheartaigh (The University of Chicago)
The National Social Life, Health, and Aging Project (NSHAP) is a longitudinal, population-based study of adults born between 1920 and 1965. It had been face-to-face since its first round in 2005-2006. The most recent round (Round 4, 2022-23) was intended to be primarily face-to-face with a test of multi-mode approaches. NSHAP had been piloting self-completion of the complex questionnaire and self-collection of biomeasures to reduce costs, allow for participation from respondents who prefer remote, and to evaluate the potential of hybrid remote/face-to-face for future work. Due to the Coronavirus pandemic, we needed to implement the multi-mode approach on a larger scale than originally planned.
NSHAP provided the opportunity to tailor contact approaches based on past experience with panel members. We partitioned the sample into two groups: (1) the most cooperative, for whom the initial approaches were remote – web-push, mailed self-administered questionnaires (SAQs), and telephone ; (2) those with whom we had had challenges, for whom the initial mode was face-to-face. In each group we followed up with alternative modes where possible. We examine the response-rates and demographics of respondents to each mode sequence.
A direct comparison of the response rates is unsatisfactory as a mode evaluation as the composition of the two groups is deliberately non-random, differentiated between cooperative and relatively non-cooperative panel members. Consequently, we incorporated a randomized experiment in the data collection, in which we assigned 400 cases from group 1 (those initially to be approached remotely) to group 2 (where they were first approached face-to-face). The results of the experiment will help in the interpretation of the overall comparison.
Our paper will be useful to survey methodologists who are considering multi-approaches for longitudinal studies.
Ms Eva Leissou (University of Michigan - Survey Research Center) - Presenting Author
Mr Paul Burton (University of Michigan - Survey Research Center)
Mr Andrew Hupp (University of Michigan - Survey Research Center)
Dr Brady West (University of Michigan - Survey Research Center)
Every six years, the Health and Retirement Study adds a new age cohort to its existing longitudinal panel. Given the narrow recruitment criteria, many sampled households screen ineligible, which requires a great deal of effort and a lengthy field period. While previous screening efforts have been primarily in-person, we invited a portion of the sample in 2022 to complete the screening questionnaire via the web to address declining response rates and increasing costs. Sampled households were randomly assigned to one of two protocols: in-person first, or web first with in-person follow-up for non-responding cases. Households assigned to in-person first were mailed a prenotification letter followed by an interviewer visit. Households assigned to web first were mailed an invitation letter containing the URL and a QR code, a brochure, and a $2 bill. Two weeks later, non-responding households were sent a reminder letter with a URL, a paper questionnaire and business reply envelope. Four weeks later, any non-responding cases were sent to the field for follow-up by an interviewer. As part of the invitation mailing, we tested two different envelope types to see which was more effective. Households were randomly assigned to either receive an envelope with the $2 visible through a small window, or a maize and blue envelope (with no cash visible). In this presentation, we will report on 1) the effectiveness of the web option compared to the traditional in-person option (in terms of response rates and effort), and 2) cost and response rate differences between the visible cash and maize and blue envelopes. Preliminary results suggest that the visible cash envelope is most effective during the web protocol, with higher completion rates than the maize and blue envelope, but we see no difference in the effort required.
Dr Mengyao Hu (University of Michigan) - Presenting Author
Professor Vicki Freedman (University of Michigan )
Mr Justin Kamens (Westat)
Mr Shawn McCloskey (Westat)
Professor Mick Couper (University of Michigan )
Limited experimental research has explored in a national context how the transition from telephone to a mix of web and telephone modes affects response rates (RR), field effort, and potential bias in estimates. We extend this research by experimentally evaluating a change from a phone-only to mixed mode design for a national follow-back study of caregivers to older adults. The National Study of Caregiving (NSOC) interviews family and unpaid individuals assisting older adults in the National Health and Aging Trends Study (NHATS). Prior NSOC rounds (e.g., 2015, 2017) were by telephone and most NSOC interviews were completed within one month of NHATS. In 2021, NHATS embedded an experiment so that half of NSOC-eligible caregivers were randomized to the usual telephone-only design and the other half to a sequential mixed-mode design offering web and telephone options. Contact information has been obtained for over 3,000 NSOC-eligible individuals. Results show a higher predicted RR at completion for the mixed-mode design (62.3% RR; about half online) relative to phone-only in 2021 (58.3%). Among cases randomly assigned to sequential mixed modes, RR and % by web varied by available contact information. Those with a mailing and email address (30%) had the highest RR and % by web (79% RR; 79% by web) and those with no mailing address (15%) had the lowest rates (41% RR; 9% by web). Average days since NHATS was only slightly longer for the mixed mode sample (20.7 days) relative to the phone-only sample (19.1 days). We also explored effects of changing to a mixed mode design on NSOC estimates and whether these effects differed by question types (subjective vs. objective) and response scales (e.g., ordinal, nominal or open-ended). Finally, implications for trend analyses are discussed.
Dr Narayan Sastry (University of Michigan) - Presenting Author
Dr Katherine McGonagle (University of Michigan)
Assessment of mode effects on interview responses is important for longitudinal studies that switch from interviewer-administered to self-administered and mixed modes. The gold standard for such an assessment is an experimental design in which respondents are randomized to mode. We implemented such a design in the 2019 Transition into Adulthood Supplement (TAS) to the US Panel Study of Income Dynamics as it shifted from telephone only in 2017 to a fully mixed mode design (online and telephone) in 2019.
TAS is a national longitudinal survey that interviews young adults biennially, covering a wide variety of topics including those potentially sensitive to mode effects such as alcohol and drug use, sexual behavior, and mental health. TAS has collected eight waves of data since its launch in 2005, with response rates of 87% to 92%.
The eligible TAS-2019 sample comprised 2,964 young adults. We randomly assigned 80% to a mixed mode data collection protocol that offered an online mode first, followed by telephone. The remaining 20%) was randomized to the telephone only mode with no option for completing the survey online.
Random assignment allowed us to undertake intent-to-treat and treatment-on-the-treated assessments of mode assignment and mode of completion effects on interview responses. We drew on the panel design of TAS to conduct a difference-in-differences analysis of longitudinal changes in interview responses due to interview mode for panel respondents (N=2,121) who had also participated in TAS in 2017 when the survey was telephone only.
This presentation will describe results from the intent-to-treat, treatment-on-the-treated, and difference-in-difference analysis of item nonresponse and of substantive response patterns by mode. We will investigate specific hypotheses for potential mode effects, including social desirability bias, satisficing, and questionnaire response