All time references are in CEST
Interviewers and measurement quality 2 |
|
| Session Organiser | Dr Michael Link (IPSOS, Atlanta, GA USA) |
| Time | Thursday 20 July, 16:00 - 17:30 |
| Room | U6-28 |
Many factors – personal, design, environmental – shape interviewer behavior and can impact measurement quality. These studies examine this complexity through various lenses. The first considers the shortage of qualified field staff for in-person survey data collection in the US as a result of technological advances and pandemic-driven work-life balance changes. Comparisons are made between US and European experiences and recommendations are provided for adapting recruitment practices and alleviating workforce demands. The second study considers the effect of mixed-mode designs on interviewer variance in data collection. The results suggest that interviewer variance may be larger in face-to-face mode for sensitive items, while larger in telephone mode for non-sensitive items. Third, the European Social Survey (ESS) seeks to minimize undesirable interviewer behavior (UIB) and monitor interviewer behavior to ensure data quality. A holistic approach is used to prevent, detect, and assess interviewer-related issues and minimize UIB. The fourth investigates the effect of interviewer's age on respondents' values in the ESS from 30 countries. Results show that older interviewers primed conservative values and reduced openness to change. Fifth, gamification has been shown to improve Field Interviewer experience, motivation and loyalty in survey research. The paper discusses the effectiveness of multiple production-driven gamification programs in a nationwide survey and their impact on Field Interviewer behavior and resulting respondent engagement. The final study investigates response styles (straight-lining, middle-, extreme-responding, and acquiescence) in face-to-face surveys using data from the Portrait Values Questionnaire in the ESS in 18 countries. Results showed significant effects of respondent-interviewer gender and age matching, while interview length was also related to the presence of response styles. The study recommends using interviewer-collected data and further studies on response styles in face-to-face surveys.
Keywords: interviewer behavior, measurement quality
Ms Wenshan Yu (University of Michigan) - Presenting Author
Professor Michael Elliott (University of Michigan)
Professor Trivellore Raghunathan (University of Michigan)
As mixed-mode designs become increasingly popular, their effects on data quality have attracted much scholarly attention. Most studies focused on the bias properties of mixed-mode designs; however, few of them have investigated whether mixed-mode designs have heterogeneous variance structures across modes. While many factors can contribute to the interviewer variance component, this study investigates whether interviewer variances are equal across modes in mixed-mode studies. We use data collected with two designs to answer the research question. In the first design, when interviewers are responsible for either face-to-face or telephone mode, we examine whether there are mode differences in interviewer variance for 1) sensitive political questions, 2) international attitudes, 3) and item missing indicators, using the Arab Barometer wave 6 Jordan data with a randomized mixed-mode design. In the second design, we draw on Health and Retirement Study (HRS) 2016 core survey data to examine the question on three topics when interviewers are responsible for both modes. The topics cover 1) the CESD depression scale, 2) interviewer observations, and 3) the physical activity scale. To account for the lack of interpenetrated designs in both data sources, we include propensities of responding in the FTF mode conditional on various demographic variables in our models. Given the small power of this study, we find significant differences in interviewer variances on one item (twelve items in total) in the Arab Barometer study; whereas for HRS, the results are three out of seventeen. Overall, we find the magnitude of the interviewer variances larger in FTF than TEL on sensitive items. However, for interviewer observation and non-sensitive items, the pattern is reversed. The analytical strategy applied in this study may serve as a tool for future interviewer monitoring in mixed-mode studies.
Dr Maksim Rudnev (University of Waterloo) - Presenting Author
The age of the interviewers in the face-to-face comparative studies varies widely across countries but little attention was paid to its effects. Existing studies link the presentation of images of older people with responses emphasizing long-term goals and delayed gratification at the cost of a quick gratification. It was explained with respondents’ identification with an older person, which in turn enhanced respondents’ adherence to social norms. We hypothesized that the presence of older adults primes conservative values and tested it on the data from 30 countries and 3.522 interviewers in the ninth round of European Social Survey (2018). The results of a three-level regression demonstrated that the age of interviewer had a positive effect on respondent's Conservation values and a negative effect on Openness to Change values. The effect sizes were small which is compatible with the characteristic stability of values. The other values were not affected by the age of interviewer. The match between respondent's and interviewer's age was a non-significant predictor supporting the reliability of the priming effect. These effects were similar across countries which points to a universality of the value effects primed by older interviewers. The relative position of the country averages of value means was not substantially affected by the correction on the interviewer’s age. We discuss the substantive and methodological consequences of our findings.
Mrs Tammy Cook (Westat)
Ms Victoria Vignare (Westat) - Presenting Author
Gamification has been shown to improve user experience, boost morale, and increase participant loyalty (Richter, Raban, and Rafaeli, 2015). It has been increasingly applied to a variety of fields like marketing, education and survey research (Puleston, 2011) and its use was further accelerated when the pandemic forced everyone to migrate to a remote environment. In 2022, the United States, like much of the world, faced the continuing effects of the COVID-19 pandemic, widespread labor shortages, and changing attitudes towards the balance of work-life benefits. These challenges effect Field Interviewer recruiting, training, staffing, production, and attrition. The employment environment in our industry generated the need for novel innovations in Field Operations Management to engage and motivate employees. How can gamification be affectively applied to Field Interviewer careers? Especially with tasks that are not enjoyable, like cold calling. Can it increase the data collector reliability? What effect does it have on production? Which elements of visualization are included in a successful program? These questions and others were tested in a large, complex, nationwide, longitudinal survey with hundreds of Field Interviewers. This paper discusses the effectiveness of multiple production-driven gamification programs and their effect on Field Interviewer behavior. Our results reveal several fascinating effects of gamification and concludes with a look at opportunities to expand these strategies to promote study respondent engagement.