Using Paradata to Improve Survey Data Quality 1
|Convenor||Professor Volker Stocké (University of Kassel, Germany )|
|Coordinator 1||Professor Jochen Mayerl (TU Kaiserslautern, Germany)|
|Coordinator 2||Dr Oliver Lipps (Swiss Centre of Expertise in the Social Sciences (FORS), Lausanne, Switzerland)|
Monitoring interviewer performance is critical to successful data collection. efforts. This paper will present results from the first experimental evaluation of the use of incorporating model-based paradata along with common management reports to better coach interviewers and improve interviewer performance. Using a large telephone survey, we randomize interviewers into two groups – (1) assessment by commonly used reports only and (2) assessment by combination of commonly used reports and the model-based paradata. The model-based paradata uses case history and other paradata to account for sample differences across interviewers to allow faster and more accurate detection of under-performing
In 2013, the National Health Interview Survey (NHIS) added interviewer observations to the Contact History Instrument (CHI), asking interviewers to record neighborhood and sample unit characteristics hypothesized to predict survey response and key survey estimates. Using data collected from January through June 2014, multilevel, multinomial logistic regression was used to assess interviewer effects on the relationship between these observations and whether a sample unit completes the interview, refuses participation, or is never contacted. This research evaluates these observations as potential indicators of interviewer and fieldwork effects to determine impacts on variance and bias when using the observations to predict nonresponse.
Survey estimates of sleep and exercise are used in a wide array of domains from public health to commercial product development. The 2013 U.S. National Health Interview Survey released both health survey data and paradata on the relative contactability of respondents. Analyses revealed significant positive correlation between extent of noncontact and frequency of physical exercise and sleep quality. Predictive models yielded discrepant output when the survey sample is filtered by extent of noncontact, even after controlling for demographics and chronic conditions. The final analysis examines the number of contact attempts needed to stabilize relative effect sizes of key predictors.
Paradata from audit trails, the record of actions and entries within a computerized questionnaire, can be used for data quality monitoring at the interviewer level. Audit trail data include a record of every key stroke and the time spent between key strokes. Indicators include the average time spent on survey questions, resolving error checks, and the frequency of “don’t know” and “refuse” responses. This presentation will discuss the implementation of an interviewer-level data quality dashboard. Examples provided will show how this data monitoring technique has been used to identify and address interviewer data quality concerns.