Tuesday 16th July
Wednesday 17th July
Thursday 18th July
Friday 19th July
Download the conference book
Use of Paradata for Production Survey Management |
|
| Convenor | Dr Katherine Mcgonagle (Institute for Social Research, University of Michigan) |
| Coordinator 1 | Dr Jason Fields (U.S. Census Bureau) |
This session addresses the rise in demand for tools that capitalize on the increasing availability of paradata. Managing surveys efficiently and continuing to collect high quality data amidst declining response rates has further increased the need for rapidly assessing survey instrument performance. This need has led to many innovative approaches for managing surveys in the field, and has given rise to new tools. From cost and response evaluations that facilitate responsive design, to interviewer training evaluations and performance management, and near real-time evaluations of data quality and estimates, the tools available for performing these tasks is expanding. Paradata encompasses a broad spectrum of realized and potential data sources. The methods for presenting these data for use in survey management are a significant and growing area of interest. This session seeks to explore the current and developing areas of paradata use and dissemination for survey management. The session will highlight maximizing the use and usability of paradata in production survey management, provide a forum for discussion of recommendations, and identify gaps between concepts and operationalization. Papers are invited which consider these topics from a variety of perspectives and may include (but are not limited to) the following topics:
- Paradata capture and presentation
- Dashboards
- Tracking systems
- Performance management
- Responsive design systems and tools
- Data quality
- Data estimates
All papers are expected to develop general themes from their experiences, rather than focus on issues solely relevant to their own projects. Especially welcome are joint papers that explore the issues from two (or more) sides of a collaboration, as well as those that bridge traditional boundaries in survey management.
This paper will describe the tools used to develop a dashboard system based on paradata collected throughout the survey process on a longitudinal, panel study. While 2012 is the eleventh wave of data collection of the Health and Retirement Study (HRS), 2006 marked the first wave that the majority of the field work was completed in person and included various sub-components, such as the collection of physical measures and biomarkers. We will provide examples of a dynamic system that allows for the comparisons of effort and summary statistics--such as response rates, number of contacts, resistance rate, consent on multiple components--across multiple waves of data collection and across subgroups within and between waves. Two primary goals of the system were to provide managers with the information needed to monitor data collection on a daily basis and to develop a flexible, dynamic system that could be modified as project needs change throughout the data collection period. The system utilizes SAS/SQL to aggregate the data in one format and Excel to create a streamlined "dashboard" view. Key performance indicators are available on a daily basis along with trend charts with comparisons to similar points in previous waves of data collection. A secondary goal of this project was to provide a simple way to access a number of components, rather than requiring multiple reports drawn from different databases. The current system is hosted on-line, within an existing web-monitoring site, thus is easily accessible to managers regardless of location.
The Survey of Income and Program Participation (SIPP) is a nationally representative, longitudinal survey administered by the U.S. Census Bureau that collects month-level details over a four-year panel. Census has undertaken the development of a replacement survey that moves from three-times-a-year to annual interviewing. Census is also developing several paradata-based tools for survey management. A keystone is the Unified Tracking System (UTS), a paradata access and presentation tool that incorporates data from survey-management systems, cost and billing systems, contact history instruments, and content elements from the data collections themselves. Accompanying UTS, a dashboard will summarize those measures into visualizations, which present summary information in four areas: Progress, Productivity, Quality, and Cost. Additionally, SIPP is implementing electronic interviewer certification evaluations to return results to field supervisors to improve interviewer management during data collection periods. During development of the new instrument, we used paradata from keystroke files (i.e., audit trails), interviewer information, and results from training certification tests to improve instrument flow and interviewer training. This paper integrates the current and expected sources of paradata and tools being developed to discuss how they could be used to inform survey management, field training, performance, quality, and cost monitoring. We draw specifics from our plans for managing the 2014 SIPP panel, extend generalizations to how these types of paradata and tools can be used, and consider what types of information would be high-priority additions to the information currently available.
Statistics Canada has developed a number of automated tools and processes to monitor interviewer performance making use of available paradata in an effort to maintain high levels of quality in the survey data collection process. This paper will provide an overview of interviewer monitoring tools used directly in Blaise based CATI and CAPI interviewing as well as others such as our Pace of Interview application used to identify aberrant interviews. It will also outline how monitoring efforts help in maintaining high quality levels in collected information. The paper touches on current paradata availability from our web-based interviewing mode (Electronic Questionnaire) and paradata gaps that need to be addressed in order to maintain similar degrees of interviewer monitoring as the agency moves towards a collection model with Electronic Questionnaire as the primary tool used in all respondent self-completed and interviewer completed surveys.
On the National Survey of Family Growth, extensive use of paradata from the sample management system, organized into a production monitoring dashboard, has allowed for the conceptualization and implementation of design features that respond to survey conditions in real time, so called "responsive designs" (Groves and Heeringa, 2006). The production dashboard uses information about data collection field work to help guide alterations in field protocols during survey data collection to achieve greater efficiency and improvements in data quality.
In addition to the paradata collected by the sample management system, paradata from audit trails, the record of actions and entries within the CAPI questionnaire, are used for data quality monitoring at the interviewer level. Audit trail data include a record of every key stroke and the time spent between key strokes. Using these data, a data quality dashboard was created in order to monitor data quality at the interviewer level. Indicators include the average time spent on survey questions, the frequency of using help screens, recording remarks, checking errors, backing up in the interview, and the frequency of "don't know" and "refuse" responses.
This presentation will discuss design and management strategies for using paradata for responsive design in survey operations to improve survey outcomes as well as discuss the implementation of the interviewer-level data quality dashboard. Examples provided will show how this data monitoring technique has been used to identify and address interviewer data quality concerns.