Deviations and Fraud in Surveys - the Impact of Motivation and Incentives 2 |
|
| Chair | Professor Katrin Auspurg (LMU Munich ) |
| Coordinator 1 | Professor Thomas Hinz (University of Konstanz) |
| Coordinator 2 | Dr Natalja Menold (GESIS) |
| Coordinator 3 | Professor Peter Winker (University of Giessen) |
A psychological motivation theory (Motivation Intensity Theory) was used to operationalize interviewers’ motivation and to analyze its effect on interviewers’ accuracy when documenting interview process (para data). According to the MIT, the effort expended on a task - which determines the accuracy and persistence - can be predicted by (1) the subjective difficulty of the task at hand, (2) the potential motivation (‘‘upper bound of what people would be willing to do to succeed’’) in a given task, and (3) the self-assessed ability to cope with task demands. In our in-field experiment, we collected data on those factors before and after each contact attempt by an interviewer. As factors, which can affect interviewers’ motivation we considered interviewers’ payment and cooperation of respondents. These factors were varied in the experiment. Two payment schemes were used: payment per interview and payment per hour. To obtain a controlled experimental setting for respondents’ cooperation, the potential respondents were instructed to cooperate, to refuse, or to break off the interview. The results show that payment per hour increases interviewers’ motivation and that there are interaction effects between payment scheme and respondents’ cooperation on interviewers’ motivation. Furthermore, interviewers’ motivation is positively related with their accuracy.
A particularly vexing problem for many face-to-face surveys is interviewer falsification, either by interviewers’ falsifying “just some” item responses or skipping items illicitly, or by so-called “curbstoning”. This means entire interviews were fabricated and the target person was never even contacted, the most drastic form of interviewer falsification. In this presentation I will first describe a very elaborate “curbstoning scheme” we discovered during the sixth wave of the Survey of Health Aging and Retirement in Europe (SHARE). The scheme was remarkable in a number of ways: the respective interviewers were initially selected to work for SHARE because of their commendable performance during the SHARE pretests and that of other studies. Next, the scheme was cleverly concocted to make detection through back-check procedures in place at the time very difficult. It involved a fairly small number of interviewers but ended up affecting a fairly large part of the net sample. So this case study suggests a high level of “criminal sophistication” by the involved interviewers. Finally, this type of “Ponzi scheme” affected gross sample units in a geographical cluster, making it a pattern of unit nonresponse of not-missing-at-random. This necessitated specific procedures to avoid sampling bias. Following our insights from this case study, my presentation will then showcase details on how we revised procedures and policies in the data production process of SHARE to prevent the occurrence of curbstoning in future waves. This involved changes in fieldwork monitoring and management, statistical control procedures (to be presented by my colleagues Bergmann & Schuller in the same session) and new protocols regarding collaboration between the various actors (SHARE Central, survey agencies and university teams). I will make recommendations as to how other large-scale f2f surveys could benefit from our experience.
Interviewer fabrication (“fake interviews”) is a problem in all interviewer-conducted surveys and repeatedly come up in the Survey of Health, Ageing and Retirement in Europe (SHARE), as well. While there are many variations and different reasons for interviewers deviating from properly administering the survey, in this project we will only deal with the most extreme deviation, i.e. interviewers’ fabrication of entire interviews.
The main aim of our project is to implement a technical procedure to identify fakes in computer administered survey data. In contrast to previous work that often used only few variables to identify fake interviews, we implement a more complex approach that uses variables from different data sources to build up a comprehensive mechanism in order to identify fake interviews. We use several indicators from CAPI data (size of social networks, avoiding follow up questions, number of proxy interviews, rounding in physical tests, extreme answering, straight-lining, number of “other” answers, number of missings) as well as paradata (interview length, number of interviews per day, number of contact attempts, cooperation rates). We combine these indicators using a multivariate cluster analysis to distinguish two groups of interviewers: a falsifier group and an honest interviewer group.
During the sixth wave of the Survey of Health, Ageing and Retirement in Europe (SHARE) we discovered a very elaborate team of falsifiers who faked a fairly large part of the net sample (see details in abstract submission from our colleague Frederic Malter in the same session). We use these known fakes as a kind of benchmark to check if our script is able to properly identify fake interviews. Thus, in comparison to most of the existing work so far, our study has the advantage to be based on a large dataset including information on actual fakes.
First results show that we are able to identify most of the faked interviews, while at the same time we are able to keep the number of “false alarms” small. Although most of the time we cannot be perfectly sure if an interview has been faked or not, our results can be used to provide survey agencies with a much more informed sample for back checking suspicious interviewers and interviews. By this, we hope that we can substantially improve the quality of our survey data.