ESRA 2019 Draft Programme at a Glance
Detecting, Explaining and Managing Interviewer Effects in Surveys 1
|Session Organisers|| Dr Daniela Ackermann-Piek (GESIS – Leibniz Institute for the Social Sciences, Mannheim, Germany)
Mr Brad Edwards (Westat)
Dr Jette Schröder (GESIS – Leibniz Institute for the Social Sciences, Mannheim, Germany)
|Time||Tuesday 16th July, 16:00 - 17:00|
How much influence do interviewers have on different aspects of the survey process and how can we better reduce their negative impact on the data quality as well as enhance their positive impact?
Although interviewer effects have been studied over several generations, still, interviewer effects are of high interest on interviewer-administered surveys. Interviewers are involved in nearly all aspects of the data collection process, including the production of sampling frames, acquisition of contact and cooperation with sample units, administration of the survey instrument, and editing and transition of data. Thus, interviewers can cause errors and prevent errors in nearly all aspects of a survey.
However, the detection of interviewer effects is only a first step. Thus, it is of interest to understand why interviewer effects occur. Although there are various studies explaining interviewer effects using multiple sources of data (e.g., paradata, interviewer characteristics, response times, etc.), the results are inconclusive. In addition, it is essential to prevent negative interviewer effects before they occur to ensure that interviewer-administered surveys can produce high-quality data. There are multiple ways to intervene: interviewer training, monitoring during fieldwork, adaptive fieldwork design or switching the survey mode, etc. However, still, relatively little is known about how all these different methods can effectively reduce interviewer error because there is a lack of experimental studies.
We invite researchers to submit papers dealing with aspects of detecting, explaining and preventing interviewer effects in surveys. We are especially interested in quasi-experimental studies on the detection, explanation, and prevention of interviewer error in surveys, and on the development or encouragement of interviewer ability to repair or avert errors. We welcome researchers and practitioners from all disciplines across academic, governmental, private and voluntary sectors to contribute to our session.
Keywords: Interviewer effects, Interviewer training, Interviewer characteristics, Paradata, Total Survey Error
Profiles of Interviewers’ Strategies in Face-to-Face Surveys
Mr Alexandre Pollien (FORS, Swiss Centre for Expertise in Social Sciences, University of Lausanne) - Presenting Author
Dr Jean-Marie Le Goff (Life Course and Inequalities Research Center, University of Lausanne & NCCR LIVEs, Lausanne.)
This paper focuses on the profiles of face-to-face interviewers in the work aimed at obtaining cooperation of the target persons. It examines their activity in two international surveys conducted every two years in Switzerland since 2010: ESS and Mosaich (ISSP). For both of them the standards procedure requires that after 5 unsuccessful contact attempts or a refusal, the conversion phase takes place along with another interviewer. We looked at these first contacts before conversion phase. For each interviewer-survey, we computed vectors of probabilities to have for each of the five attempts a cooperation, a refusal, no contact or another issue. A cluster analysis of these vectors was applied disclosing three types of profiles. These profiles are revealed through different arrangements of likely non-contacts, refusals or responses during the contact process. Our hypothesis is that interviewers’ strategies hold a temporal dimension. A first cluster comprise slower interviewers, scrupulously complying with instructions. We can also notice that they get more non-contacts. A second cluster includes interviewers with a shorter and more chaotic sequence of contacts attempts. Interviewers of this second cluster are facing more refusals in the first attempt. A third cluster includes interviewers succeeding faster to get in touch with the respondents. Interviewers of this third cluster are more seasoned and get the higher response rate. An interviewer study shows that this third cluster is characterized by interviewers with a more realistic attitude combined with a relational orientation leading to tailor their approach. The second cluster combines realism with procedural attitude leading to an offhand behavior. The first cluster combines procedural attitude and relational orientation leading to fastidious strategy of contact.
Three Approaches to Evaluate the Direct Effect of Respondent Characteristics on Intra-Interviewer Correlations
Professor Geert Loosveldt (KU Leuven) - Presenting Author
Dr Caroline Vandenplas (KU Leuven)
A frequently used procedure to evaluate interviewer effects is the calculation of the intra interviewer correlation coefficient (IIC) based on a two-level random intercept model which take into account the nesting of the respondents within the interviewers. The specification of such a basic multi-level model does not really allow to investigate the relationship between certain respondent characteristics and the extent to which these characteristics influence interviewer effects. However it is reasonable to assume that some respondents are more sensitive to interviewer effects and that in some respondent groups the IiCs are higher. In this paper, various alternative specifications of the basic model in which this relationship is explicitly specified will be explored and compared with each other. A first approach is a two-step procedure. In the first step the ICC are calculated within the categories of particular respondent characteristic. In the second step, these ICCs are used as dependent variable in a model with respondent characteristics as independent variable. In a second approach a multilevel model is specified in which the variance of the residual errors at the respondent level and the variance of the random intercept at the interviewer level are calculated conditional on the categories of a respondent. With this specification it is possible the calculate conditional ICCs. In the third approach the ‘mixed effects location-scale model’ is used. This model is an elaboration of the standard two level random intercept model and allows to evaluate the interviewers’ impact not only on the means (standard model) but also on the variability of the respondents’ answers. This means that an additional random effect for the residual variance at the respondent level is specified.
Interviewer Effects for Multiple Aspects of Survey Error
Dr Daniela Ackermann-Piek (GESIS – Leibniz Institute for the Social Sciences) - Presenting Author
To date, there have been various studies describing interviewer effects on different aspects of the survey process. However, very few studies have analyzed interviewer effects on different aspects of a survey simultaneously. In my presentation, I study the relationship between interviewer effects on survey unit nonresponse and interviewer effects on estimates of substantive survey variables. From a survey operational viewpoint it is of great interest, because interviewers who are successful in contacting sample persons and gaining sample persons’ cooperation are not necessarily good at applying standardized survey interview techniques in face-to-face interviews. This is not only relevant for the selection of good interviewers, but also for the design of interviewer trainings.
For my analyses I use data from PIAAC Germany and apply multilevel models and post-estimation tools to account for the nested data structure. I show that there are interviewer effects on survey unit nonresponse as well as on a large number of estimates of substantive survey variables. The results of the combined analyses of interviewer effects on multiple aspects of survey error show a significant relationship between the predicted probabilities of successfully making contact with sample persons and the predicted means of the majority of the substantive survey variables at the interviewer level. However, I find no significant relationship between the predicted probabilities of successfully gaining sample persons’ cooperation and the predicted means of the majority of the substantive survey variables at the interviewer level. Although, I can draw on a very rich data source of interviewer characteristics I find no clear pattern of the explanatory power of interviewer effects on survey unit nonresponse and on the estimates of the substantive survey variables under research. However, I find interviewer characteristics that explain interviewer variance in the estimates of some substantive variables.
Modelling Group-Specific Interviewer Effects on Nonresponse using Separate Coding for Random Slopes in Multilevel Models
Ms Jessica Herzing (University of Lausanne) - Presenting Author
Mrs Annelies Blom (University of Mannheim)
Mr Bart Meuleman (University of Leuven)
While there is ample evidence of interviewers affecting nonresponse and some evidence regarding the factors explaining overall interviewer effects, the literature is sparse on how interviewers differentially affect specific groups of respondents despite of the importance of this in terms of nonresponse bias. A reason for the sparse literature on interviewer effects on nonresponse bias may be limitations of standard use of multilevel models. We demonstrate how an alternative parametrization of the random components in multilevel models, so-called separate coding, delivers insights into differential interviewer effects on specific respondent groups. A multilevel model with separate coding of random coefficients makes it not only possible to estimate how the size of interviewer effects varies across different types of respondents, but also offers possibilities to investigate how interviewer characteristics affect the groups differentially.
Using the example of nonresponse during the recruitment of a probability-based online panel separately for persons with and without prior internet access (data used from the German Internet Panel), we detect that the size of the interviewer effect differs between the two respondent groups. While we discover no interviewer effects on nonresponse for persons without internet access (offliners), we find sizable interviewer effects for persons with internet access (onliners). In addition, we identify interviewer characteristics that explain this group-specific nonresponse. Our results demonstrate that the implementation of interviewer-related fieldwork strategies might help to increase response rates among onliners, as for onliners the interviewer effect size was relatively large compared to the interviewer effect size for offliners. Clearly, surveys with large imbalances among respondent groups gain from an investigation of the variation of interviewer effects. By considering group-specific interviewer effect size one is more effective when implementing or adjusting interviewer-related response-enhancement strategies and thus, one might mitigate nonresponse bias more effectively.