ESRA 2017 Programme

Tuesday 18th July      Wednesday 19th July      Thursday 20th July      Friday 21th July     

     ESRA Conference App

Tuesday 18th July, 16:00 - 17:30 Room: Q4 ANF1

Researching Sensitive Topics: Improving Theory and Survey Design 4

Chair Dr Ivar Krumpal (University of Leipzig )
Coordinator 1Professor Ben Jann (University of Bern)
Coordinator 2Professor Mark Trappmann (IAB Nürnberg)

Session Details

Social desirability bias is a problem in surveys collecting data on private issues, deviant behavior or unsocial opinions (e.g. sex, health, income, illicit drug use, tax evasion or xenophobia) as soon as the respondents’ true scores differ from social norms. Asking sensitive questions poses a dilemma to survey participants. On the one hand, politeness norms may oblige the respondent to be helpful and cooperative and self-report the sensitive personal information truthfully. On the other hand, the respondent may not trust in his or her data protection and may fear negative consequences from self-reporting norm-violating behavior or opinions. Cumulative empirical evidence shows that in the context of surveying sensitive issues respondents often engage in self-protective behavior, i.e. they either give socially desirable answers or they refuse to answer at all. Such systematic misreporting or nonresponse leads to biased estimates and poor data quality of the entire survey study. Specific data collection approaches were proposed to increase respondents’ cooperation and improve validity of self-reports in sensitive surveys.

This session is about deepening our knowledge of the data generation process and advancing the theoretical basis of the ongoing debate about establishing best practices and designs for surveying sensitive topics. We invite submissions that deal with these problems and/or present potential solutions. In particular, we are interested in studies that (1) reason about the psychological processes and social interactions between the actors that are involved in the collection of the sensitive data; (2) present current empirical research focusing on ‘question-and-answer’ based (e.g. randomized response techniques, factorial surveys), non-reactive (e.g. record linkage approaches, field experiments or administrative data usage) or mixed methods of data collection (e.g. big data analyses in combination with classical survey approaches) focusing on the problem of social desirability; (3) deal with statistical procedures to analyze data generated with special data collection methods; (4) explore the possibilities and limits of integrating new and innovative data collection approaches for sensitive issues in well-established, large-scale population surveys taking into account problems of research ethics and data protection.

Paper Details

1. Using the Crosswise Model in the field of higher education and science research
Dr David Johann (German Centre for Higher Education Research and Science Studies)
Dr Kathrin Thomas (City University of London)

In survey research, social desirability bias (SDB) describes non-sampling error in survey data due to respondents’ tendency to provide answers that present them in a positive way to others, because they think the reported attitudes or behaviours are socially more acceptable. SDB is particularly a problem for questions asking about sensitive topics. In higher education and science research we may expect socially desirable response patterns on topics such as academic misconduct, but also about the demographic background of staff and students (e.g., ethnicity, gender), the relationship between supervisors and supervisees, or substance abuse among staff in stress situations.
In order to reduce SDB in survey responses, survey methodologists continuously develop techniques to improve respondents’ anonymity and confidentially when asking questions. One promising approach is the so-called Crosswise Model (e.g., Yu et al. 2008; Krumpal et al. 2015), which has proved successful for questions about substance abuse among athletes, plagiarism, and tax fraud (Coums et al. 2011; Jann et al. 2012; Höglinger et al. 2014; Shamsipour et al. 2014). A comprehensive evaluation of the applicability of the model, especially in the area of higher education and science study, is still open to academic research.
This paper investigates different questions in this area and evaluates the respondents’ cognitive understanding of the Crosswise Model. Systematic analysis their respondents reveals how far respondents feel that their privacy and anonymity is protected by this question format and whether some groups are more suspicious than other that this is an unusual question format. The analysis is based on systematically conducted cognitive interviews about the method among respondents at different educational levels in Germany and the United Kingdom.

2. Evaluating the Crosswise Model RRT and the Item Count Technique for Surveying Sensitive Topics: An Approach that Detects False Positives
Dr Marc Hoeglinger (University of Bern)
Professor Andreas Diekmann (ETH Zurich)

The valid measurement of sensitive issues such as norm-violations or stigmatizing traits through self-reports in surveys is often problematic. Special sensitive question techniques such as the Randomized Response Technique (RRT, Warner 1965) and among its variants the recent Crosswise Model RRT (CM, Yu, Tian and Tang 2008) should generate more honest answers by providing full response privacy. Also, the Item-Count Technique (Miller 1984), which is based on a similar logic, is increasingly employed in research on sensitive topics. However, these methods sometimes do not work as expected and evaluating whether particular implementations actually improve data validity is essential before their application in substantive surveys. To this end, studies so far mostly compared prevalence estimates of sensitive traits and behaviors of different techniques. Assuming that respondents only falsely deny but never falsely admit a sensitive trait or behavior, higher prevalence estimates were interpreted as more valid estimates. However, if false positives occur, i.e., if respondents are misclassified as bearing a sensitive trait although they actually do not, conclusions drawn under this assumption are likely wrong. False positives occurred knowingly in one variant of the Crosswise Model RRT. Other RRT variants might be affected too, but studies so far have largely neglected this possibility and did not test for it. We show an evaluation approach that detects systematic false positives without the need of a validation criterion – which is often unavailable. Results from an application in a survey on “Organ donation and health” show, that the Crosswise Model RRT produced false positives to a non-ignorable extent. This finding is in line with a previous validation study, but has not been revealed by several comparative evaluation studies that did not consider false positives. We will test for false positive responses with an extended replication of the survey including various RRT and the Item-Count Technique.

3. A Unified Randomized Response Questioning Design for Quantitative Variables
Dr Andreas Quatember (Associate Professor)

The idea of reducing nonresponse and untruthful answering by applying an alternative questioning design that protects the respondent’s privacy was originally presented for binary variables. In the first effort to apply such a method also to quantitative sensitive variables (such as the number of abortions per unit in a given population of women) it was suggested to ask a respondent either with a certain probability for the true value of the interesting variable or with the remaining probability for the true value of a completely innocuous variable unrelated to the sensitive one. This second variable should have a similar range of possible values as the one under study to guarantee the privacy of the respondents within such a questioning design. After this first application, other techniques have been developed for the same purpose. The prerequisite for the possibility of applying such alternative questioning designs in statistical surveys is basic research and the elaboration of an applicable theory.
In the talk, a generalized framework for the estimation of the total of a sensitive variable is presented. It combines, as special cases, the most important techniques already published like the one from above as well as the direct questioning design under one theoretical umbrella. Furthermore, this unified approach at the same time encompasses all the other combinations not yet published. The execution of the statistical properties of this general framework for general probability sampling, being of the greatest importance for the practical application of the procedure, is based on unbiased “imputations” for the true values of the sensitive variables and the Horvitz-Thompson approach from statistical sampling theory.

4. Should we use indirect techniques for surveying sensitive topics? A meta-analysis of the Randomized Response Technique and the Item Count Technique
Dr Marc Höglinger (University of Bern)
Professor Ben Jann (University of Bern)

Indirect techniques for surveying sensitive topics have been around since the 1960ies when Warner (1965) introduced the Randomized Response Technique. Since then, many different RRT variants as well as other approaches such as the Item Count Technique (ICT, Miller 1984) have been proposed. Along with the theoretical development of these techniques numerous methodological studies have been conducted to assess whether the indirect techniques actually generate more valid measurements than standard direct questioning. Results from these studies appear mixed, but a meta-analysis published in 2005 (Lensvelt-Mulder et al.) found that, on average, data generated by the RRT is indeed somewhat more valid than data from direct questioning. Since 2005, further RRT variants, such as the Crosswise Model RRT, have been introduced, applications of the Item Count Technique became popular, and a series of new and potentially more powerful validation studies have been conducted. We therefore carried out a new meta-analysis of RRT and ICT validation studies that covers publications in scientific journals indexed in major databases up to 2016. We present results on average effect sizes of particular techniques and variants and investigate whether results depend on study characteristics such as the employed validation design. In contrast to the 2005 meta-analysis we also investigate whether results are affected by publication bias.