ESRA logo

ESRA 2023 Preliminary Glance Program

All time references are in CEST

Sensitive Questions and Social Desirability: Theory and Methods 2

Session Organisers Dr Felix Wolter (University of Konstanz)
Professor Jochen Mayerl (Chemnitz University of Technology)
Dr Henrik K. Andersen (Chemnitz University of Technology)
Dr Justus Junkermann (University of Leipzig)
TimeWednesday 19 July, 16:00 - 17:30
Room U6-21

Misreporting to sensitive survey questions is an age-old and notorious problem in survey methodology. Empirical evidence has shown that survey respondents tend to engage in self-protective behavior (e.g., by not answering truthfully) when it comes to questions on private issues, deviant behavior, or attitudes that do not conform with social norms (e.g. sex, health, income, illicit drug use, tax evasion or xenophobia). This leads to biased estimates and poor data quality. Although a large body of methodological literature addressing these issues does exist, many questions remain. Further, recent literature has called special questioning techniques such as the crosswise model or other randomized response techniques into question, for instance because they have been accused of generating false-positive bias.

This session aims at deepening our theoretical and practical knowledge with respect to why and when response- or social desirability bias occurs and how best to avoid it. In particular, we are interested in studies that (1) focus on explaining the psychological mechanisms that lead to misreporting on sensitive survey questions; (2) present conceptual or empirical research on special questioning techniques (e.g., randomized response and item count techniques, factorial surveys) aimed at mitigating response bias; (3) deal with statistical procedures to analyze data generated with special data collection methods; (4) address other topics related to social desirability and/or response bias, e.g., mixed methods of data collection, using “big data” and/or record linkage techniques, but also research ethics and data protection issues.

Keywords: Sensitive questions, social desirability, response bias, data collection techniques

Prevalence Estimates for COVID-19-Related Health Behaviors Based on the Cheating Detection Triangular Model

Professor Pier Francesco Perri (University of Calabria) - Presenting Author
Dr Adrian Hoffmann (Heinrich Heine University Düsseldorf)
Dr Shu-Hui Hsieh (Research Center for Humanities and Social Sciences, Academia Sinica )

Behavioral recommendations, restrictions, and protective measures issued by governments and public health agencies to combat the COVID-19 pandemic have had a significant impact on the daily behavior of people around the world. To evaluate the effectiveness of such interventions, it is of utmost importance to obtain valid prevalence estimates for their adherence. However, when asked about COVID-19-related behaviors with conventional direct questions (DQ), some respondents may prefer to protect their privacy rather than answer truthfully; this may lead to nonresponse, or underreporting of socially undesirable and overreporting of socially desirable behaviors. Indirect questioning techniques, such as the recently proposed Cheating Detection Triangular Model (CDTRM), aim at controlling for nonresponse and social desirability bias by guaranteeing the confidentiality of individual answers. The CDTRM has the additional advantage of providing lower and upper bounds for the estimated prevalence, taking participants’ instruction nonadherence into account. Using the probability-based online panel of Taiwanese adults surveyed by the Centre for Survey Research (Academia Sinica) between October and November 2022, we experimentally compared prevalence estimates obtained in a CDTRM (n = 1,144) and a conventional DQ condition (n = 573) for three COVID-19-related behaviors with different levels of sensitivity. For a nonsensitive behavior (having received a COVID-19 vaccine), prevalence estimates were expectedly comparable between experimental conditions (CDTRM: 87-93%, DQ: 92%) and consistent with official vaccination data in Taiwan (95%). For a behavior with low sensitivity (having been tested positive for COVID-19; CDTRM: 42-56%, DQ: 38%), and especially for a highly sensitive behavior (having concealed a positive COVID-19 test result from others; CDTRM: 18-36%, DQ: 5%), estimates in the CDTRM condition were substantially higher, thus presumably less distorted, and more valid, than in the DQ condition. Our results, therefore, suggest the CDTRM as a promising method

Who Should have Access to Artificial Reproduction? Comparing Results from Factorial Survey Experiments and Direct Survey Questions

Ms Larissa Fritsch (University of Zurich) - Presenting Author
Dr Sandra Gilgen (University of Zurich)
Ms Maila Mertens (University of Zurich)
Professor Jörg Roessel (University of Zurich)

The last decades have seen enormous innovations in artificial reproduction technologies (ART), such as sperm donation or IVF. However, this has led to controversial public debates on who should have access to and receive financial support for the use of these new technologies. The main point of controversy is whether access should be limited to (married) couples, and whether same-sex couples should be eligible. Empirical insights on this social justice issue feeding into existing inequalities regarding reproductive self-determination are scarce and much needed.
Using two methods to measure the degree of support for access to ART, we aim to make both a substantive and methodological contribution. One half of the respondents – in a nationally representative survey that will be conducted in Switzerland in early 2023 – are shown two factorial survey experiments on normative and financial support for ART while the other half are asked direct questions measuring their support.
Implementing the survey experiments allows us to measure the independent effects of multiple treatments (e.g., whether the described people are single or coupled; hetero- or homosexual; already have children or not; their age and social background) influencing people’s assessments regarding access to ART. We hypothesize that these assessments, essential to questions of reproductive justice, are shaped by the social structural position of respondents, as well as their broader gender role as well as heteronormativity attitudes.
Comparing the results from the experiments with answers to the direct questions regarding some of the subgroups allows us to assess the magnitude of the resulting discrepancy. This is especially valuable in cases pertaining to controversial topics such as ART support, or more broadly reproductive justice, that are particularly subject to social desirability bias.

Democrats Under-report and Republicans Over-report: What this Means for Measuring Public Opinion on Controversial Political Topics

Dr David Randahl (Uppsala University) - Presenting Author
Dr Sophia Hatz (Uppsala University)

Researchers seeking to measure and understand public opinion on controversial topics face a fundamental problem: survey respondents tend to report ‘socially desirable’ opinions and values, while concealing their true attitudes. In the aggregate, this translates into systematically biased measures of public opinion: undesirable views are undercounted. We make an important contribution by demonstrating that social desirability bias can cut in opposite directions across population subgroups. For example, what is social desirable to Republicans may be socially undesirable to Democrats, leading Republicans to over-report and Democrats to under-report. We use list experiments embedded in public opinion surveys to demonstrate that subgroups defined by political affiliation and race misreport sensitive opinions in opposite directions. We show that when this is the case, the common list experiment survey technique does not return accurate measures of sensitive topics. The findings have important implications for political science and survey research: public opinion may not be as polarised along subgroup-lines as current survey methodologies lead us to think.

Mesuring Corruption using Randomised Item Response Theory

Dr Felipe Torres Raposo (University College London) - Presenting Author
Dr Raymond Duch (University of Oxford)
Dr Ahra Wu (Princeton University)

Measuring the prevalence of corrupt behavior using surveys have been a challenge due to social desirability and non-responses biases. A whole number of indirect questioning survey techniques has been designed and conducted to elicit truthful answers to sensitive issue. However, most of the popular used techniques in the field of political science and economics are restrained to measure group-level prevalence of the sensitive behavior. In order to circumvent this limitation, we conducted a Randomized Item Count Response Technique (n = 6058 and n = 3692) that allowed us to estimate individual-level experiences of corrupt behaviour at the local level.

“Next time, I will definitely buy an electric vehicle!”: Over-reporting of voluntary environmental behaviors in Switzerland and the United States

Dr Keith Smith (ETH Zurich) - Presenting Author
Mr Florian Lichtin (ETH Zurich)
Ms Franziska Quoss (ETH Zurich)
Ms Sarah Gomm (ETH Zurich)

Item count techniques (ICT) survey embedded experiments are commonly implemented investigate social desirability biases in survey responses (Imai 2011). Typically, ICT has been adopted to identify embarrassing, or even illegal, behaviors or attitudes (Creighton et al. 2019), yet social desirability is normative, and relative to the respondent’s social identities. That is, what is deviant for social identity (such as an environmental activist eating a McDonald’s hamburger), may be normative for another. And people with certain social identities may be more likely to be untruthful in their responses. Using ICT experiments across three surveys in Switzerland and the United States, we investigate how responding to voluntary engagement in pro-environmental behaviors (e.g. purchasing an electric vehicle, installing household solar panels, avoiding short-haul flights) varies by individual characteristics (e.g. environmental attitudes, political orientations, socio-demographic characteristics). These analyses contribute to emerging research into over-reporting of voluntary environmental behaviors, the role of social identities shaping environmental behavioral intentions and survey embedded experimental designs.