ESRA 2025 Program
Get the ESRA 2025 App on iOS or Android:
All time references are in CEST
Satisficing in Self-Completion Modes: Theoretical Understanding, Assessment, Prevention and Consequences for Data Quality |
| Session Organisers |
Dr Daniil Lebedev (GESIS - Leibniz Institute for the Social Sciences, Germany)
Dr May Doušak (University of Ljubljana, Slovenia)
|
| Time | Wednesday 16 July, 13:30 - 15:00 |
| Room |
Ruppert D - 0.24 |
Self-completion surveys, which are increasingly preferred over face-to-face modes, present unique challenges. Rising costs, declining response rates, and interviewer effects make face-to-face surveys less viable. However, self-completion modes (web, mail, or mixed) introduce their own data-quality related challenges. Without an interviewer, respondents face a higher cognitive load, which can lead to satisficing – providing suboptimal answers – especially among those with lower cognitive ability or motivation. This behaviour increases measurement error and lowers data quality.
Survey methodologists have developed various indicators to assess response quality by detecting undesired respondent behaviour, such as straightlining and acquiescence, along with paradata-measured response styles like speeding, multitasking, motivated misreporting, and others. Questions assessing respondents' subjective enjoyment, cognitive effort, and time investment also help identify satisficing propensity. These tools can be used for detection and prevention through immediate feedback or adaptive survey designs based on survey data, paradata, or probing questions.
This session focuses on theoretical advancements in understanding satisficing and response styles in self-completion surveys, whether computer-assisted or paper-based, research on survey data-based indicators, paradata, and probing questions for assessing and preventing satisficing. Developing and implementing satisficing propensity models and tools and evaluating satisficing's impact on data quality in self-completion modes are also key topics. Contributions may address these areas and related research topics.
Keywords: satisficing, response styles, self-completion, web surveys, mail surveys, paradata, data quality indicators, probing questions, experiments, motivation, cognitive effort, cognitive ability, respondent engagement
Keywords: satisficing, response styles, self-completion, web surveys, mail surveys, paradata, data quality indicators, probing questions, experiments, motivation, cognitive effort, cognitive ability, respondent engagement
Papers
Checking the checks: Developing and validating new types of attention checks for self-completion surveys
Dr Marek Muszyński (Institute of Philosophy and Sociology, Polish Academy of Sciences) - Presenting Author
Careless/insufficient effort responding (C/IER) is a major concern of self-report data quality. If left uncorrected, it can significantly distort research conclusions (Arias et al., 2022; DeSimone et al., 2018; Maniaci & Rogge, 2014; Woods, 2006).
Attention checks, such as instructed response items or bogus items, have been employed to detect C/IER. Respondents must select a specific response (e.g., “Please select ‘Strongly agree’”) or avoid agreeing with illogical claims (e.g., “I have been to the moon”) to pass the check.
However, such checks’ validity is low, resulting in too many false positives (attentive participants failing the checks) and false negatives (inattentive participants passing the checks; Gummer et al., 2021). It appears that respondents fail checks due to reasons other than inattentiveness, e.g. purposeful noncompliance or strategic responding (monitoring the surveys for checks).
Recent research suggests that subtle attention checks, which are less conspicuous and more integrated into the survey's context, could improve validity by reducing the likelihood that participants identify them as checks (Kay & Saucier, 2023; Curran & Hauser, 2019).
This project aims to develop and validate a database of such subtle attention checks, employing the idea of frequency items generally agreed upon and infrequency items typically disagreed with (Kay, 2024). The items will be created to align with various research themes such as individual differences, political studies or market research.
Validation will occur through a series of empirical studies analyzing item endorsement rates, participant reactions in cognitive labs (think-aloud protocols), and correlations with “old” attention checks and established C/IER indicators (straightlining, person-fit, response times.) Data will be collected across diverse sample pools differing in survey experience and survey motivation, including online panels, social media recruits, and university students (Shamon & Berning, 2020; Daikeler et al., 2024).
Evaluating Attentiveness Measures in Survey Research: Experimental Insights from the German Internet Panel and Swedish Citizen Panel
Dr Joss Roßmann (GESIS - Leibniz Institute for the Social Sciences) - Presenting Author
Dr Sebastian Lundmark (SOM Institute, University of Gothenburg)
Professor Henning Silber (Survey Research Center, Institute for Social Research, University of Michigan)
Professor Tobias Gummer (GESIS – Leibniz Institute for the Social Sciences and University of Mannheim)
Survey research depends on respondents' cooperation and attentiveness during interviews as inattentive respondents often provide non-optimal responses due to satisficing response strategies. To address varying levels of attention, researchers increasingly implement attentiveness measures, yet there is limited experimental evidence comparing different types of attention checks, particularly regarding their failure rates and the risk of false positives (e.g., Berinsky et al., 2016; Curran & Hauser, 2019). False positives may occur when respondents deliberately disregard instructions, leading to misclassification of attentiveness.
To explore these issues, we conducted experiments in the German Internet Panel (GIP), a probability-based online survey (N=2900), and the non-probability online Swedish Citizen Panel (SCP; N=3800). Data were collected during summer and winter 2022. Respondents were randomly assigned to attentiveness measures, including instructional manipulation checks (IMC), instructed response items (IRI), bogus items, numeric counting tasks, and seriousness checks. These measures varied in complexity and effort required. The SCP study extended the GIP experiment by implementing two attentiveness measures per respondent, one early and one late in the survey.
Results show significant variation in failure rates across attentiveness measures. Despite generally low failure rates, IMC and IRI checks exhibited higher failure rates due to their difficulty, design flaws, and instances of purposeful non-compliance. The findings were consistent across the GIP and SCP.
We conclude that while many attentiveness checks effectively identify inattentive respondents, IMC and IRI checks may overestimate inattentiveness due to design challenges and respondent behavior. Future research should focus on refining attentiveness measures to balance accuracy and respondent engagement, ultimately improving the quality of data collected in web-based surveys.
Measuring Community Attitudes with Polarizing Formats: Can Bipolar Scales Cause Apparent Acquiescence Bias?
Professor Randall K. Thomas (Ipsos Public Affairs) - Presenting Author
Ms Megan A. Hendrich (Ipsos Public Affairs)
Ms Jennifer Durow (Ipsos Public Affairs)
Acquiescence bias occurs when survey respondents select ‘agreeable’ responses instead of responses that more accurately reflect their views. Questions with an agreement response format are believed to be more prone to elicit acquiescence bias than item-specific question types (cf. Krosnick & Presser, 2010). However, experiments that compare agreement and item-specific question types appear to have conflated response polarity (agreement scales are typically bipolar scales with higher means, while item-specific scales are typically unipolar scales with lower means; see Dykema et al., 2021). Our previous studies found no meaningful differences in distributions and validity for agreement and item-specific question types when controlling for scale polarity (e.g., unipolar agreement had similar means as unipolar item-specific items). We expanded the topics examined to replicate and extend the findings. This study employed a 2 X 2 factorial design, comparing unipolar and bipolar response formats and agreement and item-specific question types using ten questions about community issues (quality of schools, convenience of medical care, diversity of restaurants, etc.). We had 5,339 respondents from a well-established probability-based panel (Ipsos’ KnowledgePanel). We randomly assigned them to one of the four conditions (bipolar agreement, unipolar agreement, bipolar item-specific, or unipolar item-specific). Respondents also completed overall evaluations of their community to evaluate the criterion-related validity of the experimental items. Response distributions and means for agreement and item-specific question types were similar when using the same polarity – bipolar formats had higher means than unipolar formats. We also found that there was little difference in the criterion-related validity between the two question types (agreement vs. item-specific). Based on these results, prior findings of higher acquiescence bias in agreement question types failed to account for the typical response patterns that occur with bipolar scales more generally.
