ESRA 2017 Programme

Tuesday 18th July      Wednesday 19th July      Thursday 20th July      Friday 21th July     

     ESRA Conference App


Thursday 20th July, 11:00 - 12:30 Room: Q2 AUD3

Online probing: Cognitive interviewing techniques in online surveys and online pretesting 1

Chair Dr Katharina Meitinger (GESIS Leibniz Institute for the Social Sciences )
Coordinator 1Dr Dorothée Behr (GESIS Leibniz Institute for the Social Sciences)
Coordinator 2Dr Lars Kaczmirek (GESIS Leibniz Institute for the Social Sciences)

Session Details

Online probing is a cognitive interviewing technique which can be used in online surveys and is especially useful in cross-cultural research (see Willis 2015 for a research synthesis on cross-cultural cognitive interviewing). The main advantages are: large sample sizes, explanation of response patterns in subpopulations, possible evaluation of prevalence of question problems and themes, higher likelihood of identifying problems during pretesting, and higher anonymity. Online probing is a fully scripted approach and the procedure is highly standardized (Braun et al. 2015; Meitinger & Behr 2016). Automatic on-the-fly analysis and coding of answers during the interview is also possible which can be used to ask automatically issued follow-up questions (for example to detect and reduce item nonresponse, Kaczmirek, Meitinger, Behr, forthcoming).
Online probing has already been applied to reveal diverging or overlapping interpretations and perspectives with regard to a variety of substantive topics, such as gender attitudes (Behr et al. 2013), xenophobia (Braun, Behr, & Kaczmirek 2013), civil disobedience (Behr et al. 2014a), satisfaction with democracy (Behr & Braun 2015), health (Lee et al. forthcoming), and national identity (Meitinger & Behr 2016).
Several methodological studies have addressed the optimal design and implementation of online probing, e.g., on the size of answer boxes (Behr et al. 2014), on sequence effects of multiple probes (Meitinger, Braun, Behr, forthcoming) and on its feasibility for Amazon MTurk (Fowler et al. 2016).
Although online probing has been successfully applied to several substantive and methodological topics, several research gaps remain. For example, due to the large sample size and qualitative nature of the probes, data analysis is rather work-intensive and time-consuming. Also, most of the previous online probing studies focused on Western countries and the majority of studies used the method after official data collection to follow-up on problematic items. Thus, the full potential of the method has not been explored, yet.
For this session, we invite papers on the method of online probing for substantial research and as part of pretests or methods research, and studies that compare online probing with other pretest methods. We especially welcome (1) presentations with a substantive application of online probing and (2) presentations that address some of the methodological challenges and considerations of online probing

Paper Details

1. Finding the Optimal Sample Size for Online Probing: How Many Respondents are Enough?
Dr Katharina Meitinger (GESIS - Leibniz-Institute for the Social Sciences )
Dr Lars Kaczmirek (GESIS - Leibniz-Institute for the Social Sciences )
Dr Michael Braun (GESIS - Leibniz-Institute for the Social Sciences )

Online probing is an innovative approach that applies the verbal probing technique used in cognitive interviewing in web surveys. The implementation of this technique within web surveys offers respondents a higher level of anonymity of their answers in comparison to the laboratory situation of cognitive interviewing. It can easily realize large samples sizes in several countries which enable an evaluation of the prevalence of themes, and can explain the response patterns of specific subpopulations (Braun et al. 2015). However, due to the absence of an interviewer, responses in online probing show a lower response quality than in cognitive interviewing that needs to be compensated by a larger sample size (Meitinger & Behr 2016). But which sample size is sufficient in order a) to arrive at a saturation of results, b) to judge the prevalence of a specific theme, and 3) draw cross-national comparisons? Previous research with regard to sample size in cognitive interviewing already indicates that serious problems that were not detected in small samples were consistently found in larger samples (Blair & Conrad 2011). The same probably holds true for online probing. However, the focus of online probing is in most cases on an assessment of validity and, therefore, it does not directly aim for an error but a theme detection.
Following the approach of Blair & Conrad (2011), the presentation tries to close a research gap by assessing the relationship of sample size and theme detection in online probing. For this purpose, we will draw for different sample sizes repeated random samples and compare the average scores obtained for the number of themes mentioned by respondents. We use data from a web survey conducted in May 2014 with 2,685 respondents from Germany, Great Britain, Mexico, Spain, and the U.S. The respondents came from a nonprobability online panel and were selected by quota for gender, age, and education.
In particular, we want to answer in this presentation the following research questions: 1) Which sample size is necessary to detect the total number of identifiable themes? 2) Which sample size is necessary to detect major themes and with which samples size it is also possible to detect themes which occur less frequently? 3) Can we find, also with small sample sizes, problematic themes that indicate that respondents misunderstood the item? 4) Can we observe cross-national differences with regard to necessary sample sizes? 5) Which sample sizes are necessary to capture cross-national differences in prevalent themes adequately?


2. Evaluating survey questions with different levels of comprehension difficulty using crowdsourced online probes
Dr Ana Slavec (University of Ljubljana)
Dr Vasja Vehovar (University of Ljubljana)

Cognitive psychology research has shown that words less commonly used in daily speech are recognised and processed more slowly than more commonly used words (Howes and Solomon 1951; Broadbent 1967). Thus, using terms that are less familiar to respondents, can lead to comprehensibility problems and decreases response quality. Those can be detected with different pre-testing methods, in particular using cognitive interviewing.
As part of a larger project that compared different survey pre-testing methods on how effective they are in detecting less familiar words that decrease question comprehensibility, we also conducted a small experiment using online probes. Two versions of the same set of 13 questions were compared in online split-ballot cognitive interviews that used the paraphrasing and definition techniques to study how respondents understand wording alternatives with the same meaning but with different comprehensiblity levels. Using the crowdsourcing platform Prolific Academic we recruited 120 respondents that were randomly allocated either to either the version with less common terms or the version with more common terms based on their frequencies in text corpora. In both versions we asked the respondent either to paraphrase the question or to define a certain word in a question, depending on the item.
In most cases, when presented with a less common wording, respondents used its more common alternative to define it or to paraphrase the whole item. Another finding is that there was much more variation in participants’ answers in certain cases of more common wordings, which indicates that these wording alternatives might have less clear and more ambiguous meanings. Finally, we compare the results with two other question evaluation methods, exper reviews and an online panel of 2,966 participants where the sample was split into four groups that responded to four different versions of the same items that differed in their comprehensibility level defined by the amount of less common words.


3. Examining Response Processes underneath Numerical Reports of Subjective Longevity Expectations: Evidence from Simple Probing Questions
Dr Pamela Giustinelli (University of Michigan Survey Research Center)
Dr Sunghee Lee (University of Michigan Survey Research Center)
Miss Michelle Tran (University of Michigan )
Miss Erica Dawson (University of Michigan)
Ms Colleen McClain (University of Michigan)
Dr Dorothée Behr (GESIS)
Dr Katharina Meitinger (GESIS)
Professor Gabor Kezdi (Central European University)

Judgements about the likelihood of future events are an important input for predictions and decisions by citizens, policy makers, and researchers alike. From the early 1990s on, surveys have increasingly measured the subjective expectations of individuals on a 0-100 scale of chance. In the field of Economics, for example, these measures have become widely employed in models, analyses, and predictions of individual and household decisions under uncertainty and have been proven effective tools for documenting and quantifying mechanisms that cannot be unambiguously inferred from choice data alone and for explaining empirical evidence at odds with predictions of economic theory.

A widely measured expectation is subjective life expectancy (henceforth SLE), whereby respondents are asked the percent chance that they will live to be a certain target age or older. SLE has received particular attention among expectations, as it is a critical measure for social insurance programs, private insurance and annuity contracts, consumption and saving decisions, and so on. Because the end date of one’s life is unknown, individuals’ life-cycle decisions, insurance contracts, etc. are based on some prediction of how long individuals are going to live.

While the importance of SLE is uncontroversial, survey questions eliciting probabilistic SLE using a numerical scale of chance likely demand high levels of cognitive --and possibly emotional-- effort from respondents. First, respondents are asked to make a prediction about an event which often refers to a somewhat distant future (depending on the age of the respondent at the time of the survey and the target age used by the question). Second, respondents are asked to think probabilistically. Third, the topic of the question can be perceived as emotional by some respondents. In terms of the survey response process model, major difficulties with these questions are likely to arise in the memory retrieval and judgment steps.

In this project we aim to better understand the cognitive process of respondents when answering probabilistic SLE questions. To this end, we use a probing follow-up which we included in three surveys: 1) a non-probability sample Web survey conducted across the U.S., the U.K., Germany, Spain, and Mexico; 2) a non-probability sample Web survey in the U.S.; and 3) a probability sample telephone survey in the U.S. Specifically, right after answering the SLE question, respondents received an open-ended question asking them to indicate how they arrived at their answer about the SLE. Here we analyze qualitative data from the probing question to examine respondents’ cognitive processes and their sensitivity to respondents’ cultural backgrounds (proxied by respondents’ country/language), as well as to methodological features such as mode.