ESRA logo
Tuesday 14th July      Wednesday 15th July      Thursday 16th July      Friday 17th July     




Friday 17th July, 09:00 - 10:30 Room: O-206


Potential and Challenges of Cognitive Interviewing and Online Probing in a Cross-National Context

Convenor Mr Andrew Johnson (Ipsos MORI )
Coordinator 1Ms Katarina Meitinger (GESIS)

Session Details

A key consideration of cross-national surveys is that their questions possess functional equivalence, allowing for comparative analysis across people of differing cultural and linguistic backgrounds.

Cognitive interviewing plays an invaluable role in uncovering problems of non-equivalence and offers at the same time the possibility to identify the causes of non-equivalence. However, its application in a cross-national setting faces unique challenges, such as the need to have multiple interviewing teams, often speaking different languages and possessing varying levels of interviewing experience and training. Teams may also have more or less experience of analyzing cognitive interviewing findings, and may be better or worse at making the sometimes fine analytical distinctions between problems of functional equivalence, translation error or poor source question design.

Recently, the supplemental method of online probing has been developed that implements probing techniques in cross-national web surveys. It allows for a relatively inexpensive increase in sample size, probe standardization and a quantification of results (Behr et al. 2012) but faces its own challenges, such as mismatching answers or probe non-response.

For this session, we invite papers that address the topic of equivalence testing by making use of either cognitive interviewing or online probing. What can the different methods achieve and where do they face challenges? And most importantly: How can these challenges be resolved?

Papers are welcome on both substantive findings and on methodological challenges and considerations. This session also invites papers that can provide a helpful contribution to understanding how best to achieve a uniformity of approach in cross-country cognitive interviewing, with a focus on, but not restricted to:
• Interviewer training and briefing
• Interview protocols
• Collection, organization and analysis of cognitive interviewing data

Co-organizers: Tom Frere-Smith, Ipsos and Dorothée Behr, GESIS

Paper Details

1. The Effect of Providing Think-Aloud Examples and Practice on Cognitive Interviewing in Nepal
Ms Jennifer Kelley (University of Michigan)
Ms Kristen Cibelli (University of Michigan)
Dr Ting Yan (Westat)

Cognitive interviewing is most effective when respondents verbalize their thought process in response to the survey questions and interviewer probes. This process may be unfamiliar and difficult for some respondents, particularly in contexts outside of the United States and Europe where there is little survey tradition. Little guidance exists in the literature on how to apply cognitive interviewing in such contexts. This paper examines the effect of providing respondents with examples and practice think-alouds before the main cognitive interview in an experiment carried out in rural Nepal where experience with surveys and cognitive interviewing is rare.


2. Collecting, organising and analysing data from cross-national cognitive interviewing - a review of approaches used and suggestions for improvements
Miss Sally Widdop (Ipsos MORI)
Mr Tom Frere-smith (Ipsos MORI)
Ms Sara Grant-vest (Ipsos MORI)

Cognitive interviewing allows us to explore how respondents interpret survey questions and assess whether they are understood as intended. It is increasingly used to pre-test questions developed for cross-national surveys to ensure equivalence of meaning in different languages and cultures. In this context, achieving standardisation in the way that interviews are conducted and the data are organised and analysed is an important but challenging task. This paper will review the approaches that have been used to collect, organise and analyse data from cognitively testing questions for several European surveys and suggest how improvements could be made.



3. Challenges of Cognitive Interviewing in a Cross-National Context
Dr Rossalina Latcheva (European Union Agency for Fundamental Rights)

The presentation addresses potential challenges and drawbacks of conducting cognitive interviews in a multi-national, multi-cultural contexts by identifying both issues of comparability and conceptual equivalence and the challenges that researchers face when trying to apply a standardised uniform approach, including a systematic comparative analysis across diverse settings.
The presentation is based on a cross-country/ cross-group database of 280 cognitive interviews collected in the framework of an extensive cognitive pre-test study, conducted in 2014 in eight EU Member States to inform the development of the second European Union Minorities and Discrimination Survey, which the European Union


4. Cognitive probing in a cross-national survey: What can we learn about measurement context?
Dr Sunghee Lee (Michigan Program in Survey Methodology)
Ms Colleen Mcclain (Michigan Program in Survey Methodology)
Dr Dorothee Behr (GESIS - Leibniz-Institute for the Social Sciences)

Questions asking respondents to report self-rated health and subjective life expectancy are commonly included in cross-national surveys. While both items have been shown to be strong predictors of actual mortality and health, each encompasses distinct and complex tasks of comprehension, retrieval, judgment, and mapping. Thus, investigating cognitive processes by which respondents arrive at their answers may be key to understanding the utility of these questions in general. We present results from cognitive probing items implemented in a cross-national Internet survey, fielded in five countries and three languages, to examine whether cultural context frames and influences response.


5. Insights from Online Probing: Can the General Pride Item Serve as a Measure for Nationalism or Patriotism in a Cross-national Context?
Ms Katharina Meitinger (GESIS Leibniz Institute for the Social Sciences)

Cross-national studies investigating elements of national identity frequently use the item “How proud are you of being German/American/British?“ as a substitute to measure either patriotism or nationalism. However, it is questionable that this item is a valid measure for these complex constructs because respondents in different countries could associate various constructs with it.

The presentation will scrutinize the appropriateness of the general pride item as a measure of nationalism and/or patriotism. It will combine quantitative results from the ISSP 2013 with online probing results from a web survey conducted with 2,685 respondents in five countries.