ESRA 2017 Programme

Tuesday 18th July      Wednesday 19th July      Thursday 20th July      Friday 21th July     

     ESRA Conference App


Thursday 20th July, 16:00 - 17:30 Room: Q2 AUD3

Online probing: Cognitive interviewing techniques in online surveys and online pretesting 3

Chair Dr Katharina Meitinger (GESIS Leibniz Institute for the Social Sciences )
Coordinator 1Dr Dorothée Behr (GESIS Leibniz Institute for the Social Sciences)
Coordinator 2Dr Lars Kaczmirek (GESIS Leibniz Institute for the Social Sciences)

Session Details

Online probing is a cognitive interviewing technique which can be used in online surveys and is especially useful in cross-cultural research (see Willis 2015 for a research synthesis on cross-cultural cognitive interviewing). The main advantages are: large sample sizes, explanation of response patterns in subpopulations, possible evaluation of prevalence of question problems and themes, higher likelihood of identifying problems during pretesting, and higher anonymity. Online probing is a fully scripted approach and the procedure is highly standardized (Braun et al. 2015; Meitinger & Behr 2016). Automatic on-the-fly analysis and coding of answers during the interview is also possible which can be used to ask automatically issued follow-up questions (for example to detect and reduce item nonresponse, Kaczmirek, Meitinger, Behr, forthcoming).
Online probing has already been applied to reveal diverging or overlapping interpretations and perspectives with regard to a variety of substantive topics, such as gender attitudes (Behr et al. 2013), xenophobia (Braun, Behr, & Kaczmirek 2013), civil disobedience (Behr et al. 2014a), satisfaction with democracy (Behr & Braun 2015), health (Lee et al. forthcoming), and national identity (Meitinger & Behr 2016).
Several methodological studies have addressed the optimal design and implementation of online probing, e.g., on the size of answer boxes (Behr et al. 2014), on sequence effects of multiple probes (Meitinger, Braun, Behr, forthcoming) and on its feasibility for Amazon MTurk (Fowler et al. 2016).
Although online probing has been successfully applied to several substantive and methodological topics, several research gaps remain. For example, due to the large sample size and qualitative nature of the probes, data analysis is rather work-intensive and time-consuming. Also, most of the previous online probing studies focused on Western countries and the majority of studies used the method after official data collection to follow-up on problematic items. Thus, the full potential of the method has not been explored, yet.
For this session, we invite papers on the method of online probing for substantial research and as part of pretests or methods research, and studies that compare online probing with other pretest methods. We especially welcome (1) presentations with a substantive application of online probing and (2) presentations that address some of the methodological challenges and considerations of online probing

Paper Details

1. Measuring attitudes towards Social Justice and Fairness across Europe: Assessing cross-cultural validity of related concepts, items and response scales
Dr Michael Weinhardt (Bielefeld University)
Mrs Jule Adriaans (Bielefeld University)

Over the last few decades, European societies have witnessed unprecedented increases in social inequalities. The impact of rising equalities will depend on the extent to which such inequalities are perceived and evaluated as unjust by the people. To assess and compare the impact of such justice evaluations, it will be necessary to collect data on social justice attitudes that are comparable across the countries of Europe. This study is part of the works accompanying the development of the rotating module on “Social Justice and Fairness in Europe” in round 9 of the European Social Survey (ESS) and will investigate the issue of cross-cultural equivalence of the measurement of social justice attitudes. We employ a mixed methods approach, combining focus group discussions, online probing and cross-cultural cognitive interviewing to tackle the following questions: To what extent are the concepts of social justice and fairness understood equally in different countries and languages? What differences do exist and how large are they? First, focus group discussions are used to clarify the the understanding of the concepts of justice and fairness in different countries. Respondents from different cultural backgrounds that will feature in round 9 of the ESS are invited to discuss their understanding of social justice and fairness to pinpoint similarities and differences at a conceptual level. Second, cross-cultural cognitive interviewing (CCCI) is used to test items intended to measure social justice attitudes in different languages. CCCI is a version of cognitive interviewing used for assessing the cognitive processes behind the response process in personal interviews with a small number of respondents in a cross-cultural setting. Third, online probing (OP) is used; a relatively new tool, combining the advantages of CCCI for the assessment of survey questions and that of an online survey, achieving a greater sample size and broader coverage of issues. Questions on social justice attitudes are administered in an online survey, followed by comprehension and category-selection probes. Initially, in both CCCI and OP, we present existing items of the ESS tapping the concepts of justice to a convenience sample of university students and employees from an international background covering a variety of European countries. Information gathered during this process will feed into formulation of new items which will then be presented to the same pool of respondents again for fine-tuning of the final items. The study will focus on German, English, Polish and Russian as these languages cover both Eastern and Western Europe and are likely to be represented in the pool of students and employees we plan to recruit for the study. The combined information from the three methods employed will provide us with a broad but detailed picture of the converging and diverging meanings and understandings of the concepts of social justice and fairness and the related items intended to measure these. The discussion of the results will focus on how problems of lacking congruence of concepts in different countries may be tackled by developing culturally equivalent items for inclusion in ESS9.


2. “Individual solutions” for the division of labor between men and women
Professor Michael Braun (GESIS - Leibniz-Institute for the Social Sciences)
Dr Katharina Meitinger (GESIS - Leibniz-Institute for the Social Sciences)
Dr Dorothée Behr (GESIS - Leibniz-Institute for the Social Sciences)

Web probing, that is, implementing verbal probing techniques traditionally used in cognitive interviewing in online surveys, is a method to substitute for or complement quantitative techniques to establish functional equivalence of item batteries in cross-cultural research. It can also been applied to single questions or items. It is particularly useful to evaluate the appropriateness of newly developed questions and response alternatives.
This approach is illustrated with one question from the 2012 module on “Family and Changing Gender Roles” of the International Social Survey Program (ISSP). In this survey, two new measures were included to address preferences for specific types of division of labor between men and women. Six types were presented, ranging from “the mother stays at home and the father works full time” to the opposite division of labor, and respondents were asked to indicate what, according to their opinion, was the best and the least desirable way to organize for the couple. This question forced respondents to decide for one alternative without making this dependent on additional conditions (such as preferences of the partners, capabilities, earning potential), the fact of which might be criticized for encouraging superficial and stereotypical answer behavior.
Therefore, we implemented an experiment in non-probability online surveys in Germany, Great Britain, Mexico, Spain, and the U.S. with a total of 2,689 respondents. Data collection was in June 2014. In this survey, half of the respondents received the original ISSP question, the other half of the respondents received a variant of the ISSP question in which an additional category “Each family should find the solution which works best for them” was added. Nearly half of the respondents used this additional category when it was offered. The respondents who selected the additional answer category also received a probing question regarding the reasons for opting for “individual solutions.”
The open-ended answers were then translated and coded into an elaborated category schema, which represents main criteria (such as preferences, capabilities, earning potential) used by respondents. In parallel, respondents also had to answer closed questions which also represent the main criteria and which were formulated on the basis of a previous study including a similar question.
In this presentation, we will show which perspectives respondents adopt when answering the newly developed ISSP items and whether the respondents from the five countries differ in their associations with regard to “individual solutions”. Results from the closed questions constructed in anticipation of the probing results will also be compared to the results from the probing question. Finally, we will try to answer the question whether the new ISSP question meets the expectations and where remaining problems are located.


3. Detecting and explaining Inequivalence: the case of patriotic feelings
Professor Peter Schmidt (University of Giessen and Humboldt Research Fellow Cardinal Wysczinski University Warsaw)
Dr Katharina Meitinger (GESIS Mannheim)
Professor Michael Braun (GESIS Mannheim)

M.Braun(GESIS Mannheim)/K.Meitinger(GESIS Mannheim)/P.Schmidt(University of Giessen)
Detecting and explaining inequivalence: The case of “patriotic feelings”

The 2013 ISSP Module on National Identity contains a newly developed item battery asking respondents about the potential effect of strong patriotic feelings in their country on different issues (e.g., intolerance, feeling of unity). However, it remains unclear whether all items of this item battery serve as indicators for one factor that captures the concepts of “patriotic feelings”, how this factor is associated with nationalism and different facets of proudness and whether it is cross-national comparable. Given previous qualitative findings with regard to national identity (e.g., Fleiß et al. 2009; Latcheva 2011; Meitinger 2016), we were also uncertain whether the term “patriotic feelings” is equally understood by respondents in different countries.
Therefore, we used a twofold approach: In a first step, we assessed the factor structure of the “patriotic feelings” battery with the 2013 ISSP data set for five countries (Germany, Great Britain, Mexico, Spain, and the U.S.) using quantitative approaches (confirmatory factor analysis) and evaluated the cross-national comparability with measurement invariance tests. In particular, we tested the equivalence with strict (MGCFA) and approximate (Bayes/alignment) measurement invariance approaches. In a second step, we replicated the ISSP item battery in a web survey conducted in May 2014 with 2,685 respondents from the same five countries as in the quantitative tests. The respondents came from a nonprobability online panel and were selected by quota for gender, age, and education.
In this presentation, we will discuss the findings from the quantitative measurement invariance tests concerning the dimensionality of the battery, error correlations, and cross loadings and the qualitative online probing. In particular, we will assess whether findings from online probing can help to explain the results of the measurement invariance tests