Online probing: Cognitive interviewing techniques in online surveys and online pretesting 3 |
|
| Chair | Dr Katharina Meitinger (GESIS Leibniz Institute for the Social Sciences ) |
| Coordinator 1 | Dr Dorothée Behr (GESIS Leibniz Institute for the Social Sciences) |
| Coordinator 2 | Dr Lars Kaczmirek (GESIS Leibniz Institute for the Social Sciences) |
Over the last few decades, European societies have witnessed unprecedented increases in social inequalities. The impact of rising equalities will depend on the extent to which such inequalities are perceived and evaluated as unjust by the people. To assess and compare the impact of such justice evaluations, it will be necessary to collect data on social justice attitudes that are comparable across the countries of Europe. This study is part of the works accompanying the development of the rotating module on “Social Justice and Fairness in Europe” in round 9 of the European Social Survey (ESS) and will investigate the issue of cross-cultural equivalence of the measurement of social justice attitudes. We employ a mixed methods approach, combining focus group discussions, online probing and cross-cultural cognitive interviewing to tackle the following questions: To what extent are the concepts of social justice and fairness understood equally in different countries and languages? What differences do exist and how large are they? First, focus group discussions are used to clarify the the understanding of the concepts of justice and fairness in different countries. Respondents from different cultural backgrounds that will feature in round 9 of the ESS are invited to discuss their understanding of social justice and fairness to pinpoint similarities and differences at a conceptual level. Second, cross-cultural cognitive interviewing (CCCI) is used to test items intended to measure social justice attitudes in different languages. CCCI is a version of cognitive interviewing used for assessing the cognitive processes behind the response process in personal interviews with a small number of respondents in a cross-cultural setting. Third, online probing (OP) is used; a relatively new tool, combining the advantages of CCCI for the assessment of survey questions and that of an online survey, achieving a greater sample size and broader coverage of issues. Questions on social justice attitudes are administered in an online survey, followed by comprehension and category-selection probes. Initially, in both CCCI and OP, we present existing items of the ESS tapping the concepts of justice to a convenience sample of university students and employees from an international background covering a variety of European countries. Information gathered during this process will feed into formulation of new items which will then be presented to the same pool of respondents again for fine-tuning of the final items. The study will focus on German, English, Polish and Russian as these languages cover both Eastern and Western Europe and are likely to be represented in the pool of students and employees we plan to recruit for the study. The combined information from the three methods employed will provide us with a broad but detailed picture of the converging and diverging meanings and understandings of the concepts of social justice and fairness and the related items intended to measure these. The discussion of the results will focus on how problems of lacking congruence of concepts in different countries may be tackled by developing culturally equivalent items for inclusion in ESS9.
Web probing, that is, implementing verbal probing techniques traditionally used in cognitive interviewing in online surveys, is a method to substitute for or complement quantitative techniques to establish functional equivalence of item batteries in cross-cultural research. It can also been applied to single questions or items. It is particularly useful to evaluate the appropriateness of newly developed questions and response alternatives.
This approach is illustrated with one question from the 2012 module on “Family and Changing Gender Roles” of the International Social Survey Program (ISSP). In this survey, two new measures were included to address preferences for specific types of division of labor between men and women. Six types were presented, ranging from “the mother stays at home and the father works full time” to the opposite division of labor, and respondents were asked to indicate what, according to their opinion, was the best and the least desirable way to organize for the couple. This question forced respondents to decide for one alternative without making this dependent on additional conditions (such as preferences of the partners, capabilities, earning potential), the fact of which might be criticized for encouraging superficial and stereotypical answer behavior.
Therefore, we implemented an experiment in non-probability online surveys in Germany, Great Britain, Mexico, Spain, and the U.S. with a total of 2,689 respondents. Data collection was in June 2014. In this survey, half of the respondents received the original ISSP question, the other half of the respondents received a variant of the ISSP question in which an additional category “Each family should find the solution which works best for them” was added. Nearly half of the respondents used this additional category when it was offered. The respondents who selected the additional answer category also received a probing question regarding the reasons for opting for “individual solutions.”
The open-ended answers were then translated and coded into an elaborated category schema, which represents main criteria (such as preferences, capabilities, earning potential) used by respondents. In parallel, respondents also had to answer closed questions which also represent the main criteria and which were formulated on the basis of a previous study including a similar question.
In this presentation, we will show which perspectives respondents adopt when answering the newly developed ISSP items and whether the respondents from the five countries differ in their associations with regard to “individual solutions”. Results from the closed questions constructed in anticipation of the probing results will also be compared to the results from the probing question. Finally, we will try to answer the question whether the new ISSP question meets the expectations and where remaining problems are located.
M.Braun(GESIS Mannheim)/K.Meitinger(GESIS Mannheim)/P.Schmidt(University of Giessen)
Detecting and explaining inequivalence: The case of “patriotic feelings”
The 2013 ISSP Module on National Identity contains a newly developed item battery asking respondents about the potential effect of strong patriotic feelings in their country on different issues (e.g., intolerance, feeling of unity). However, it remains unclear whether all items of this item battery serve as indicators for one factor that captures the concepts of “patriotic feelings”, how this factor is associated with nationalism and different facets of proudness and whether it is cross-national comparable. Given previous qualitative findings with regard to national identity (e.g., Fleiß et al. 2009; Latcheva 2011; Meitinger 2016), we were also uncertain whether the term “patriotic feelings” is equally understood by respondents in different countries.
Therefore, we used a twofold approach: In a first step, we assessed the factor structure of the “patriotic feelings” battery with the 2013 ISSP data set for five countries (Germany, Great Britain, Mexico, Spain, and the U.S.) using quantitative approaches (confirmatory factor analysis) and evaluated the cross-national comparability with measurement invariance tests. In particular, we tested the equivalence with strict (MGCFA) and approximate (Bayes/alignment) measurement invariance approaches. In a second step, we replicated the ISSP item battery in a web survey conducted in May 2014 with 2,685 respondents from the same five countries as in the quantitative tests. The respondents came from a nonprobability online panel and were selected by quota for gender, age, and education.
In this presentation, we will discuss the findings from the quantitative measurement invariance tests concerning the dimensionality of the battery, error correlations, and cross loadings and the qualitative online probing. In particular, we will assess whether findings from online probing can help to explain the results of the measurement invariance tests