ESRA 2019 Draft Programme at a Glance

Recent Developments in Question Testing 1

Session Organisers Dr Cornelia Neuert (GESIS - Leibniz Institute for the Social Sciences)
Dr Timo Lenzner (GESIS - Leibniz Institute for the Social Sciences)
TimeThursday 18th July, 09:00 - 10:30
Room D17

It is universally acknowledged that testing survey questions prior to administering them to respondents is a vital part of a survey as pretesting reduces potential measurement error and helps to improve the quality of the data collected. For question testing, survey methodologists have a broad set of methods at their disposal (cognitive interviews, behavior coding, response latency measurement, vignettes, expert reviews). Recently, innovative techniques and new data sources are added to the survey researcher’s toolbox, such as eye tracking, web probing, mouse movements or crowdsourcing.
So far, few methodological studies have addressed the effectiveness of these newer methods in improving questionnaires and how they compare to traditional pretesting methods.

This session invites papers that…
(1) explore innovative uses of new methods or techniques for question testing;
(2) highlight the relative effectiveness of different pretesting methods
(3) demonstrate how new and existing techniques might best be used in combination (best-practice examples) to offer additional insights.

We also invite presentations discussing (new) question testing methods in a cross-cultural context.

Keywords: Question testing, Web Probing, Cognitive interviewing, Eye Tracking, Pretesting Methods

A Best Practices Approach to Cognitive Interviewing

Ms Margaret Roller (Roller Research) - Presenting Author
Ms Karen Kellard (Social Research Centre)

Although survey researchers have debated quality in their designs for decades, a quality approach to the cognitive interviewing method is new. Verification strategies, the role of reliability and validity, and attention to “qualitative rigor” are well-documented in the literature. In 2015, a new approach was introduced that incorporates research principles at each phase of the qualitative research process without stifling the goals of inquiry, creative methods, and interpretations utilized by qualitative researchers. This approach is the Total Quality Framework (TQF).

The TQF offers a conceptual foundation from which qualitative researchers are able to think about the quality of their cognitive interviewing designs. The TQF is rooted in the belief that cognitive interviewing and all qualitative research methods must be: credible, analyzable, transparent, and useful. These core objectives represent the four interconnected components of the TQF: Credibility (the completeness and accuracy of the data); Analyzability (the processing and verification of the data); Transparency (the completeness [“thickness”] of disclosure in the final research document, and transferability); and Usefulness (the ability to do something of value with the outcomes).

Utilizing best practices within each component provides qualitative researchers conducting cognitive interviews a way to (a) give explicit attention to quality issues, (b) critically examine the possible sources of variability and bias, (c) incorporate features into their designs that try to mitigate these effects, and (d) take the implications of these effects into consideration during analysis.

This presentation defines the TQF components, provides examples of best practices in the cognitive interviewing method, and discusses the interrelationship between the components and the ultimate value of this approach.

Cognitive Testing Groups vs.Cognitive Interviews - a comparison of approaches

Ms Karen Kellard (The Social Research Centre, Australian National University) - Presenting Author

This paper will contribute to discussions on the relative effectiveness of different qualitative pre-testing approaches - focus groups and individual in-depth interviews (IDIs) for the cognitive testing of survey questions. Whilst the use of IDIs for cognitive interviews is a well-recognised and used technique, the researchers have observed that for some individuals (such as young people or others who may be vulnerable) cognitive testing can be somewhat daunting, even with very skilled interviewers (potentially making it challenging to uncover measurement errors). Such individuals appear to be more comfortable and relaxed in a focus group setting with other similar people. Nevertheless, cognitive testing in focus groups has both known and unknown limitations, not least conforming behaviours and a reluctance to disagree with group members, thus hiding areas of measurement error.

Drawing on findings from a recent small experiment (testing the same set of questions in groups and individual interviews) conducted by qualitative researchers at the Social Research Centre (Australian National University), we will review the relative strengths and weaknesses of each approach in identifying measurement errors through the assessment of comprehension, judgement, retrieval and response, and provide an overview of when the two different approaches may work best.

Cognitive or qualitative? Qualitative Pretest Interviews in Questionnaire Development

Dr Christina Buschle (FernUniversität in Hagen)
Dr Arne Bethmann (Max-Planck-Institute for Social Law and Social Policy) - Presenting Author
Dr Herwig Reiter (German Youth Institute)

Cognitive interviews have become one of the most important pretest procedures in the development and evaluation of questionnaires. Different techniques such as thinking aloud or different probing approaches examine the comprehensibility of questions and statements and uncover the reasons for difficulties of respondents in answering questionnaires. What is missing in cognitive interviewing is the methodological framing of these techniques. Against this background our contribution proposes such a methodological framing on the basis of similarities of cognitive interviewing procedures with qualitative interviewing strategies. Both involve direct research interactions that can be regarded as communication processes between interviewer and respondent based on intersubjective understanding (Fremdverstehen). We introduce methodically integrated communication strategies of two established qualitative interviewing techniques - the problem-centred interview and the discursive interview - and discuss their potential for the development of a pretest interview approach, which includes the techniques of the cognitive interview and accounts for the essentially social character of processes of clarifying comprehension. We suggest the term Qualitative Pretest Interview (QPI) in order to avoid a narrow understanding of pretest procedures as focused on the problem of ambiguous cognitions. Finally, we discuss the potential practical benefits of this approach of interviewer-respondent interaction for standardized survey research and provide examples from QPIs that were conducted during actual questionnaire development.

Quick, cheap and good: Face to face versus video cognitive interviews

Dr Ruxandra Comanaru (NatCen Social Research) - Presenting Author

There are increasing demands to pre-test survey questions in a quick, cheap and thorough manner. These three elements – time, cost and quality – cannot easily be prioritised one over another, and thus questionnaire designers and testers often find themselves at a loss, as pressures on one of these elements will have implications on the others.
The present study investigates the use of video conferencing tools (such as Skype, GoToMeeting or FaceTime) to conduct cognitive interviews. We compare the results and findings from these video interviews with traditional face-to-face cognitive interviews on various aspects that are of interest, in an attempt to elucidate whether video cognitive interviews are a viable alternative to the traditional face-to-face cognitive interviews. Specifically, we look at impact it has on the three elements of interest:
- Time: we look at the ease or difficulty in recruiting different groups of participants and scheduling the interviews.
- Cost: what is saved in travel costs, venue, reduced incentives versus the cost of the software and other costs.
- Quality: we investigate the ease or difficulty in building rapport with the participants, ensuring that the technological aspects are working appropriately, using screen shares instead of showcards for testing, the time the participants are willing to spend engaging with the cognitive interviewer, whether the findings of the video cognitive interview differ from the face to face findings, and if so, in what ways.
We will argue that although the golden standard of cognitive interviewing will always be face-to-face, in certain situations we need to make adjustments to the pre-testing methods we use in order to accommodate the demands of a project. Video cognitive interviews will not replace the face-to-face interviews, however, they do provide a quicker, cheaper and comparable quality alternative. Nonetheless, the limitations need to be clear from the start.