Frenemies: Understanding and interpreting the impact of answer scale orientation on survey data quality |
|
| Coordinator 1 | Dr Yongwei Yang (Google) |
| Coordinator 2 | Dr Mario Callegaro (Google) |
| Coordinator 3 | Mr Aaron Sedley (Google) |
Survey researchers are wary of answer scale design choices’ impact on data quality and respondent experience. One important area of work surrounds scale orientation with visually-displayed rating scale questions. Answer scale orientation may be described in two ways: display orientation (vertical or horizontal) and positive pole location (top or bottom for vertically orientated scales; left or right for horizontally orientated ones). Important theories about how answer scale orientation may affect survey response have been offered alongside empirical evidence. These include primacy effect due to satisficing (Krosnick & Alwin, 1987; Chan, 1991; Garbansky, Schaeffer, & Dykema, 2019; Mavletova, 2013; etc.), primacy effect due to anchor-and-adjust (Yan & Keusch, 2015), and interpretation heuristics (Tourangeau, Couper, & Conrad, 2004, 2013). The applicability of these theories in 3MC context has also been explored (Ferrall-Nunge & Cooper, 2011; Yang, Timpone, Callegaro, Hirschorn, Achimescu, & Natchez; etc.). Nevertheless, this body of work remains limited and there is a need for further testing the generalizability of competing theories in diverse contexts and for clearer and practical recommendations. This session will facilitate the expansion of this important area of work. Specifically, we seek contributions that systematically evaluate the generalizability of existing or new theories about answer scale orientation along various dimensions, such as:
-- Different substantive constructs (e.g., customer satisfaction, user trust, political attitude)
-- Construct polarity (bipolar versus unipolar)
-- Using verbal labels versus emojis
-- Full versus partial labeling
-- Language and cultural settings
For quantitative studies, we encourage contributions that use high quality samples. We also encourage contributions that dig deeper into the response mechanisms using qualitative or observational techniques.
Finally, we encourage works that carefully investigate existing evidence through systematic reviews or meta-analysis.