Making Long Web Surveys Feasible: Matrix Design and Multiple Imputation in Cross-Cultural Surveys |
|
| Session Organisers | Dr Michael Ochsner (FORS) Dr Gudbjorg Andrea Jonsdottir (University of Iceland) Dr Tobias Gummer (GESIS) |
| Time | Wednesday 17th July, 09:00 - 10:30 |
| Room | D18 |
Cross-cultural surveys are an important instrument for studying change over time and differences in attitudes and values between groups as well as cultures regarding various subjects and topics. To a great extent such large-scale surveys rely on face-to-face interviewing. However, recent developments, such as a more active life style which leads to a lower contact success rate and a much wider internet penetration, make the use of web surveys increasingly promising. Yet, web surveys have the major drawback that the recommended survey length hovers around 20 minutes while cross-cultural general population surveys usually are designed for 60 minutes’ interviews in order to cover a broad range of variables to be analysed together. One way to solve this problem is the application of a matrix design (or split questionnaire design) where the questionnaire is split into modules and not every respondent answers every module. This design can be implemented in various ways, but will result in a data matrix with missing values. Multiple imputation provides one way of analysing these data.
There is still scarce knowledge on how to best apply matrix designs in cross-cultural general population surveys and the methodological implications of doing so. The lack of research is unfortunate since such large-scale surveys are often subject to high costs and require rigid methodological guidelines and standards to ensure cross-national comparability. This session aims at advancing the research on matrix designs and especially welcomes—but is not limited to—submissions that focus on how …
- a matrix can be designed, operated, and implemented in cross-cultural surveys
- feasible the web and other self-administered modes are to conduct general-population surveys with matrix designs
- to detect and resolve issues of cross-cultural comparability when applying matrix designs
- multiple imputation or alternative methods can be applied to analyse data from matrix designs.
Keywords: matrix design; split questionnaire design; web survey; mixed mode; cross-cultural surveys
Dr Tobias Gummer (GESIS) - Presenting Author
Dr Guðbjörg Andrea Jónsdóttir (University of Iceland)
Professor Ruud Luijkx (Tilburg University)
Dr Michèle Ernst Stähli (FORS)
In many countries, large-scale general population surveys are challenged with decreasing response rates and increasing survey costs. Switching from face-to-face interviewing to conducting self-administered interviews in web or mail mode might be one possible solution to counteract these trends. However, the questionnaires used in general population survey frequently reach lengths of 60 minutes or more and, thus, exceed the survey length recommended for self-administered survey modes. Applying a matrix design and, thus, splitting the questionnaire into smaller parts can help to reduce the overall length of the questionnaire and enable fielding it in web or mail mode. However, for many countries, little is known about how self-administered survey with matrix designs will perform and would actually proof an alternative or supplement to face-to-face interviewing in term of outcome. Moreover, conducting self-administered surveys with a matrix design in a cross-cultural survey program becomes even more challenging as this requires several countries to implement the matrix design in a comparable way while designing their surveys to perform well in their country-specific contexts.
In the present study, we will provide insights into how a matrix design was implemented in the European Values Study 2017 (EVS). We describe the methodological challenges that motived the EVS to considering self-administered modes, introduce the design and cross-cultural implementation of the matrix design, and give an overview of how the self-administered surveys were designed in several countries (e.g., Switzerland, Iceland, Netherlands, Germany) as complement to face-to-face EVS-surveys. We then show how these surveys performed in each participating country with respect to their outcome in comparison to the corresponding face-to-face surveys.
Mr Stefan Thor Gunnarsson (The social Science Research Institute of the University of Iceland) - Presenting Author
Dr Gudbjorg Andrea Jonsdottir (The social Science Research Institute of the University of Iceland)
Mr Arni Bragi Hjaltason (The social Science Research Institute of the University of Iceland)
In Iceland, data for the European Values Study (EVS) was collected using a mixed-mode survey in 2017. As in all previous EVS surveys, a face-to-face (f2f) method was used as a basis, but in this round a web version was also used on another sample drawn from the population.
EVS is an extensive survey and long web surveys can be wearisome for participants. For that reason, a matrix design was used, where the questionnaire was divided into blocks. The majority of participants were not introduced to all the blocks of questions, which reduced the time it took these participants to complete the questionnaire.
This paper describes differences in results from the two different modes used in EVS 2017 in Iceland. The aim is to analyse whether the demographics of respondents receiving the two modes of the survey reflect the characteristics of the population. The difference in response and completion rate between the two survey modes and between those that got the whole web survey and those that got shorter versions of the questionnaire was also examined.
The results show that the response rate was higher using the web than f2f, but the rate of complete responses was lower. Overall differences in answers between the modes were minimal except to questions on sensitive issues, where web survey participants were more prone to give answers that are not socially accepted, indicating presence of social desirability response set bias in f2f. The web mode was more representative of the population, but that could be due to Iceland's exceptionally high percent of internet users. Response and completion rate was higher for those that got the shortened version of the web survey than those that got the whole questionnaire, but that difference was in our opinion not great enough to compensate for the missing data resulting from the matrix.
Dr Michael Ochsner (FORS) - Presenting Author
Dr Gudbjorg Andrea Jonsdottir (University of Iceland)
Dr Tobias Gummer (GESIS)
Dr Jessica Herzing (University of Lausanne)
There is scarce knowledge on how to field cross-cultural general-population surveys on the web because such surveys usually consist of multiple modules and exceed the recommended length for web surveys. One solution to reduce survey length is to field it in a matrix design. The disadvantage of such a design is that there are missing values by design that reduce the power of statistical analysis.
Theoretically, missing values by design are not an issue for statistical analysis as these missings are missing completely at random. However, problems might arise when analysing subsamples or topics that need a large number of observations, such as working couples or discrimination. Another problem might occur when composite indices are used which include questions that are spread over several questionnaire splits.
Multiple imputation offers the possibility to analyse data containing missing values without reducing power too much and at the same time taking the uncertainty of missing values into account.
However, not much is known about how to apply multiple imputation for a matrix design in cross-cultural settings. In our presentation, we present the potential and pitfalls of multiple imputation for the analysis of cross-cultural data from a matrix design. We use data from the European Values Study 2017 fielded as a web survey in a matrix design in Switzerland, Germany, and Iceland shedding light on when to impute, how to impute the data, and how to analyse the imputed data in a cross-cultural setting.
Miss Miriam Magdolen (Karlsruhe Institute of Technology) - Presenting Author
Mr Sascha von Behren (Karlsruhe Institute of Technology)
Dr Bastian Chlond (Karlsruhe Institute of Technology)
Professor Peter Vortisch (Karlsruhe Institute of Technology)
In travel behaviour research, an increasing number of travel surveys include attitudinal questions to determine the effects of the psychological dimension. In the case of international studies, the focus is on the comparability of the data collected and the results. Especially when attitudinal questions are used, a potential difference in the response behaviour of people from different cultures must be considered. Using the data from a standardized travel survey conducted in Shanghai (China), Berlin (Germany) and San Francisco (USA), we analyse, detect and compare different variations of response bias. We aim to investigate whether the given response behaviour results from different response strategies and whether there are cross-cultural differences. In the survey, an item set with 38 statements is used to question the respondents’ attitudes towards different means of transport. A 5-point Likert-scale is applied within a grid structure. This allows the participants to easily answer the questions. However, this also allows rapid answers without reading the single statements, which leads to poor data quality. There are several types of questionable response behaviour, e.g., Straightlining, where each item is assigned the same response category as the previous one. To identify response bias in our data, we use different classifications of Straightlining, which vary in the extent of non-differentiation and the complexity of calculation. In addition to visual biases, we also examine content-related aspects such as the tendency to agree or to disagree with all statements. An ordered logit regression identifies factors that influence the response behaviour. The analyses indicate that people from Shanghai and San Francisco show an increased tendency for response bias. We also detect the highest level of central tendency (choosing the middle category) among people from Shanghai. Overall, the results show differences between the response behaviour of people from different cultures and therefore underline the importance of such cross-cultural comparisons.