Keynote: Design and implementation of comparative surveys by Professor Lars Lyberg

Surveys in multinational, multiregional and multicultural contexts, also called 3MC or simply comparative surveys, are becoming increasingly important in society. Examples of such surveys include the European Social Survey (ESS), the Gallup World Poll, the Programme for International Student Assessment (PISA), and the Programme for International Assessment of Adult Competencies (PIAAC). Also the European Statistical System is comparative in nature. The purpose of these surveys is to compare populations such as nations or regions. But there is of course also a strong national and local interest in these surveys.

The design and implementation of 3MC surveys are associated with numerous problems. Comparing often implies a ranking of nations, especially in surveys that make assessments of various kinds. Literacy surveys among adults and student assessments are examples of such surveys, where the ranking might lead to results that can hurt national pride but also a “league table” of unemployment rates can accomplish that. Even though a ranking seldom is a primary goal in 3MC surveys it is something that is easily picked up by media, politicians and eventually the public, resulting in extensive discussions and sometimes ill-founded reforms and other inefficient initiatives.  There is need for a thorough discussion of the ranking practice also from the perspective that positions in the ranking table are very sensitive to small variations in local implementations.

Problems that we experience in a mono-survey situation are magnified in a 3MC situation. The general experience seems to be that the importance of having a solid infrastructure in place for the design and implementation of comparative surveys cannot be emphasized enough. A solid infrastructure implies input harmonization rather than output harmonization, a central team making sure that standards and requirements are reasonably met, and that there is an element of continuous improvement of the survey. Most comparative surveys do not have these ingredients in place. In some surveys everything is allowed to vary. The only thing that exists is a source questionnaire and a project leader and countries are asked to conduct the survey. Thus, we have a spectrum here ranging from the very ambitious such as the European Social Survey to all those not so ambitious.

In this talk I will discuss some of the ten golden rules for comparative research formulated by the late Sir Roger Jowell, the founder of the European Social Survey. Based on my experiences with 3MC surveys, including the usual nonsampling error suspects, 3MC-specific error sources such as translation, adaptation, cultural-specific cognition phenomena, and the need for a suitable quality control system, I will then add a few golden rules related to organizational, process and product quality of 3MC surveys with a view towards the future survey landscape.