Tuesday 14th July      Wednesday 15th July      Thursday 16th July      Friday 17th July     

Tuesday 14th July, 14:00 - 15:30 Room: L-102

Experimental designs in online survey research 2
Convenor	Mr Henning Silber (Göttingen University )
Coordinator 1	Mr Jan Karem Hoehne (Göttingen University)
Coordinator 2	Professor Dagmar Krebs (Giessen University)

Session Details

Experimental studies have become increasingly popular in survey research and are carried out in various disciplines such as sociology, political science, linguistics, economics and psychology. In survey research experimental designs are useful tools to get a better understanding of cognitive processes in order to give better practice advice for improving study and questionnaire design. In particular, the technological advances have made it significantly easier to use experimental designs in online field experiments as well as in computerized laboratory experiments.

This session invites presentations on empirical studies and theoretical discussions of experimental designs in online survey research.

- Empirical online research can include studies on response behavior and social desirability bias, as well as experiments on response rates and question design effects. Furthermore, we especially encourage presentations with replicated experimental results and welcome replications in different social contexts such as different cultural, educational and ethnic groups.
- Additionally, we invite presentations that discuss the value of experiments from a theoretical perspective. Theoretical presentations could contrast the merits and the limits of different forms of experimental study designs or provide a future outlook on the prospects of online experiments in survey research.

Presentations could cover the following research areas:

- Theory of experimental study designs
- Replication of experimental results
- Comparisons between different experimental designs (e. g., laboratory and field experiment)
- Split-ballot experiments (e. g., context effects, question order, response order, acquiescence, visual design effects, verbal effects)
- Choice experiments
- Laboratory experiments on response behavior (e. g., using eye tracking)
- Experiments with incentives
- Vignette studies
- Future prospects of experimental designs

Paper Details

1. Does Personalized Feedback Increase Respondent Motivation?
Mr Simon Kühne (Socio-Economic Panel (DIW Berlin))
Professor Martin Kroh (Socio-Economic Panel (DIW Berlin))

Web-surveys technically allow to feedback personal information to respondents based on their previous responses. This personalized feedback may not only be used for targeted follow-up questions, but we argue that returning personal information to respondents that they consider novel and important may also increase their interest in the survey. We implemented an experiment in the context of the Berlin Aging Study II providing graphical representation of respondents’ scores in a standardized personality test to a randomly allocated treatment group. We use self-reported and paradata to estimate the effect of the personalized feedback on respondents' motivation and behavior.

2. Redistribution preferences and acquiescence bias
Dr Elias Naumann (University of Mannheim)

Redistribution preferences are a widely researched topic and it is well established that the cultural and social context plays an important role in shaping attitudes. The ‘standard’ measurement of redistribution preferences uses AgreeDisagreeScales which are subject to acquiescence bias. Such bias would be in particular problematic for the substantive findings if it also varies with socio-cultural context of respondents. In this paper we use data from the ESS and a Splitballot-experiment to examine whether the acquiescence bias differs across countries and between social groups. We replicate our analysis in three European online surveys to evaluate possible mode effects.

3. Cheating in web surveys. Evidence from a split-ballot repeated experiment on knowledge questions on basic EU facts
Mr Cristiano Vezzoni (University of Trento, Italy)
Mr Riccardo Ladini (University of Trento, Italy)

The paper analyses a split-ballot repeated experiment on knowledge questions on basic EU facts in a panel web survey carried out in Italy before and after the 2014 European Elections (N > 2500). The control group received a neutral prompt to the knowledge questions, while the treatment group received a normative prompt inviting the respondents to answer without using internet. The first results show a consistent lower proportion of correct answers for the treatment group, lack of relation between “cheating” and socio-demographic characteristics, and a misleading effect of increasing consistency of the knowledge scales when “cheating” is more diffuse.