ESRA 2017 Programme

Tuesday 18th July      Wednesday 19th July      Thursday 20th July      Friday 21th July     

     ESRA Conference App


Friday 21st July, 09:00 - 10:30 Room: Q4 ANF2

Testing the Behavioral Validity of Survey Experiments

Chair Dr Knut Petzold (Catholic University of Eichstätt-Ingolstadt )
Coordinator 1Mr Volker Lang (Bielefeld University)

Session Details

Many survey experiments, such as factorial surveys, conjoint analyses or choice experiments are designed to measure behavioral intentions. However, there is an ongoing discussion how well intentions measured by such experiments predict actual behavior. Especially, it remains an important open research questions how and to what extent the levels and determinants of behavioral intentions and choices made in actual decision situations are related.

Even given high quality measurements of intentions by survey experiments testing the behavioral validity of these intentions is challenging. First, actual behavioral alternatives are restricted by non-experimental set environmental conditions. Hence, the relevance of intentions for behavioral outcomes might substantially depend on the situational context. Second, related survey data is often prone to unobserved heterogeneity. Finally, it is often a practical challenge to collect data on intentions and comparable actual behavior of the same sample of participants.

This session addresses issues linked to cross-validating experimentally measured intentions in surveys with corresponding behavioral data. Welcome are all contributions discussing theoretical models, empirical designs, analytic strategies and exemplary applications aiming to overcome related problems. In particular we invite papers dealing with at least one of the following questions:

- Which theoretical models are useful to specify the relationship between intentions and actual behavior in a general and empirical testable fashion?
- Which empirical designs can help to test the behavioral validity of intentions?
- Which data sources can serve as behavioral benchmark for validating intentions?
- What are the advantages and limitations of a validation strategy using declared behavior in surveys?
- What are the advantages and limitations of a validation strategy using observed behavior in laboratory or field experiments?
- What are the advantages and limitations of a validation strategy using observed behavior in natural situations?
- Which validation strategies are appropriate in which circumstances?
- In which situations and under which conditions is it appropriate to measure behavioral intentions using survey experiments?

Paper Details

1. Validating behavioral intentions from survey experiments in the field. A comparison of supposed and realized everyday life discrimination
Dr Knut Petzold (Catholic University of Eichstätt-Ingolstadt)
Professor Tobias Wolbring (University of Mannheim)

Survey experiments, such as factorial surveys or choice experiments, are becoming increasingly popular in recent years and are used in a variety of social science research fields. In addition to the advantages of a combination of experimental techniques with large-scale surveys one major merit of factorial survey is specifically seen in a reduced degree of social desirability. In particular, compared to direct questioning a reduced degree of socially desirable responses can be expected due to the complexity of the decision situation and the trade-offs to be made by the respondents. Based on these arguments a number of studies examined presumed behavioral reactions using factorial surveys. Such applications – usually implicitly – assume that the determinants of intended behavior that are identified in vignette studies do affect actual behaviour in a similar way. Since this fundamental assumption is largely untested, in particular for studying discrimination, we critically investigate it in the present study. For this purpose, student participants of a survey were randomly assigned to one of two experiments: a concealed unobtrusive field experiment based on the design of misdirected e-mails and a factorial Survey, which contains identical images of the e-mails in a survey. The e-mail informs about the reception of a student scholarship, while the message importance (high stakes vs. low stakes of money) and the name of the actual recipient of the misdirected email (german vs. arabic) were systematically varied. Results show that in the field and in the survey experiment the relative frequency of responses to the email increases with increasing importance of the message. In contrast, neither in the field experiment nor in the factorial survey we find systematic effects of recipient names. The findings indicate that the determinants of social discrimination revealed by vignette studies are very similar to determinants of actual discrimination behavior, at least in the examined situation and for the student sample under investigation. The results are finally discussed in terms of their outreach, general methodological considerations on the validation of survey experiments are given, and recommendations for further investigations are derived.


2. Behavioral intentions, actual behavior and the role of personality traits
Dr Katrin Drasch (Friedrich-Alexander-University of Erlangen-Nuremberg and Institute for Employment Research (IAB))

Factorial surveys are a powerful tool for collecting information on norms and attitudes. In addition, they can also be used to draw conclusions about behavior or more precisely about behavioral intentions. A topic of much controversy, however, is if and how personality traits influence intentions and actual behavior. Some scholars argue that individuals with different personality traits might answer fictitious situations differently because they are stimulated differently by them. Furthermore, recent research finds an influence of personality traits on actual behavior (e.g. Wichert 2010). This paper addresses this subject matter by analyzing the following research questions: 1) Are behavioral intentions as measured by factorial surveys related to actual behavior? 2) How do personality traits influence both actual and intended behavior? Theoretical basis is the theory of planned behavior. Here the compatibility principle claims that behavior intentions and actual behavior that deal with the same decision should be closely related. Due to this compatibility principle rather general personality traits are contrary to existing research results expected to have no direct influence on behavior itself.
The research questions will be addressed with data from a factorial survey collected among 395 female labor market re-entrants. These were asked about their willingness to accept lower wages if compensated by positive nonmonetary job-characteristics. The factorial survey contains information on behavioral intentions, i.e. on the likelihood to accept a given job-offer with certain characteristics as well as a short version (15 item-version as used in the German Socio-Economic Panel) of the Five-Factor Model (Big Five) which collects information on neuroticism, extraversion, openness to experience, agreeableness and conscientiousness. A follow-up study after one year also includes information on actual behavior, i.e. whether the woman has found a job and if yes, about this job`s characteristics.
Data are analyzed with a series of multi-level regression models. My results are that personality traits only have a minor influence on behavioral intentions. This confounds previous non-experimental research results. The distribution of the Big Five measures in the factorial survey, however, closely resembles the distribution in the representative GSOEP, suggesting a high level of construct validity. To conduct a real world validation, I use a logistic regression model on realized re-entry. The analysis reveals that women who are willing to accept more unfavorable job-characteristics are more likely to re-enter employment, suggesting a high correlation between results from the factorial survey with actual behavior. Comparing results from the factorial survey and the real world validation one arrives at similar conclusions despite the differences in the magnitude of some influence factors. Estimating models with and without personality traits leads to similar results, suggesting that it is not in general necessary to include them in factorial surveys.


3. Normative attitudes, beliefs and behavior: The new NS6 scale measuring social norms and a comparison with the social value orientation slider measure.
Dr Fabian Winter (Max-Planck-Institute for Research on Collective Goods)
Professor Heiko Rauhut (Universität Zürich, Institut für Soziologie)
Dr Marc Höglinger (Universität Bern, Insitut für Soziologie)
Dr Jürgen Fleiss (Universität Graz)

Surveys usually measure normative behaviour and attitudes by asking whether some (un)desirable action has been performed in the past, and how this behaviour is evaluated. This may lead to biased estimates if the respondents have an incentive to over-report or under-report due to e.g. social desirability. Instead of relying on anonymising techniques to measure prosocial behaviour (e.g. RRT, item count) , we propose a complementing, truly behavioural measure of pro-social behaviour and beliefs. In a series of studies in different populations, we implemented the SVO Measure (Murphy et al. 2014) and the NS6-Scale (Rauhut et al. 2016) in an online survey (N=200) and a group administered paper-pencil questionnaire (N=2000). Both measures are based on small tasks in which the respondents distribute a sum of money between themselves and others, and are relatively easy to implement in a survey. Other than the SVO, the NS6 additionally asks for normative and empirical beliefs about what one will and should do. To validate our measures, we use a series of behavioural tasks, e.g. cheating about the outcome of a die or the willingness to answer additional questions, and correlate these control tasks with our new measures as well as with more traditional survey measures. Our results suggest that the new measures are a forceful predictor for constructs usually associated with normative behaviour, such as cheating, cooperation, or political attitudes, and may therefore be a valuable tool for future studies.


4. Cross-validating survey experiments on justice principles with data from the lab
Ms Sandra Gilgen (University of Bern)

When deciding on how to divide goods or resources fairly, individuals can rely on different justice principles, the most prominent of them being 1) equity (or deservingness/merit), 2) equality and 3) needs. The reasons for choosing one principle over another can lie in rather time-constant individual (e.g. class – Robinson and Bell 1978; Shepelak 1989) or contextual factors (place of residence – Arts and Gelissen 2001; Henrich, Fehr, and Gintis 2004) or time-variant situational logics (e.g. family vs. workplace context – Deutsch 1975). While there is a long line of research focusing on single aspects of justice attitudes, a comprehensive overview on who opts for which principle of distributional justice in which situation is missing. Focusing on mechanism-based explanations, the research project “Justice: An individual, contextual or situational decision?” tackles these questions using a mixed-mode design (paper pencil and online) with a national sample of individuals 16 years and older in Switzerland. In order to address the problem of social desirability bias, which is especially salient in the context of attitudinal research, factorial survey experiments are embedded in the questionnaires. Instead of applying the most prevalent approach of capturing justice principles by presenting respondents with descriptions of people and stating their income and then asking for an evaluation thereof (Alves and Rossi 1978; Jasso and Rossi 1977; Liebig et al. 2009), an alternative design is developed. In place of evaluating incomes based on vignettes, the respondents are asked to actively distribute resources of different amounts (e.g. 9 x CHF 1000,- or 9 x CHF 10,-) among people or institutions characterized in vignettes, according to their preferences in regard to distributional justice. This modified version of a factorial survey (active distribution instead of evaluation on a scale) has the advantage of capturing the trade-offs between the different distributional principles – which often stand in contradiction to one another – more directly. In order to test the validity of the survey responses, a subset of respondents – within a reasonable travel distance to the laboratory – will be randomly assigned to additionally participate in a laboratory experiment, in which they will be asked to perform distributional tasks. The focus will lie on individual and situational factors, since larger scale contextual factors cannot be considered due to the distance-from-laboratory restriction. That way, survey responses to distributional tasks can be compared to behavioral data from the laboratory, thus cross-validating the robustness of data from survey experiments. The extension of the data collection to the laboratory setting further enables the consideration of altruist versus selfish motives – potential sources of bias to the implementation of distributions judged just by the respondent – in distributional tasks. On the one hand, the project is expected to provide insight into the mechanisms driving the distributional preferences of individuals. On the other, it offers the rare opportunity of comparing the survey and laboratory data of the same sample of participants – thus enabling promising methodological insights in regard to the validity and robustness of experimental data on