Testing the Behavioral Validity of Survey Experiments |
|
| Chair | Dr Knut Petzold (Catholic University of Eichstätt-Ingolstadt ) |
| Coordinator 1 | Mr Volker Lang (Bielefeld University) |
Survey experiments, such as factorial surveys or choice experiments, are becoming increasingly popular in recent years and are used in a variety of social science research fields. In addition to the advantages of a combination of experimental techniques with large-scale surveys one major merit of factorial survey is specifically seen in a reduced degree of social desirability. In particular, compared to direct questioning a reduced degree of socially desirable responses can be expected due to the complexity of the decision situation and the trade-offs to be made by the respondents. Based on these arguments a number of studies examined presumed behavioral reactions using factorial surveys. Such applications – usually implicitly – assume that the determinants of intended behavior that are identified in vignette studies do affect actual behaviour in a similar way. Since this fundamental assumption is largely untested, in particular for studying discrimination, we critically investigate it in the present study. For this purpose, student participants of a survey were randomly assigned to one of two experiments: a concealed unobtrusive field experiment based on the design of misdirected e-mails and a factorial Survey, which contains identical images of the e-mails in a survey. The e-mail informs about the reception of a student scholarship, while the message importance (high stakes vs. low stakes of money) and the name of the actual recipient of the misdirected email (german vs. arabic) were systematically varied. Results show that in the field and in the survey experiment the relative frequency of responses to the email increases with increasing importance of the message. In contrast, neither in the field experiment nor in the factorial survey we find systematic effects of recipient names. The findings indicate that the determinants of social discrimination revealed by vignette studies are very similar to determinants of actual discrimination behavior, at least in the examined situation and for the student sample under investigation. The results are finally discussed in terms of their outreach, general methodological considerations on the validation of survey experiments are given, and recommendations for further investigations are derived.
Factorial surveys are a powerful tool for collecting information on norms and attitudes. In addition, they can also be used to draw conclusions about behavior or more precisely about behavioral intentions. A topic of much controversy, however, is if and how personality traits influence intentions and actual behavior. Some scholars argue that individuals with different personality traits might answer fictitious situations differently because they are stimulated differently by them. Furthermore, recent research finds an influence of personality traits on actual behavior (e.g. Wichert 2010). This paper addresses this subject matter by analyzing the following research questions: 1) Are behavioral intentions as measured by factorial surveys related to actual behavior? 2) How do personality traits influence both actual and intended behavior? Theoretical basis is the theory of planned behavior. Here the compatibility principle claims that behavior intentions and actual behavior that deal with the same decision should be closely related. Due to this compatibility principle rather general personality traits are contrary to existing research results expected to have no direct influence on behavior itself.
The research questions will be addressed with data from a factorial survey collected among 395 female labor market re-entrants. These were asked about their willingness to accept lower wages if compensated by positive nonmonetary job-characteristics. The factorial survey contains information on behavioral intentions, i.e. on the likelihood to accept a given job-offer with certain characteristics as well as a short version (15 item-version as used in the German Socio-Economic Panel) of the Five-Factor Model (Big Five) which collects information on neuroticism, extraversion, openness to experience, agreeableness and conscientiousness. A follow-up study after one year also includes information on actual behavior, i.e. whether the woman has found a job and if yes, about this job`s characteristics.
Data are analyzed with a series of multi-level regression models. My results are that personality traits only have a minor influence on behavioral intentions. This confounds previous non-experimental research results. The distribution of the Big Five measures in the factorial survey, however, closely resembles the distribution in the representative GSOEP, suggesting a high level of construct validity. To conduct a real world validation, I use a logistic regression model on realized re-entry. The analysis reveals that women who are willing to accept more unfavorable job-characteristics are more likely to re-enter employment, suggesting a high correlation between results from the factorial survey with actual behavior. Comparing results from the factorial survey and the real world validation one arrives at similar conclusions despite the differences in the magnitude of some influence factors. Estimating models with and without personality traits leads to similar results, suggesting that it is not in general necessary to include them in factorial surveys.
Surveys usually measure normative behaviour and attitudes by asking whether some (un)desirable action has been performed in the past, and how this behaviour is evaluated. This may lead to biased estimates if the respondents have an incentive to over-report or under-report due to e.g. social desirability. Instead of relying on anonymising techniques to measure prosocial behaviour (e.g. RRT, item count) , we propose a complementing, truly behavioural measure of pro-social behaviour and beliefs. In a series of studies in different populations, we implemented the SVO Measure (Murphy et al. 2014) and the NS6-Scale (Rauhut et al. 2016) in an online survey (N=200) and a group administered paper-pencil questionnaire (N=2000). Both measures are based on small tasks in which the respondents distribute a sum of money between themselves and others, and are relatively easy to implement in a survey. Other than the SVO, the NS6 additionally asks for normative and empirical beliefs about what one will and should do. To validate our measures, we use a series of behavioural tasks, e.g. cheating about the outcome of a die or the willingness to answer additional questions, and correlate these control tasks with our new measures as well as with more traditional survey measures. Our results suggest that the new measures are a forceful predictor for constructs usually associated with normative behaviour, such as cheating, cooperation, or political attitudes, and may therefore be a valuable tool for future studies.
When deciding on how to divide goods or resources fairly, individuals can rely on different justice principles, the most prominent of them being 1) equity (or deservingness/merit), 2) equality and 3) needs. The reasons for choosing one principle over another can lie in rather time-constant individual (e.g. class – Robinson and Bell 1978; Shepelak 1989) or contextual factors (place of residence – Arts and Gelissen 2001; Henrich, Fehr, and Gintis 2004) or time-variant situational logics (e.g. family vs. workplace context – Deutsch 1975). While there is a long line of research focusing on single aspects of justice attitudes, a comprehensive overview on who opts for which principle of distributional justice in which situation is missing. Focusing on mechanism-based explanations, the research project “Justice: An individual, contextual or situational decision?” tackles these questions using a mixed-mode design (paper pencil and online) with a national sample of individuals 16 years and older in Switzerland. In order to address the problem of social desirability bias, which is especially salient in the context of attitudinal research, factorial survey experiments are embedded in the questionnaires. Instead of applying the most prevalent approach of capturing justice principles by presenting respondents with descriptions of people and stating their income and then asking for an evaluation thereof (Alves and Rossi 1978; Jasso and Rossi 1977; Liebig et al. 2009), an alternative design is developed. In place of evaluating incomes based on vignettes, the respondents are asked to actively distribute resources of different amounts (e.g. 9 x CHF 1000,- or 9 x CHF 10,-) among people or institutions characterized in vignettes, according to their preferences in regard to distributional justice. This modified version of a factorial survey (active distribution instead of evaluation on a scale) has the advantage of capturing the trade-offs between the different distributional principles – which often stand in contradiction to one another – more directly. In order to test the validity of the survey responses, a subset of respondents – within a reasonable travel distance to the laboratory – will be randomly assigned to additionally participate in a laboratory experiment, in which they will be asked to perform distributional tasks. The focus will lie on individual and situational factors, since larger scale contextual factors cannot be considered due to the distance-from-laboratory restriction. That way, survey responses to distributional tasks can be compared to behavioral data from the laboratory, thus cross-validating the robustness of data from survey experiments. The extension of the data collection to the laboratory setting further enables the consideration of altruist versus selfish motives – potential sources of bias to the implementation of distributions judged just by the respondent – in distributional tasks. On the one hand, the project is expected to provide insight into the mechanisms driving the distributional preferences of individuals. On the other, it offers the rare opportunity of comparing the survey and laboratory data of the same sample of participants – thus enabling promising methodological insights in regard to the validity and robustness of experimental data on