Tuesday 14th July      Wednesday 15th July      Thursday 16th July      Friday 17th July     

Thursday 16th July, 09:00 - 10:30 Room: L-102

Multifactorial Survey Experiments (Factorial Surveys, Choice Experiments and Conjoint Analysis) 1
Convenor	Professor Katrin Auspurg (Goethe-University Frankfurt )
Coordinator 1	Dr Carsten Sauer (Bielefeld University)
Coordinator 2	Professor Peter M. Steiner (University of Wisconsin-Madison)

Session Details

There is a fast growing trend in the social sciences to combine the advantages of multifactorial experimental designs with surveys. Factorial surveys – often labelled as “vignettes studies” – have been used for more than 30 years to gather data on descriptions of hypothetical situations or objects to explore principles of judgment and decision making. Choice experiments help to explore respondents’ preferences and willingness to pay. In addition there is increasing use of conjoint analyses in sociology and political sciences. The experimental design provides a high internal validity, while the survey design improves external validity. Computer assisted interviewing that allows implementing many different treatments have made these methods even more popular.

Despite frequent use there are still many open questions concerning design features of multifactorial survey experiments that offer most reliable and valid results. We are interested in methodological research on the design of factorial surveys, choice experiments or conjoint analyses (e.g., validity of tabular vs. text presentations, video-vignettes), sampling techniques to select the experimental treatments (random sampling vs. fractional sampling), analysis strategies (e.g. accounting for the multi-level structure of response data; testing validity in regard to respondents’ attitudes, beliefs, and behavior).


Questions could be:
Design of questionnaire: How to present the vignettes to the respondents? What kind of answering scale provides most valid results? How to prevent order effects? How do respondents cope with the information provided on vignettes or choice sets?
Sampling techniques: What are the benefits and drawbacks of fractional versus random sampling? How do sampling techniques like blocking by respondent strata and interviewers influence efficiency of estimates?
Analysis strategies and validity: Which models provide unbiased estimates? How to address possible censoring of responses? Respondents’ idiosyncracies? When to use multilevel analyses, and how to validate results?

Paper Details

1. The Factorial Survey: Reliability and Internal Validity of Random and Quota Designs
Dr Hermann Duelmer (University of Cologne)

The factorial survey is an experimental design in which the researcher combines varying descriptions of situations (vignettes) for judgement by respondents. With an increasing complexity of the research question it becomes impossible for a respondent to judge all possible vignettes. To address this bottleneck, most of the time random designs are recommended, whereas quota designs are not discussed at all. The aim of this contribution is to compare the reliability and internal validity of random and different quota designs. The benchmark for empirical comparison is established by the estimators from a parsimonious full factorial design, each answered by 132 students.

2. Realism versus Simplicity in Creating Factorial Survey Vignettes: Implications for Analysis
Dr Ruth Ludwick (Kent State University, Robinson Memorial Hospital)
Dr Kristin Baughman (Northeast Ohio Medical University)
Dr David Jarjoura (The Ohio State University)

Factorial surveys have been used to determine which factors are most influential in healthcare professionals’ clinical decisions. As our research moved from exploratory to hypothesis-driven models we simplified vignettes. In beginning we used a more traditional approach in which each vignette was randomly selected from a large universe of possible vignettes. Later we limited the vignettes to only 5 dichotomous variables allowing us to more efficiently and accurately estimate main effects and random effect variances. By simplifying vignettes we lost some details of earlier vignettes, but were able to provide a more robust analysis of the data.

3. Over- and under-complexity in factorial surveys. A case study on professional judgment
Mrs Marjolijn De Wilde (University of Antwerp - Centre for Social Policy)
Mr Peter Goos (University of Antwerp - Faculty of Applied Statistics & KULeuven - Faculty of Bioscience Engineering)

Over- and under-complexity are two possible treats to multi-factorial experimental research. We compared questionnaires with a varying number of factors on a professional judgement topic. We used three sorts of analysis to test the validity of the questionnaires: a comparison of the level-one variance and of the regression coefficients and a six-fold cross-validation. Our results suggest that the consistency of respondents’ answers is quite similar among the three designs (Spearman rank correlation = 0,68-0,81). Therefore, if researchers wish to minimize the risk of over-complexity, we recommend to use a split-plot design.

4. Assessing Dimensions of Integration: A Rating and Ranking Comparison with the Factorial Survey
Dr Sonja Pointner (LMU Munich)
Mrs Ilona Pap (University of Zurich)

In our study we analyze what kind of dimensions of integration are perceived as vital for a migrant’s degree of integration into society. To allow a relative comparison of the dimensions we use a ranking method in the factorial survey. Each respondent received three decks, each containing four vignettes. The vignettes were designed as fact sheets with information on fictitious migrants. Respondents had to rank all the vignettes on the deck according to the presumed degree of the migrant’s integration. Furthermore, we compare the results of the ranking method with rating conditions. Results show differences in the evaluation.