ESRA 2017 Programme

Tuesday 18th July      Wednesday 19th July      Thursday 20th July      Friday 21th July     

     ESRA Conference App

Tuesday 18th July, 11:00 - 12:30 Room: F2 108

Mixed Methods and the Limits of Social Research 2

Chair Dr Andrea Hense (Sociological Research Institute Göttingen (SOFI), Germany )
Coordinator 1Professor Udo Kelle (Helmut Schmidt University Hamburg, Germany)
Coordinator 2Dr Felix Knappertsbusch (Giessen University, Germany)

Session Details

The term "mixed methods" usually indicates a mixture of qualitative and quantitative sampling procedures, data collections, or data analyses within a single study or longitudinal program of inquiry. One of the most important prospects of mixed and multi-method research (MMMR) is its ability to overcome the limitations of traditional mono-method approaches. Method combinations may be used to corroborate or generalize findings, or they may serve to present a more nuanced and multi-facetted view of a given phenomenon. Moreover, MMMR approaches may even raise awareness for possible shortcomings which would otherwise go unrecognized by proponents of mono-method traditions.

Yet at the same time MMMR itself is often confronted with certain limitations ranging from incomplete methodical knowledge, conflicts between methodological camps and rigid disciplinary boundaries to limited material resources and time constraints. One of the most important problems of mixed methods research is how to combine qualitative and quantitative data analysis in generating “meta-inferences”. In practical research applications specific difficulties arise which are not covered by existing methods literature: How do we deal with convergent, complementary, and divergent findings? What implications do combined analyses have for fieldwork processes, data organization, and data documentation? How can we use joined displays during the research process to facilitate comparison between results of qualitative and quantitative analyses and how can we use them to communicate the joined results?

This session invites papers discussing the possibilities of MMMR for overcoming the limits of mono-method research as well as current problems and limitations within the diverse and evolving field of method integration. It will provide a forum for discussing problems of joined data analyses which were faced during specific research projects and the solutions found for them. Especially, it will focus on strategies that intend to relate qualitative and quantitative results explicitly instead of reporting two different studies that are only loosely linked to each other. Papers should either present a specific methodical issue with regard to concrete empirical MMMR projects, or discuss a methodological problem specifically pertaining to the prospects and limits of method integration.

Paper Details

1. Like a bridge over troubled water? Combining unexpected results from quantitative research with qualitative interview material by using quote matrixes
Dr Christiane Gross (University of Hanover, Germany)
Mr Thomas Gurr (University of Hanover, Germany)
Professor Monika Jungbauer-Gans (German Centre for Higher Education Research and Science Studies, Hanover, Germany)
Mr Sebastian Lang (University of Hanover, Germany)

Previous research with mixed-methods designs has shown several alternatives how to combine qualitative and quantitative approaches. Nevertheless, the debate needs to focus on precise, functional and application-orientated advice to mix both types of data. We intend to contribute to this issue by discussing one way to combine unexpected results from quantitative research with qualitative interview material. In this regard, we would like to present chances and difficulties which occur when displaying these results in segment or quote matrixes.

Our approach is based on Leech und Onuewegbuzie (2009) and is most similar to a mixed concurrent dominant status design with a dominant quantitative strand. We use the panel study “Labour Market and Social Security” (PASS) as quantitative data source and 12 guided interviews with long term unemployed people as qualitative data source to investigate the association of stigma consciousness and sanctions by the German employment service. Initially, combining different methods was applied for providing contextual understanding and “putting ‘meat on the bones’ of ‘dry’ quantitative findings” (Byrman 2006: 106-107). By using side by side joint displays to integrate both results and the unexpected results of the quantitative analysis it became more appropriate to understand contradictions between both strands and the conceptual framework of the study. In general, with these joint displays the results of the quantitative examination strand can be seen in a particular and useful way. The same applies to the linkage to the theoretical framework of the study.

Even if scrutinizing the quantitative results works very well, some questions will still remain open. Discussing the integrated results is suitable for bridging, but also bears the risk of driving apart quantitative and qualitative research again. This attempt clarifies the well-known problems of compatibility and the specific and converse shortcomings of both approaches such as the poor quality of some variables, missing reality, etc. on the one hand and e.g. generalizability on the other hand. In addition, other issues emerge which are mainly related to the communication and the presentation of the results, such as different nomenclature, how to structure the manuscript or which journal to choose for publication, etc.

However, quote matrixes serve as suitable tools for analyzing the inferences in our mixed-methods design. Notably analytical depth and transparency can be increased and further research issues developed. However, the limitations of this approach need to be addressed.

2. Mixed Methods Data Analysis: Qualitative Analysis and Data Transformation
Dr Susanne Vogl (University of Vienna)

The challenge in data analysis often lies in taking account of the multi-dimensionality and complexity of the data and at the same time discovering patterns, which requires reduction and simplification. Quantitizing qualitative data can serve as an additional step during data analysis to broaden the perspective and collect complementary views. Examples from research illustrate how information gathered during one data collection can generate different types of data or how qualitative data can be transformed to be analysed statistically. Practi-calities of integrating qualitative and quantitative analyses are illustrated to offer good-practice examples for transformation designs. With the methodological reflection of research practice we evaluate consequences for the field of mixed methods research, in which qualitative and quantitative analyses are usually independent of each other and the statistical analysis of codes created during an interpretive phase plays a minor role.

3. Mixed Methods Research: a convergent or a complementary approach?
Dr Noemi Novello (Università degli Studi di Milano-Bicocca)
Professor Alessandra Decataldo (Università degli Studi di Milano-Bicocca)

Within mixed methods research (MMR) two principal approaches can be identified. The first one aims to a logic of complementarity, under which bringing qualitative and quantitative strategies together may be seen as an attempt of information enrichment, to obtain a more comprehensive understanding of a phenomenon [1]. The second one concentrates on a convergence perspective that focuses on the possibility to overcome single methods biases through data integration [2]. The assumption behind this second approach is that qualitative and quantitative methods have specific strengths and limitations and merging them may increase the overall validity of a study. While the first objective appears rather unproblematic both theoretically and empirically, the second one poses some challenges, especially in relation to the elicitation of “meta-inferences”.
The purpose of this work is to explore how the two perspectives of complementarity and convergence are intended and endeavored by MMR studies within the social sciences. In order to provide an answer to this issue, research synthesis seems particularly appropriate. We will conduct, indeed, a research synthesis using a mixed strategy to investigate those studies located in the MMR field. We partially refer here to the framework mixed methods research synthesis (MMRS), which is “a systematic review applying the principles of mixed methods” [3, p. 662]. Still, this study would only consider mixed researches from a methodological point of view and it is not intended to synthesize findings from mixed, qualitative and quantitative researches.
We will realize a literature dépistage on MMR in social research, identifying significant articles from scientific journals databases. Here the WoS (Web of Science) and Scopus databases are explored from an exhaustive perspective, retrieving all articles published in international journals (2003-2016). The phase of evaluation would regard the screening of the retrieved mixed studies, according to relevant criteria. Analyses are conducted following three main data analysis techniques: automated content analysis, citation network analysis and scholars interviews. The first analysis would help to distinguish between theory and praxis, underling eventual discrepancies or concurrences between the declared theoretical approach of the article and the actual solution implemented in the research design, thanks to a coding protocol. The second one would identify nodes and relationships among the authors of these articles to identify clusters of citations/authors. Then, on the basis of pinpointed clusters, interviewees will be selected among researchers, so that actors’ perspectives may emerge, enriching the analyses and being compared with outcomes from the text and citation analyses.
Finally, we expect the approach of complementarity to show some consistency in respect to the two level of theory and praxis, while it is likely that convergence poses differences in ways it is theoretically considered and modalities it is implemented empirically.

[1] Jick T., (1979), Mixing qualitative and quantitative methods: Triangulation in action, in “Administrative Science Quarterly”, 24:4.
[2] Erzberger C., Prein G., (1997), Triangulation: Validity and empirically-based hypothesis construction, in “Quality&Quantity”, 31.
[3] Heyvaert M., (2013), Mixed methods research synthesis: Definition, framework and potential, in "Quality&Quantity", 47:2.

4. Integrating Learning Analytics, Survey Self-Reports, and Qualitative Data: Insights from Two Pilot Studies
Dr Evgenia Samoilova (University of Mannheim)
Professor Florian Keusch (University of Mannheim)
Professor Tobias Wolbring (University of Mannheim)

New sources of data such as, for example, learning analytics (LA) provide opportunities for addressing new research questions as well as alternative approaches to answer the old ones. It has been argued that these new sources of data (known as found or organic data) are not able to replace traditional forms of data and that they should instead be viewed as a complimentary source of information. While survey researchers have been actively discussing integration strategies with respect to found or organic data, discussion on the role and advantages of qualitative data given these new challenges is at the outset. Drawing on two pilot studies, we illustrate how qualitative data can be integrated with LA and survey self-reports for two purposes: evaluation of data quality and answering substantive research questions. While in the first approach qualitative data help interpret discrepancies found while comparing survey and LA data, in the second application it is used to provide addition insights missed by the other two methods. Two pilot studies were implemented during two online courses conducted between February and September 2016 as part of an online program for working professionals at the University of Mannheim. The survey and LA data stem from 16 participants observed over 12 weeks of the 1st course (Nit=192) and 15 participants in the 2nd course (Nit=180).

The first approach of integrating qualitative data is centered around the problem of student workload measurement. As self-reported data obtained from tradition survey instruments can suffer from various problems, LA can offer an alternative way of measuring workload in online or blended learning environments. We compared results provided by video watching logs and a weekly evaluation survey. We found that survey self-report data and LA result in different estimates of video watching workload. Given that neither of the two methods is free from errors, understanding of the observed differences called for additional data. To provide further insights for the observed differences, we have conducted a series of qualitative interviews with 13 study participants. The interviews included two components. The first one included cognitive interview questions aimed at evaluation of the survey data. Furthermore, we included questions addressing how students interacted with videos, which helped us understand the data generating process of LA data.

The second application of qualitative data included the evaluation of two design changes for the purpose of increasing student engagement: implementation of additional video material and comparison of synchronous and asynchronous communication techniques with students. Following the design-based research perspective popular in the learning science, we integrated LA, weekly survey data as well as qualitative semi-structured interviews in order to draw a more complete picture of students’ engagement. Qualitative interviews were conducted at the start (N=16) and at the end of the courses (N=13).

We discuss differences in integration strategies arising from the two distinct purposes. We conclude with a discussion of ethics and data protection implications.