All time references are in CEST
Analyzing Open-Ended Questions in Survey Research 3 |
|
| Session Organiser | Dr Alice Barth (Department of Sociology, University of Bonn) |
| Time | Wednesday 19 July, 14:00 - 15:00 |
| Room | U6-21 |
Open-ended questions in surveys provide information on respondents’ personal perspectives, their interpretation and understanding of concepts. In addition, respondents get an opportunity for individualized feedback. Open-ended questions are almost indispensable in collecting data on issues that are too diverse for standardized questions, e.g. job designations. While many researchers used to refrain from open-ended questions due to the arduous process of transcribing and coding responses, the rise of computer-assisted data collection and software solutions for analyzing large amounts of text data can now accelerate the work process. Nevertheless, in dealing with open-ended questions, researchers need a specific methodological toolbox that is adapted to analyzing unstructured text data.
This session aims at discussing methods for processing and analyzing responses to open-ended questions. Topics of particular interest are, for example,
- coding responses to open-ended questions (manual, semi-automated or automated coding)
- text mining / natural language processing approaches
- qualitative content analysis of responses
- data quality and mode effects in open-ended questions
- open-ended probes as a means of evaluating comparability, validity and comprehension of questions
- analyzing respondent feedback on the survey process
- using information from open-ended questions to complement or contradict results from standardized questions
We are looking forward to contributions that highlight the methodological and/or substantial potential of open-ended questions in survey research.
Keywords: open-ended questions; data quality; text mining; text data; content analysis
Mr Alexandre Pollien (FORS) - Presenting Author
Dr Jacques-Antoine Gauthier (University of Lausanne)
Given technological advancements for analysing large amounts of text data, open-ended questions come more and more in the forefront of the contemporary survey methodology. Assumed to provide deeper information on respondents’ perspectives, about interpretation and understanding of concepts, they are also victims of their potential. Open-ended questions provide scattered information in a format and material, making them hard to be analysed. Our paper addresses these resources by analysing the pragmatic-reflexive status of responses, their figurative level, referential or ironic status, that we confront to semantic-thematic. We will consider these aspects from the point of view of personal response style. The pragmatic-reflexive status of discourse raises issues of context, including cross-cultural comparison, intrinsic to attitudinal and pragmatic-reflexive dimensions. Researchers need methodological that prevent missing pragmatic of the discourse. Using experimental data from a survey conducted in Switzerland in 2022, we compare answers across a design where respondents answered six times with some variations to a similar request of interpretation ("say in 2 sentences..."). We test several algorithms/methods to analyse to what extent we can predict personal styles of answers. The design allows using information from standardized questions to assess outcomes from open-ended questions. We are looking for some recommendations of how survey designer can handle scattering of pragmatic dimension of responses, and analysis methods that can integrate this.
Ms Kübra Annac (Witten/Herdecke University) - Presenting Author
Dr Yüce Yilmaz-Aslan (Witten/Herdecke University)
Professor Patrick Brzoska (Witten/Herdecke University)
During the COVID-19 pandemic, ensuring patient-centered rehabilitation while minimizing the risk of infection has been challenging for rehabilitation facilities. The aim of this cross-sectional study was to investigate which problems directors of rehabilitation facilities have encountered in addressing the pandemic and where there is still a need for support. Methodically, different types of survey questions and modes have been explored for this purpose.
By means of an online and an additional postal survey, all medical rehabilitation facilities in Germany (n=1,629) were surveyed between June and September 2021. The questionnaire was developed in collaboration with experts based on previous research results as well as on literature. Closed-ended questions with predefined choices and in-depth open-ended questions were used. Responses were available from 535 facilities (response rate=32.8%) and were analyzed descriptively. Responses from open-ended questions were analyzed using qualitative content analysis.
50.8% of the facilities had problems implementing measures on processes and structures. 36.7% of respondents considered the implementation of some types of therapies to be problematic. The open-ended questions about the reasons for encountered challenges generated up to 298 extensive responses each. The responses revealed that primarily financial factors, a deficient infrastructure, and a lack of human resources were obstacles to implementation. Factors promoting implementation are financial support from health care organizations, increased digitalization of services, and uniform regulations.
The study illustrates the methodological and substantive potential of open-ended questions in quantitative surveys – in the present case identifying deficits and informing about how systematic change in the health care system can be initiated through adequate multi-stakeholder communication. In order to exploit the full potential of open-ended questions in quantitative surveys, strategies need to be explored to effectively process and analyze large amounts of qualitative data.
Professor Alessandra Decataldo (Università degli studi di Milano Bicocca)
Professor Brunella Fiore (Università degli studi di Milano Bicocca)
Dr Noemi Novello (Università degli studi di Milano Bicocca)
Mr Federico Paleardi (Università degli studi di Milano Bicocca) - Presenting Author
Dr Sara Recchi (Università degli studi di Milano Bicocca)
This contribution aims to discuss the methodological and substantive implications of open-ended responses, in comparison with standardised responses, within a longitudinal study on tutors of host organisations involved in Projects for Transversal Competences and Orientation (PCTOs), an Italian education policy. PCTOs offer high-school students a period of practical experience within an organisation, educating and orienting them by promoting on-the-job training. The email survey, carried out in two waves one year apart, investigates the perspective of tutors in organizations and institutions about PCTO design, objectives, implementation, and practical functioning.
We consider in this paper two specific questions: one is related to the main activities carried out by the host organisation; the other refers to the professional role of the respondent. Methodologically, the choice was motivated by the impossibility of coding the broad spectrum of possible activities in a structured question, which would have resulted in valuable information loss. The limited population (of 367 projects) allowed the manual coding of the answer in both waves. In addition, the second wave introduces standardized answer modalities, besides open-ended questions: within a triangulation framework, the aim is to analyse convergence and discrepancies in responses. By adopting two different response modalities, we have the chance to assess the quality of manual coding procedure. Moreover, the comparison is relevant for its substantial impact as well, as we have the chance to assess variations over time in manual coding and standardised answers regarding activities carried out within host organisations. Relying on in-depth interviews with a sub-sample of 15 tutors, eventual discrepancies in responses might also be addressed, providing additional qualitative data. This provides us with insights on the respondents' professional paths, particularly in their role of tutors. Furthermore, the tutor activity is compared to other variables regarding PCTOs' implementation processes.
Ms Rebekka Atakan (University of Bonn, Department of Sociology) - Presenting Author
Open-ended survey questions are a great way for survey participants to express their opinions in their own words. However, the analysis of the responses can be very time consuming. In recent years, forms of automated text analysis have been introduced for a time-saving analysis of open-ended questions.
Typically, if one is interested in the automated analysis of unstructured text data, topic modeling is a popular method to cluster similar words and, by doing this, detect abstract topics that occur in the document corpus. Topic modeling has originally been developed for the analysis of longer documents; therefore, a debate emerged around adequate topic modeling procedures for the time-saving automated analysis of short texts such as responses to open-ended survey questions (Roberts et al. 2014; Finch et al. 2018; Pietsch and Lessmann 2018; Nanda et al. 2021).
In this presentation, we compare topic modeling to an alternative, but less discussed method for analyzing open-ended survey questions: multiple correspondence analysis (MCA). While topic modeling aims at the clustering of similar words in topics, MCA is a scaling technique for categorical variables (Le Roux and Rouanet 2010; Husson and Josse 2014). Therefore, in addition to detecting similarities, the relational structure of the terms used by the participants can be visualized. With MCA, it is possible to map the relationship of all terms to each other, meaning that also contrasting subject matters can be identified.
Using data of the fifth wave of the Cologne Dwelling panel, we will analyze the responses of two open-ended questions on respondents’ perception of their neighborhood with topic modeling and MCA. Strengths and weaknesses of both procedures will be compared and we will discuss ways to integrate the results.