ESRA 2025 Program
Get the ESRA 2025 App on iOS or Android:
All time references are in CEST
Assessing and improving survey data quality in low- and middle-income countries (LMICs) |
| Session Organisers |
Professor Timothy Johnson (University of Illinois at Chicago)
Mrs P. Linh Nguyen (University of Essex, University of Mannheim)
Dr Yfke Ongena (University of Groningen)
|
| Time | Tuesday 15 July, 13:30 - 15:00 |
| Room |
Ruppert rood - 0.51 |
Researchers working in low- and middle-income countries from diverse disciplines, such as development economics, demography, and other social sciences, are increasingly engaged in investigating different aspects of the Total Survey Error to improve data quality. They are especially concerned by how survey data quality affects substantial results, as well as poverty and demographic rates.
This session aims to mainstream research on all aspects of survey data quality stemming from LMICs where the historical development, the conditions, and the implementation of survey methodology differs from other contexts. We see this session as unique opportunity to foster the network of like-minded researchers and practitioners, as well as to promote research results focusing on LMICs in preparation for the ESRA conference in 2027 organised jointly with the World Association for Public Opinion Research (WAPOR) and the first WAPOR conference in a Sub-Saharan African country, Kenya, in 2028.
Researchers may present their work on any issue(s) encountered along the full survey lifecycle from questionnaire development and testing, including scale development; translation, adaptation, and assessment of questionnaires into local languages; sampling innovations using unconventional sample frames; survey participation, data collection challenges and solutions through innovative uses of technology; minimizing measurement error; interviewer effects; survey data quality control; respondent comprehension and burden; etc.
There is no specific regional focus and papers may cover a variety of topics. Nevertheless, the studies to be considered should rely on data coming from LMICs. Cross-national comparisons in these contexts are also welcome.
Keywords: cross-cultural survey methods
Papers
Studying language switching in multilingual survey interviews – Evidence from a Zambian Face-to-Face Survey
Mrs P. Linh Nguyen (French Institute for Demographic Studies (INED), University of Essex, University of Mannheim) - Presenting Author
Most low- and middle-income countries are multilingual leading to the situation that both interviewer and respondent are speaking multiple local languages to a varying degree. As multilingual respondents, especially those with lower education, differ in their proficiency in the survey language, some will exhibit more cognitive processing problems evidenced by audible manifestations of problematic interactional behaviours during the interview (i.e., through seeking for clarification or repetition of the question).
Using the interactional analysis of the recordings of ten selected questions in a survey on financial behaviour and attitudes in Zambia on a sample of more than 800 interviews in two local languages (Bemba and Chewa), we analyze the relationship between six indicators of problematic interactional behaviours and interviewer effects (1. language switches by either interviewers or respondents; 2. exact reading of the question; 3.any pre-emptive and follow-up behaviours by interviewers to obtain a codable answer (such as providing explanations without being asked or feedback or probing); 4. seeking clarification; 5. indicators of uncertainty (including providing pauses, fillers, and repairs, as well as verbal expressions of uncertainty).
This study’s objective is to document this process of switching language and its implication to survey data quality. The Zambian survey we rely on estimates that about 2 to 7 percent of interviews exhibit some form of language switching based on the analysis of 10 recorded and analysed questions for more than 1,000 respondents from a probabilistic household survey in Zambia. The results show for both provinces that language switching does not occur as a single phenomenon but always in co-occurrence with other problematic interactional behaviour. Thus, we can categorise language switching as another problematic behaviour indicating the breakdown of the cognitive answer process and a disruption to the ideal question-answer sequence.
Evaluating the Impact of Mode Transition from CAPI to CATI on Data Quality: Evidence from TURKSTAT’s Life Satisfaction Survey
Dr Hilal Arslan (Hacettepe University Institute of Population Studies) - Presenting Author
Mrs Aslıhan Kabadayı (Hacettepe University Institute of Population Studies)
Like many countries, the COVID-19 pandemic has forced Turkish Statistical Institute (TURKSTAT) to make changes in its data collection and fieldwork strategy due to limited opportunities to conduct face-to-face interviews. Therefore, for majority of the surveys face-to-face CAPI sampled are substituted with CATI data collection mode suddenly. Based on Total Survey Error approach main types of measurement errors stemmed from data collection mode are due to nonresponse, social desirability bias, satisficing behavior, differences in handling of “don’t know” or refusal response options, contextual information, using visual or auditory cues, difficulty in attention, presence of others during interview, questionnaire design and interview length that may influence the respondent’s answers. Against this background, our study aimed to investigate the differences in the survey estimates of target variables i.e. life satisfaction, happiness and satisfaction with subdomains of life by interviewing by telephone instead of face-to-face on important target variables. Up to our knowledge this is the first study to check data quality for CAPI-CATI administrative mode transition for TURKSTAT surveys and we were particularly interested in whether these differences are due to mode-specific measurement errors including nonresponse rather than coverage errors by comparing subjective well-being indicators derived from data collected in person (CAPI) and those collected over the phone (CATI) by checking descriptive statistics and applying multivariate statistical models. For data analysis, in order to explore the potential impact of the mode difference on our survey estimates, we analyzed survey data for the years 2003-2023 by taking the complex sample design (stratification, clustering, and weighting) into account. Preliminary findings of the study show that there are statistically significant changes in the distribution of the response categories for the selected attitudinal questions and subjective measures.
Gender Role Attitudes In The Philippines And Elsewhere
Professor Harry Ganzeboom (VU University Amsterdam) - Presenting Author
The Gender Role Attitude (GRA) index is a 5-11 indicator instrument to measure differences in opinions about the proper division of household tasks. The instrument has been used in ISSP’s Gender&Family modules since 1988. Using a multiple-correspondence analysis, the ISSP1994 data on GRA were critically examined by Blasius (2006), who scorns particularly the Philippine data for being "completely inconsistent", and concludes that “it makes no sense to compare the data (…) from the Philippines with that of any of the other countries”. We take up the challenge by examining the GRA data from a broader and less technical design. First, we analyse measurement quality in all four ISSP Gender&Family modules, which brings a considerably broader database of countries and time-points. Second, we examine measurement validity not by internal associations of the instrument, but in relationship to external validation criteria, such as gender and cohort. Third, we also seek to separate validity from reliability and test whether an instrument with poor (but some) reliability can still bring out structural relationships with the validation criteria.
Our first finding is that the Philippine GRA-data indeed have low reliability but are far from randomly generated. This low reliability is consistent across waves and is replicated for a number of ISSP countries that are similar either in socio-economic development or geographical/cultural location: Brazil, Mexico, Thailand, India, Japan, China, Taiwan and Venezuela. The problem may be more substantive than technical. Second, we find that despite low reliability, the cross-national ranking of the Philippines as very gender-role conservative is consistent between ISSP waves, which confirms that data with low reliability can still contain substantive meaning. Third, we find that the usual individual determinants of differences work in the Philippines just the same as elsewhere.
