ESRA logo
Tuesday 14th July      Wednesday 15th July      Thursday 16th July      Friday 17th July     

Wednesday 15th July, 16:00 - 17:30 Room: HT-104

The impact of questionnaire design on measurements in surveys 3

Convenor Dr Natalja Menold (GESIS )
Coordinator 1Ms Kathrin Bogner (GESIS)

Session Details

Questionnaire design is crucial for obtaining high-quality survey data. Still, there is a great need for research that helps to better understand how and under which conditions different design aspects of questionnaires impact measurement process and survey data quality. Therefore, researchers are invited to submit papers dealing with questionnaire design features such as question wording, visual design and answer formats, instructions, introductions and other relevant design aspects of questionnaires. Also, different means of measurement such as questions with nominal answer categories, rankings, ratings, sematic differentials or vignettes can be addressed or can be matter of comparison. Of interest is the impact of questionnaire design on response behavior, on systematic as well as non-systematic error or on validity. In addition, respondents’ cognition or motivation can be in focus of the studies.

Paper Details

1. The Who: experimental evidence on the effect of respondent selection on collecting individual asset ownership information
Mr Talip Kilic (The World Bank)
Ms Heather Moylan (The World Bank)

This paper reports on a randomized survey experiment that field-tested in Uganda five approaches to respondent selection while collecting data on individual ownership of and rights to assets. Differentiation across legal, economic and reported ownership, and the bundle of rights was key and individuals associated with each were identified in all treatment arms at the asset-level for all asset classes. The paper compares the effects of the survey treatments on individual-level asset ownership and control, documents the correlates of intra-household discrepancies in reporting, and provides operational guidance on individual-disaggregated asset data collection in developing countries.

2. Portuguese National Health Examination Survey: questionnaire development
Ms Ana Santos (Epidemiology department, National Health Institute Doutor Ricardo Jorge)
Professor Ana Paula Gil (Epidemiology department, National Health Institute Doutor Ricardo Jorge)
Professor Marta Barreto (Epidemiology department, National Health Institute Doutor Ricardo Jorge)
Ms Vânia Gaio (Department of Epidemiology, National Health Institute Doutor Ricardo Jorge)
Dr Sónia Namorado (Department of Epidemiology, National Health Institute Doutor )
Ms Irina Kislaya (Department of Epidemiology, National Health Institute Doutor Ricardo Jorge)
Ms Heidi Lyshol (Norwegian Institute of Public Health)
Dr Baltazar Nunes (Department of Epidemiology, National Health Institute Doutor Ricardo Jorge)
Dr Carlos Matias Dias (Department of Epidemiology, National Health Institute Doutor Ricardo Jorge)

This communication discusses the development of the questionnaire for the First Portuguese National Health Examination Survey. The questionnaire was evaluated in face-to-face semi structured interviews with a convenience sample of 30 participants. The main issues encountered by participants were: lack of knowledge on specific terminology, difficulty in estimating time or ranking and sensitive topics, such as mental health and functional evaluation. To improve the questionnaire the following solutions were implemented: reformulation of questions, inclusion of notes to interviewers and participants clues, development of glossaries and introduction of response cards to facilitate the response and approach in sensitive questions.

3. An integrated household survey for Wales
Dr Steven Marshall (Welsh Government)

The Welsh Government and its partner organisations carry out a range of surveys of individuals which provide valuable information for policy decisions. To provide both robust survey data and achieve value for money five surveys will be merged from April 2016. Three of the surveys are currently conducted face-to-face while one is self completion and one is carried out via telephone. A new combined questionnaire is being developed that will provide a coherent single whole and will aim to minimise context effects and discontinuities due to mode changes. A major field test will be run during 2015.

4. Undesired response to surveys, wrong answers or poorly worded questions? How respondents insist on reporting their situation despite unclear questioning
Dr Eva Lelièvre (INED)
Dr Loïc Trabut (INED)

When researchers and statistical institutions collect data on families, they start by making assumptions on what constitutes a family and how relationships and households are organized. These shape questionnaires and influence respondents’ answers. Nevertheless, self-administered surveys leave room for remarks and personalized declarations. Drawing examples from the last French Family survey, we present how respondents describe their situation in the restricted space of questionnaire forms, especially when their circumstances do not fit the design. We examine how respondents react to allusive questions of the Family survey, which were designed to spare them the difficulty of speaking about deceased children.

5. Ask a Positive Question and Get a Positive Answer: Evidence on Acquiescence Bias from Health Care Centers in Nigeria
Dr David Evans (World Bank)
Professor Mario Macis (Johns Hopkins University)
Mr Felipe Dunsch (World Bank)
Ms Qiao Wang (World Bank)

Acquiescence bias may affect the results of surveys to measure the quality of health care in developing countries. This study tests for acquiescence bias in the context of 938 patient exit surveys at health facilities across Nigeria. Respondents were randomly assigned to either receive a positively-framed list of Likert items, a negatively-framed list, or a random mix of items. Across 11 items, respondents with negative items were significantly more likely to rate items unfavorably on 10 of the 11 items. The rate of unfavorable response rose by 450% (10 percentage points) from a base of few unfavorable responses.