ESRA 2019 Draft Programme at a Glance
Mixed-device online surveys: A total survey error perspective 2
|Session Organisers|| Dr Olga Maslovskaya (University of Southampton)
Professor Gabriele Durrant (University of Southampton)
Professor Patrick Sturgis (University of Southampton)
|Time||Thursday 18th July, 14:00 - 15:30|
We live in a digital age with widespread use of technologies in everyday life. Technologies change very rapidly and affect all aspects of life, including surveys and their designs. Online data collection is now common in many countries and researchers are adapting online surveys in response to the requirements of mobile devices, especially smartphones. Mobile devices can facilitate collection of new forms of data such as sensor data. It is important to assess different sources of error in mixed-device online surveys.
This session welcomes submissions of papers on different sources of error in online surveys in both cross-sectional and longitudinal contexts. The following topics are of interest:
• Coverage issues
• Data quality issues
• Item nonresponse
• Unit nonresponse
• Breakoff rates
• Completion times
• Response styles
• Mobile device use
• Optimisation of surveys and adaptation of question design for smartphones and data quality,
• Impact of different questions’ designs or presentations on response quality across devices,
• New types of data collection associated with mobile devices such as sensor data and data quality.
We encourage papers from researchers with a variety of backgrounds and across different sectors, including academia, national statistics and research agencies.
This session aims to foster discussion, knowledge exchange and shared learning among researchers and methodologists around issues related to increased use of mobile devices for survey completion. The format of the session will be designed to encourage interaction and discussion between the presenters and audience.
The session is proposed by the National Centre for Research Methods (NCRM) Research Work Package 1 ‘Data Collection for Data Quality’. ‘Data Collection for Data Quality’ is funded by the UK Economic and Social Research Council (ESRC) and led by a team from the University of Southampton. The project investigates amongst other topics mobile device use in mixed-device online surveys.
Keywords: data quality, online survey, total survey error, sensor data, sources of error
Uptake and Data Quality in UK Mixed-Device Online Surveys: Results from Experiments in the ONS Online Household Study
Dr Olga Maslovskaya (University of Southampton)
Professor Gabriele Durrant (University of Southampton) - Presenting Author
Professor Peter Smith (University of Southampton)
Social surveys are increasingly conducted via online data collection. They also started embracing smartphones for data collection. In the UK, there is a significant move towards online data collection, including the ambition to move established household surveys such as Labour Force Survey (LFS) as well as the next UK 2021 Census online. Not much research has been conducted so far in the UK to address uptake and response quality among the general population in mixed-device online surveys. This research is timely and fills the knowledge gap in these areas.
We use Office for National Statistics (ONS) data for LFS online experiments (Test 1 and Test 2) which were conducted in 2017. The main aim of these experiments was to move LFS online.
Descriptive analysis and then linear, logistic or multinomial logistic regressions are used depending on the outcome variables to study data quality indicators associated with different devices in the survey. The following data quality indicators are assessed for different devices used by respondents: break-off rates, response latencies, timeout rates, restart rates and differential reporting.
The good news is that we can be less concerned about allowing smartphone data completion in the contexts where mobile-first design is used for questionnaires as data quality are not very different by devices.
The findings from this analysis will be instrumental to better understanding of data quality issues associated with mixed-device online surveys in the UK in general and specifically for informing online versions of social surveys and the next UK Census 2021. The results can help improving designs of the surveys and response rates as well as reducing survey costs and efforts.
Designing a device-agnostic online survey for 17 year olds: Experiences of the Millennium Cohort Study
Dr Emily Gilbert (Centre for Longitudinal Studies, UCL) - Presenting Author
Ms Lucy Lindley (Ipsos MORI)
We live in an age where there are a proliferation of technologies in widespread use in daily life. Increasingly, survey respondents expect to be able to complete online questionnaires on a multitude of devices, including PCs, tablets and smartphones. However, this creates challenges in ensuring the information captured is equivalent across devices, as well as making sure respondents have a positive experience of completing the surveys.
The Millennium Cohort Study (MCS) is a longitudinal cohort study in the UK, following the lives of 19,000 young people born at the turn of the century. As part of the Age 17 Survey, study members were asked to complete an online questionnaire. The questionnaire was optimised for a range of devices, including PCs, tablets and smartphones, as well as for a variety of operating systems and browsers. This involved design choices about question formats, particularly to make all questions smartphone-friendly, as well as decisions about usability of the questionnaire for each device.
This presentation will focus on the development of the device-agnostic questionnaire, and discuss take-up of different devices, selection into devices, and data quality across the different modes of completion.
Looking up the right answer: Errors of optimization when answering political knowledge questions in web surveys
Dr Jan Karem Höhne (University of Mannheim) - Presenting Author
Dr Carina Cornesse (University of Mannheim)
Mr Stephan Schlosser (University of Göttingen)
Professor Mick P. Couper (University of Michigan)
Professor Annelies Blom (University of Mannheim)
Design of grid questions for mixed-device surveys
Ms Deirdre Giesen (Statistics Netherlands) - Presenting Author
The classic presentation of grid question as seen on paper, PCs and tablets is not easily transferable to the smaller smartphone screens. For Statistics Netherlands surveys, we tested in the lab three possibilities for presenting grid questions on smartphones:
1) a paging design: all items of the grid are presented on separate pages, the stem of the grid is repeated on each screen.
2)a stem-fix scrolling- by-respondents design: the stem of the grid is fixed and respondents have to scroll vertically to see all items
3)a stem-fix auto-scroll design: the stem of the grid is fixed, after completion of an item the questionnaire automatically vertically scrolls to the next item.
The labtest was conducted with 20 respondents of various age groups and backgrounds. We observed and recorded completion of the questionnaire on multiple devices and with multiple grid question designs, including the classic grid design on PC and tablet. We evaluated with respondents the validity of their answers and their perception of the response task. The test showed that the tested version of the stem-fix auto-scroll design was not working well; the auto-scroll worked too fast and confused respondents. In the short run it was not possible to improve this.
For an experiment in the School leavers Survey that will run in the first quarter of 2019, we will randomize the presentation of grid questions. Respondents self-select the device they use. For PCs and tablets respondents will either receive a paging design or the classis grid design;for smartphones, respondents will either receive a paging design or a stem-fix scrolling-by-respondent design. Next to data on completion times and breakoff rates, we will collect information on the respondents’ perception of the questionnaire.
In the presentation I will present the results of the labtest and first results of the