ESRA 2019 Draft Programme at a Glance
If I Text, Will You Respond? Mobile Text Messaging During Recruitment and Data Collection
|Session Organisers|| Dr Ipek Bilgen (NORC)
Dr Antje Kirchner (RTI International)
|Time||Thursday 18th July, 14:00 - 15:30|
As the use of cell phones and smartphones has grown in popularity globally, the use of text messaging (also known as short message service or SMS) in public opinion research has been increasing exponentially within the last couple decades. Due to decreasing response rates, shifts in technology, and technological enhancements around the way people use text messaging, researchers have started exploring new ways to incorporate texting during the survey life cycle. However, these new trends present challenges and researchers are only just beginning to understand the impact of text messaging on total survey error. Accordingly, in this session we welcome contributions on a range of topics related to the use and impact of texting during survey recruitment and data collection from a total survey error perspective. Specifically, we welcome papers including the following topics:
• Issues related to coverage and sample representativeness, including issues related to sampling frames and sample sources for text-based surveys or mixed mode surveys.
• The impact of text recruitment on response (e.g., response rates, nonresponse error and bias) and design decisions related to response including but not limited to
o Tailoring text communications and the content of messages
• Data collection and questionnaire design, including but not limited to
♣ Break-offs and item nonresponse
♣ Measurement limitations
o The use of short message service (SMS) versus multimedia messages (MMS)
o Comparison with other self-administered modes (such as web, mail, IVR)
• Trade-offs and interactions between survey error sources
• Analyses from a survey operations perspective (e.g. day/time of the day, two-way versus one way interactions, etc.)
• Privacy and safety regulations (such as TCPA and GDPR)
• Cross-national and cross-cultural differences
o Implications for the future (the increase in the use of smartphones)
o Implications for developing countries
We are interested in examples from a range of different types of studies including but not limited to mixed mode surveys, online panels, and longitudinal studies that include SMS components. We welcome papers from researchers with a variety of backgrounds across different nations and sectors including academia, government agencies, and industry. The format of the session will be designed to foster discussion between the presenters and audience around issues related to potential uses of SMS methodology within the survey life cycle from a total survey error perspective.
Keywords: SMS, Recruitment, Data Collection, Total Survey Error, Rules and Regulations
The Impact of Advance Text Messages on Outcome Rates, Efficiency and Representativeness of Phone Surveys
Dr Benjamin Phillips (The Social Research Centre)
Mr Darren Pennay (The Social Research Centre) - Presenting Author
Dr Paul Lavrakas (Independent Consultant)
Ms Tina Petroulias (The Social Research Centre)
In jurisdictions where sending advance text messages (SMS) is allowed, they offer the opportunity to increase efficiency and cooperation and response rates of phone surveys. This paper addresses two applications of text messages in conducting surveys: 1) for geographic screening and 2) for prenotification. This paper adds to the research on the impact of SMS prenotification in mobile phone surveys (Dal Grande et al. 2016; Kunz and Fuchs 2012; Steeh, Buskirk, and Callegaro 2007). Like many European countries, the Australian mobile frame lacks geographic assignment, resulting in high screening costs for sub-national samples. We experimented with the use of SMS for screening in a state-level RDD survey, randomly assigning 50% to the treatment (SMS) condition and 50% to the control (no SMS) condition; sample size was n=209,403. In the treatment condition, an advance text message was sent 24 hours in advance of the first call attempt, asking the recipient to reply ‘1’ if they were a resident of the state in which the survey was focused on, ‘2’ if they were not a resident of that state and ‘3’ to opt out. In a second survey, this time national, we randomised respondents into two conditions: an advance SMS with an option to opt out by replying ‘1’ (n=11,758) and no SMS (n=9,884). In a third experiment (n=4,170), respondents were randomised into three conditions: reply ‘1’ to opt out, call a toll-free number to opt out or for more information, and no SMS. We examine level of effort, response, cooperation and contact rates and the demographics of respondents: advance SMS increased cooperation rates and reduced level of effort while impact on response and cooperation rates and representation of younger adults was mixed. Our research adds to the body of evidence on the use of prenotification text messages for telephone surveys.
Mobile Text Messaging and Response Rate: A Randomised Trial
Mr Ole-Martin Vangen (Regional Center for Child and Adolescent Mental Health, Eastern and Southern Norway) - Presenting Author
Mr Knut-Petter Leinan (Regional Center for Child and Adolescent Mental Health, Eastern and Southern Norway)
Dr Tore Wentzel-Larsen (Regional Center for Child and Adolescent Mental Health, Eastern and Southern Norway)
Dr Annette Jeneson (Regional Center for Child and Adolescent Mental Health, Eastern and Southern Norway)
Use of smartphones changes rapidly, and the overall aim of this study is to gain current data on effects of invitation mode to guide recruitment and data collection practices. Previous studies suggest that invitations via text message yields slightly higher response rates compared to invitation via email, especially for respondents who complete the survey on a mobile.
To explore response rates using mobile Text messages, compared to emailing, to send out survey invitations and reminders.
This study comparing the effects of two invitation modes for data collection is a “trial within a trial”.
Web surveys are sent to parents of young children who are in day care centres receiving a program aimed to increase the quality of care. To test the effect of this program intervention, parents are invited to answer a 30-minute web survey pre- and post- intervention. The pre-intervention survey is currently active, and we expect approximately 2300 completed surveys. Invitations to the post-intervention survey will be sent May 2019. In the current pre-intervention survey, we ask respondents if they are willing to register a mobile number so that we might send the post-intervention survey invitation via text message. Consenting respondents will be randomized to receive the post-intervention invitation, with survey link, via text message or e-mail. We will also test the effects of these different invitation modes on response to reminders. Stratified randomization will be used to ensure balance of groups with respect to device used to complete the pre-intervention survey (Mobile versus PC; approximately 60% versus 40% in surveys completed so far).
The recruitment is currently ongoing and we expect to have completed our data collection by Spring 2019. So far, 1087 respondents (49,3% of the total number respondents so far) have registered a mobile number and are eligible for this trial.
Survey recruitment in 160 characters: Composition and Quality of a new mobile sampling strategy
Ms Hannah Bucher (GESIS) - Presenting Author
Mr Matthias Sand (GESIS)
The strongly increasing number of people who own a mobile device with internet access implements new developments into web-survey-research. This increasing rate also has an impact on the devices used to complete a web-survey. Further, some studies investigate differences in demographics of the respondents associated with the used device; therefore it seems interesting to examine the possibilities of recruiting survey units via mobile phones.
As previous studies either used mobile phones as a dedicated sampling frame for CATI-surveys or used text messages to invite participants of an access panel to take part in specific web-surveys, we combined these approaches and investigate a new sampling approach for web-surveys: The recruitment of people for a Web-Survey via a mobile RDD-mobile phone sampling with an invitation via text message that contains a tiny URL to the questionnaire. This would generally also allow for frequentist estimation. Hence, this paper aims to examine the possibilities of such recruitment methods. This new approach will, therefore, be discussed, regarding times and days of contact. Of particular interest is furthermore the sampling population that can be recruited by this approach as well as its outcome rates and costs.
Preliminary results indicate that this sampling method is not very suitable for questioning the broad population, due to problems occurring at different levels:
On the one hand, considerable technical problems arose in the process of verifying automatically generated mobile phone numbers (HRL-lookup). On the other hand, less than one percent of the sent text messages lead to an actual interview.
Our research is the first to investigate a new probabilistic recruitment strategy for mobile web surveys. Therefore, it contributes substantively to the exploration of new survey recruitment strategies to replace or complement existing recruitment strategies that increased in costs such as face to face surveys or decrease in quality such as
Text Message Invitations and Reminders for Smartphone Optimized Web Surveys
Professor Ioannis Andreadis (Aristotle University of Thessaloniki) - Presenting Author
This paper examines the findings from a smartphone optimized web survey, when all invitations and reminders are sent as text messages via the short message service (SMS) to the mobile phones of the target group. The web survey under study was conducted for the National Theatre of Northern Greece (NTNG), the largest theatrical organization of Greece comprising of five winter venues and two open-air theatres. The list of mobile phone numbers was provided by the organization and the aim of the survey was to collect data from the people who have attended events organized by NTNG.
The paper summarizes the results of an experiment conducted among respondents: several groups of respondents have been created randomly. In each group, the time of the day the invitations and reminders were sent, the length of the period between reminders and the text of the invitations and reminders have been manipulated. The findings suggest that when the mobile phone numbers of the target group are available, although the limited length of the SMS texts makes it more difficult to convince respondents to participate in the survey, it is feasible to conduct a successful large-scale web survey using SMS as the only contact mode. The paper concludes with suggestions based on the experiment findings about the best practices that produce the maximum results for a smartphone optimized web survey when all invitations and reminders are sent via SMS.
If I Text, Will You Respond? A Meta-Analysis of the Impact of Text Messaging During Recruitment and Data Collection on Total Survey Error
Dr Ipek Bilgen (NORC)
Dr Antje Kirchner (RTI International) - Presenting Author
As cell phones and smartphones have grown in popularity globally, the use of text messaging (aka short message service or SMS) in survey research has been increasing exponentially within the last couple decades. Due to decreasing response rates, shifts in technology, and technological enhancements around the way people use text messaging, researchers have started exploring new ways to incorporate texting during the survey life cycle. However, these new trends present challenges and researchers are only just beginning to understand the impact of text messaging on total survey error. As such, this paper investigates in which context(s) text messaging presents a viable alternative to other modes of recruitment and data collection from a total survey error perspective.
Conducting a meta-analysis of published and unpublished studies, we first analyze the impact of text message recruitment and/or data collection on survey response rates and nonresponse error. We also assess the challenges of text message recruitment and data collection associated with coverage and sample representativeness, including issues related to sampling frames and sample sources. We then investigate the effect of text message questionnaire design decisions on measurement error. As part of our analyses we are not only investigating each error source individually but also account for the trade-offs of and interactions between survey error sources also in comparison to other self-administered modes. Additionally, we examine text message use within the context of mixed mode methodology.
To fully understand the potential of text message surveys as a tool for future data collection efforts, the above analysis also includes an assessment of privacy and regulatory issues specific to text messaging (e.g., TCPA and GDPR), and their implications on data quality and efficiency also from a cross-national and cross-cultural perspective. The paper concludes with a set of recommendations regarding the use of text messaging in surveys.