ESRA 2017 Programme

Tuesday 18th July      Wednesday 19th July      Thursday 20th July      Friday 21th July     

     ESRA Conference App


Friday 21st July, 13:00 - 14:30 Room: N AUD4

What do we tell them, and how? Reviewing current practices for communicating about surveys with respondents 2

Chair Mr Alfred Tuttle (US Census Bureau )

Session Details

Surveys generally involve more than just the interaction between a respondent and the survey instrument or interviewer. Surveyors must also communicate information about a survey that prepares respondents for response – what types of information will be collected, how to access the survey, time frame of completion, availability of modes, etc. In addition, surveyors may provide information intended to convey the importance of the survey – planned and past uses of results, names of important sponsors and other stakeholders, etc. In short, surveyors attempt to affect respondents’ decisions 1) to respond 2) in a timely manner and 3) with the effort needed to ensure the reporting of accurate data.

This long-standing problem is complicated by rapid technological and social changes. Today’s surveyors, faced with new challenges such as declining response rates, changes in communication behaviors among target populations, competition from telemarketers for respondents’ attention, the emergence of cyber-security threats, etc., must adapt their communications to address changes in the survey environment.

This session will explore surveyors’ experiences with developing, implementing, and evaluating survey communications. Of particular interest are (but are not limited to):
• Empirical evaluations of the effectiveness of survey communications
• Development and evaluation of messages related to privacy, data security, use of administrative records, and other recent topics of concern to respondents and surveyors
• The use of newer communication modes such as email, SMS, social media, etc., as well as novel uses of traditional modes
• Research into respondent behaviors with regard to processing survey materials and making decisions about cooperation
• Application of theoretical approaches to crafting messages to positively affect respondents’ decision to participate, e.g., social exchange theory, cost-benefit analysis, principles of psychological influence, etc.
• The impacts of imposed requirements – legal statutes, informed consent policies, institutional review boards, etc. – and how surveyors adapt their communications to meet these requirements

Paper Details

1. Communication strategies in the German sample of the European Social Survey (ESS)
Dr Michael Weinhardt (Bielefeld University)
Mrs Jule Adriaans (Bielefeld University)

There is a relational dependency between survey research and the wider public. On the one hand, rigorous research provides empirical grounding for public debates and political decisions. On the other hand, funding for large-scale, high quality survey research depends on the fact that decision-makers and politicians actually acknowledge this importance. In addition, it will be easier to persuade potential respondents to take part in a survey if they are convinced that they participate in something worthwhile that has actual, real world impact. For these reasons, large-scale research infrastructures such as the European Social Survey (ESS) need to think carefully about how they present themselves to the world. This paper evaluates the communication efforts targeting respondents and the general public in the German sample of the ESS. In addition to standard practice, such as advance letters for respondents and press releases for the wider public, three innovations were introduced for round eight of the study: (1) a short, web based film explaining the main features of the survey, (2) targeted press releases to local newspapers in the local area of primary sampling units, and (3) the use of social media profiles on Twitter and Facebook. First, a short video was produced to explain the importance and advantages of the ESS to the general public, but also to support fieldwork efforts. The video was shown at interviewer briefings to boost interviewer motivation and is placed on the website specifically dedicated to potential respondents. Getting possible respondents to see such a video in a face-to-face survey such as the ESS is however a main challenge because the mode of first contact is in writing via the advance letter sent by the survey agency. Respondents could only be pointed to the video by a link in this letter. We investigate the success of this enterprise using page views of the website showing the film. In addition, we present the results of an experiment regarding the placement of the link in the advance letter. Second, we evaluate the effect of press-releases sent out to local newspapers in the specific area of primary sampling units. Before such a measure may impact on actual response behavior, newspapers must actually decide to write something about the survey. We evaluate success at this step using findings from a screening of the targeted newspapers after the press releases were sent. Third, Twitter and Facebook accounts were set up, complementing a study-specific website, in order raise overall awareness of the study. While Twitter is more suitable for targeting media and communication experts, Facebook is potentially more valuable in reaching out to “ordinary citizens”. We present our considerations and challenges in finding and attracting followers, discussing our experience of experimenting with types and topics of content, based on (non-)reactions of friends and followers. We discuss advantages and challenges related to each measure presented here in terms of impact and cost-effectiveness as parts of an overall communication strategy for repeated, cross-sectional population surveys.


2. Do Focus Group Participants Mean what they Say? A Field Test of Mailing Materials Updated Based Qualitative Feedback
Dr Rachel Horwitz (U.S. Census Bureau)
Mr John Finamore (National Science Foundation)
Mrs Jennifer Tancreto (U.S. Census Bureau)

All types of surveys are experiencing declining response rates, but longitudinal surveys also have to combat attrition between cycles. Effectively communicating the importance of the survey and the value it provides can help attract new respondents and keep old respondents invested. Additionally, as technologies change and respondents are busier and more difficult to reach, survey contacts must adapt as well. The research presented here begins with a description of the evolution of the contact materials and mailout strategy for a national longitudinal survey, including the content of the letters, the envelopes, and the number of mailings, and culminates with an experiment that tests an updated set of materials and mailout strategy aimed at increasing survey appeal while reducing burden and cost. The experiment reduces the total number of mailings, while increasing the variety of mailing pieces (in look, format, and content) and making the message more motivational. Using feedback from focus groups and cognitive interviews, the experimental messages give specific examples of how the data are used and who uses them, including an infographic based on survey data from prior cycles designed to show potential respondents the value of the survey and to provide feedback to prior respondents. By comparing the results from the existing materials to the updated materials, we will provide insight into the impact that a package of mailing materials with a variety of envelopes with specific, visually appealing messaging has on response rates, response timing, respondent burden, and cost.


3. "Whatever decision you make, we would like you to take part in the survey": Asking for administrative data linkage consent in the Next Steps Age 25 Survey
Dr Darina Peycheva (Centre for Longitudinal Studies, UCL Institute of Education)
Dr Lisa Calderwood (Centre for Longitudinal Studies, UCL Institute of Education)
Mr Mehul Kotecha (NatCen Social Research)

There is an increased interest among survey researchers in linking administrative records to survey data. This is valuable because data from administrative records may sometimes be more accurate, or more detailed than it is possible to obtain in surveys.

Linking administrative records to survey data, however, requires the informed consent of the participant and thus makes the communication of the proposed request an important aspect of the study and crucial to obtained high consent rates.

How to best communicate to participants’ the consent request, while appropriately addressing their major concerns and ensuring they have sufficient information for their consent to be fully informed, thus forms a central part of the communication to participants about data linkage.

Asking for consent becomes more challenging in mixed mode studies, and in particular in web surveys, in which there is no immediate support from an interviewer for clarification of questions and reassurance of one’s privacy at the time of the interview. The web, however, provides other opportunities such as better content visualisation, likely to help participants absorb key messages quickly, or quick access to other useful sources of information.

Administrative data holders and ethical committees form another major determinant of the consent process and protocol development setting further legal and ethical requirements.

This paper will discuss how we addressed the challenges in asking for administrative data linkage consent in the Next Steps Age 25 Survey. Formerly known as LSYPE 1, Next Steps is a longitudinal study of over 16,000 young people in England born in 1989/1990. The most recent sweep of data collection took place in 2015-2016 when respondents were aged 25/26, using a sequential mixed mode design involving web, telephone and face-to-face interviews. Seeking data linkage consent was a major component of the study. Nine linkages were sought from 4 different domains (education, health, economic, crime). A 16-page leaflet about data linkage was produced and sent to all participants, and all participants received a letter confirming which consents they had given after the interview.

As data linkage consent was such a major part of the study, the communication of the consent request, as well as the broader process and protocol were a key area for testing prior to the main data collection stage. Exploratory qualitative work tested its general acceptability to participants and contextual issues; all participant materials and operational procedures were tested in the study pilot and approved by data holders and ethical committees.

We will summarise the main findings of this developmental work, particularly in relation to the design of the participant communication materials. We will also cover how we utilised the opportunities of the web mode, for example to embed video content and hyperlinks to help screens and external websites, to enhance participant communications. Finally, we will also report the experience we had with its main stage data collection implementation, and the consent outcomes we achieved and use the web interview paradata to better understand the experience of providing consent online.


4. Advance mailing experiments on the Crime Survey for England and Wales
Mr Luke Taylor (Kantar Public)
Miss Catherine Grant (Kantar Public)

The Crime Survey of England and Wales (CSEW) is a continuous face-to-face cross sectional study conducted by Kantar Public on behalf of the Office for National Statistics. Advanced letters are used on the CSEW to pre-notify sampled households that they have been selected for participation and that an interviewer will be calling round to conduct an interview. These letters aim to increase co-operation and (hopefully) participation by ‘selling’ the study to the residents of each sampled household.

Improvements to the advance mailings therefore have the potential to increase participation in the study, and even relatively small improvements in the mailings would have two clear benefits. Firstly, the potential to make estimates from the study more robust (a higher level of response reduces the risk of non-response bias) and, secondly, to reduce costs (maximising response to the original fieldwork phase reduces the need for costly reissues).

Around 53,000 addresses are issued into field each year for the CSEW; this large number of issued addresses provides us with sufficient power to experimentally test the impact of relatively minor changes to the advance mailings. We have recently run experiments on the survey to test the impact of:

(1) Adding the following motivational statement to the advance letter: “Most people we ask to take part in the survey agree to help us”. This statement was developed using principles from the nudge theory, on the basis that by normalising the act of participating in the survey, sampled individuals would be more willing to take part, improving the original issue response rate.

(2) Getting rid of the survey leaflet which traditionally accompanies the letter (which includes additional background on the study and assurances relating to privacy and information security). We tested removing the leaflet, but including a summary of the information on the back of the letter. The hypothesis tested in this experiment was that removing the leaflet (which would reduce costs) would not have a negative impact on the study.

This paper examines the impact which these experiments had on response rates, the number of contact attempts required for interviews to be achieved, and the proportion of respondents that reported having read the advanced mailings.