Probability-based research panels 1 |
|
| Chair | Mr Darren Pennay (Social Research Centre, Australian National University ) |
In addition to being a sample source for substantive research in social and marketing research, online panels are often used in survey methodology field. In particular, they serve as (1) an object of research itself or (2) a sample source for various experimental studies on survey data quality. This calls for a review of their usage and quality as used in survey methodology. Within this context we present a systematic review that aims to assess (i) the types of online panels which are used in survey methodology, (ii) the quality of these online panels, and (iii) the usage of online panels as a sample source for research on survey data quality.
We use a bibliographic database by WebSM, which is the main information source on web survey methodology (Hewson et al., 2016; Callegaro et al., 2015; Lozar Manfreda and Vehovar, 2007), to select empirical studies from survey methodology that relate to online panels. To address our research questions we define specific coding categories for the type and other characteristics of online panels (e.g., panel size, target population, etc.), and purpose of the usage of online panels. Related to the purpose, we code various (i) panel data quality aspects for studies on the quality of the panel itself, and (ii) methodological questions for studies using panels as a sample source for survey methodology research. We analyzed 66 journal papers and book sections on survey methodology, published in English from 2012 to 2016, which refer to online panels.
The results show that online panels are a valid sample source for various experimental studies in survey methodology. In addition, the systematic review highlights partially or entirely unexplored issues about the quality of panelists’ answers, while coverage and nonresponse issues in online panels are already well researched.
This paper casts light on features characterising the setting of different types of online panels, points out the main indicators of their data quality, and studies the feasibility of the usage of online panels for survey methodology. A more general aim is to increase the knowledge on the usage of online panels, which is one of the current and future trends in survey research.
The MESS project facilitates advanced data collection using the Internet, new measurement devices, and the latest technology. The project is carried out by CentERdata (Tilburg University, The Netherlands). The core element of the project is the LISS panel: an Internet panel specifically intended for scientific research (Longitudinal Internet Studies for the Social sciences).
The LISS panel consists of approximately 8000 individuals who participate in monthly Internet surveys and advanced experiments. The panel is based on a random sample drawn from the population registers. Persons not included in the original sample cannot participate, so there can be no self-selection. People without a computer or Internet connection are provided with equipment to participate, so the offline population is included as well.
Every month, LISS panel members complete online questionnaires of about 30 minutes in total. They receive a monetary incentive for each completed questionnaire. One member of the household provides the household data and updates this information at regular time intervals.
In this presentation we discuss the initial procedures for setting up such a panel (i.e. sample and recruitment of panel members) and how to maintain it. We show results of several incentive experiments, how this is helpful in the recruitment phase, how it can help prevent attrition during panel participation and the effects on overall response. We also discuss the results of our ‘sleepers study’, an experiment on encouraging and maintaining participation in the LISS panel (using letters, incentives and feedback) to minimize attrition.
The Understanding America Study (UAS) is a probability-based internet panel. Panel members are recruited exclusively through Address Based Sampling; respondents without prior access to the internet receive a tablet and paid broadband internet access. The UAS started in 2013 and currently comprises about 6000 panel members 18 and over. Panel members usually answer one or two surveys per month (with a maximum length of 30 minutes per survey). The result is a very large set of background characteristics available for all panel members, including the complete core instrument of the US Health and Retirement Study, which is administered every two years, personality measures (the big five), and numeracy, cognitive, and financial literacy measures. Respondents receive compensation for their time spent answering questions at a rate of $20 per 30 minutes of interview time. Annual attrition rates are modest (on the order of 7-8% per year). The UAS panel’s response rates, while low compared to historical survey samples, tend to be somewhat higher than the falling rates experienced by many telephone surveys. After an initial multi-step recruitment process (Dillman 2014), on average, more than a third (36%) of our national samples returns the initial paper survey which includes consent to contact them about participating in the panel. After a series of empanelment steps, about 12% eventually become active participants. During our 2016 recruitment drive, we implemented a series of experiments aimed at increasing response and retention at each stage of the process, and in this paper we share results and next steps. We argue that our methods increase the panel’s representativeness of the U.S. population over what can be achieved without providing internet devices. Of the current nearly 6000 panel members, 310 have received an internet connected tablet (5.2%). Similar to Leenheer and Scherpenzeel (2013) for the LISS panel in the Netherlands, we find that in comparison to the Internet households, non-Internet households in our panel are more likely to have low incomes and low education, to be non-white, less healthy, and older (70+). Importantly, the recruiting rate among these households varies little across subgroups, except for elderly who tend to sign up somewhat less. This suggests that with respect to the demographics studied, the recruited non-Internet households are representative of the part of the population without Internet (except possibly with respect to age). Since, moreover, the non-Internet households are very different from households with prior Internet, this suggests that adding a non-Internet sample addresses the coverage deficiency of an Internet-only sample. We report on the methods we use to recruit and retain our panel, as well as lessons learned from our recruiting experiments.
Online survey panels enable cost-effective and timely data collection. However, obtaining a representative sample can be challenging and expensive: sample frames for probability sampling of the general population usually do not include online contact information, and recruitment thus relies on offline contacts that either suffer from low recruitment rates (telephone and mail) or involve large costs (face-to-face). The problem multiplies when researchers embark upon cross-national data collection, where challenges and solutions vary across countries. Random selection of individual respondents within households, for example, poses different challenges depending on the available sampling frames. Instead of setting up a brand new recruitment effort to build a panel, researchers could potentially piggy-back on existing survey projects that have already addressed the issue of selecting and contacting sample units, inviting those who agree to participate in the survey to continue cooperation further. Cost-efficient as this approach may seem, it has been met in the past with concerns about the additional burden for respondents and interviewers. A pilot study was set up to test this “piggy-backing” approach using the European Social Survey data collection efforts. Participants of the CROss-National Online Survey (CRONOS) panel were recruited at the end of the 2016 ESS interviews in UK, Slovenia and Estonia. Participation involved completing six 20-minute online surveys over one year. Respondents were offered gift cards for £5/€5 with every new survey invitation. Those without internet access for private purposes were offered an internet-enabled tablet, training on how to use it, and telephone support. This paper evaluates efficiency of this approach by examining recruitment rates across countries and across different subpopulations. We will analyse representativeness of the recruited sample, comparing those who participated to those who did not in important variables from the ESS interview. The initial recruitment rate varies by country—between 51% in Estonia and 61% in the UK; by age—between 80% for respondents younger than 30 and 46% among those older than 60; by gender—54% for women vs 60% for men; and by internet access—50% for those without internet access for personal use compared to 78%. In addition, this paper will consider the challenges of setting up a cross-national probability-based online panel.
The presentation will look into the processes and outcomes of designing, setting up and maintaining a probability-based online panel in Andalusia. The Institute for Advanced Social Studies (IESA), a unit of the Spanish National Research Council (CSIC), recruited the Citizen Panel for Social Research in Andalusia (PACIS) from September 2014 to March 2015. PACIS members were recruited face to face using the official address list (catastro) to select the units in the sample and off-line members are contacted and interviewed by telephone (landline and mobile). 3.439 people living in 2.809 households compose the panel that has served as a sampling frame for three cross-sectional studies conducted in 2015 and 2016.
The paper describes sampling design and the procedures for recruiting and maintaining the panel. Then we will present the outcomes rates attained at the recruitment stage and on each of the three waves conducted until now. Finally, we analyze the sample composition and potential biases produced by the combination of nonresponse at both the recruitment and the surveying stages.