Tuesday 14th July      Wednesday 15th July      Thursday 16th July      Friday 17th July     

Wednesday 15th July, 09:00 - 10:30 Room: HT-102

Potentials and constraints of weighting to improve survey quality
Convenor	Dr Stephanie Steinmetz (University of Amsterdam )
Coordinator 1	Professor Kea Tijdens (University of Amsterdam)

Session Details

As scientific surveys are indispensable instruments of social research, their results can impact significantly on public opinion formation and official decision making. Therefore, the accuracy of a survey is of paramount importance. However, it is determined by many aspects of the survey process, including sampling, patterns of response and non-response as well as survey design and data collection procedures.
The introduction of the web as a new mode of data collection, however, has triggered a heated debate on their scientific validity and the degree to which their results can be generalised for the whole population – in particular for non-probability web surveys. To deal with problems arising from sample biases, it has been emphasised that weighting procedures are necessary for generalising web survey results for the whole population, even though particularly the implications of propensity score or other advanced adjustment techniques are still under discussion. As the application of such adjustment procedures has produced rather diverse results, it is not entirely clear whether the representativeness of web surveys can be improved through weighting.
The session aims to evaluate the potentials and constraints of different adjustment procedures to improve survey quality. Papers are invited which address the methodological foundation of different adjustment techniques, and provide insights into their simple and advanced application for offline and online surveys. Contributors are particularly encouraged to explore, compare and critically discuss the efficiency of different weighting procedures to improve survey quality.

Paper Details

1. Representativity Through Statistical Adjustment via Matching? Ways to Control for Sampling Effects of Different Modes in the Context of the German Federal Election 2013
Mr Sven Vollnhals (University of Mainz)

High refusal rates in probability samples and the rising use of non-probability access panels threaten the quality and comparability of surveys from different modes of data collection.

Although the efficiency is often disillusioning, weighting remains a possible solution to achieve (conditional) representativity between different modes.

The paper evaluates the possibility of balancing distributions from different modes (F2F, telephone, internet) of the “German Longitudinal Election Study” (GLES) through “Propensity Score Matching”. The potential and limits of different matching algorithms (e.g. “Genetic Matching”) in reducing discrepancies on auxiliary and key variables of interest will be shown and evaluated.

2. Improving the Quality of Volunteer Web Panels: Evaluating Different Weighting Methods for the Dutch Leisure Panel
Mrs Stephanie Steinmetz (University of Amsterdam)
Mrs Vera Toepoel (University of Utrecht)
Mrs Annamaria Bianchi (University of Bergamo)

With the rise of the Internet more and more data are collected via volunteer web panels, which are not representative for the general population. Post-survey adjustment techniques are often used to improve data quality. However, which methods work best and which variables need to be taken into account differ per survey. To structure such methods for the volunteer Dutch Leisure Panel, different weighting methods are evaluated and results are compared to outcomes of a probability-bases survey. Finally, the effectiveness of weighting for volunteer panels and if or when probability-based panels should be preferred given cost-efficiency will

3. The dubious zero-correlation between weighting variables and target variables: A false way to upgrade survey power?
Dr Koen Beullens (Centre for Sociological Research - KU Leuven)
Professor Geert Loosveldt (Centre for Sociological Research - KU Leuven)

Survey statisticians have introduced the correlation between the weighting and target variable as an important consideration for the selection of weighting variables to inform the weighting strategy: whenever this correlation is zero, the mean of the target variable will not change, only the standard error potentially increases. This is why it is often proposed to exclude such variables in the weighting model. This paper seeks to demonstrate that such a selection criterion to select weighting variables is troublesome as it artificially or even fraudulently tries to increase the power of the sample or analysis.

4. How good are VAA to estimate voters' positioning? Some explorations for Greece and Iceland
Dr Ioannis Andreadis (Aristotle University of Thessaloniki )
Dr Stephanie Steinmetz (University of Amsterdam)
Dr Annamaria Bianchi (University of Bergamo)
Dr Gudbjorg Andrea Jonsdottir (University of Iceland)

Many researchers use Voting Advice Applications (VAA) data to estimate voters’ position on the statements used in VAAs. VAA sites can attract thousands or even millions of users and generate large and cheaply collected datasets. However, as these data are collected on a non-probability basis they are not representative of the total population. The paper explores the sample bias of HelpMeVote in two countries, Iceland with a high internet penetration and Greece with a low penetration. An attempt is made to overcome the problem of the non-representativeness in HelpMeVote by applying different post-adjustment techniques.

5. Estimating the Number of Farms in the U.S. Using Capture-Recapture
Ms Denise Abreu (USDA National Agricultural Statistics Service)

USDA’s National Agricultural Statistics Service (NASS) conducts the June Agricultural Survey (JAS) annually. Using the Census Mailing List (CML) in years that the census of agriculture is conducted and the NASS list frame in non-census years, capture-recapture methods have been developed to improve the JAS estimates of the number of U.S. farms and land in farms. Although an operation's farm status is determined in census years, the status is not always fully determined in non-census years, providing additional technical challenges. A comparison of the capture-recapture estimates using the CML and the NASS list