ESRA 2017 Programme

Tuesday 18th July      Wednesday 19th July      Thursday 20th July      Friday 21th July     

     ESRA Conference App


Friday 21st July, 13:00 - 14:30 Room: Q4 ANF3

Measurement issues in household surveys

Chair Dr Tobias Schmidt (Deutsche Bundesbank )
Coordinator 1Dr Junyi Zhu (Deutsche Bundesbank)
Coordinator 2Dr Alessandra Gaia (ISER, University of Essex)
Coordinator 3Dr Jonathan Burton (ISER, University of Essex)

Session Details

While for policy evaluation the provision of high quality survey data on household finances (i.e. income, wealth, expenditures, benefits take up, etc.) is crucial, survey error on this topic seems not negligible: empirical evidence from the UK suggests that income is underreported at the extremes of the distribution and that this is one of the items with higher levels of non-response.

Within the “Total Survey Error” conceptual framework, various sources of error may impact on estimates of household finances. For example, non-response error may arise if top income earners are less likely to participate in surveys; coverage error may arise when homeless people are excluded from the sample frame; and measurement error may happen when sample members fail to recall all sources of household income or misreport benefit take-up due to social desirability.

It is unclear which are the main mechanisms leading to survey error in estimates of household finances and how different mechanisms interact with each other. On one hand, surveys on household finances are often long and burdensome for respondents; on the other hand, these studies are also prone to recall error – as they often rely heavily on the respondents’ ability to recall financial information – and to social desirability bias, as they usually rely on the respondents’ willingness to reveal sensitive information, such as income and benefits take-up.
Assessing which mechanisms lead to survey error in data on household finances is of utmost importance to choose the best strategies for improving data quality as different methods have different levels of effectiveness depending on which mechanisms influence data quality. For example, “forgiving introductions” can lower social desirability, but may not be particularly effective in reducing recall bias.

We encourage submissions from academic scholars, researchers from national statistical offices, and researchers from market research companies on the challenges in estimating household finances.

Specifically, we are interested in:
- research (including cross-countries comparisons) on the different mechanisms (e.g. recall bias, response burden, social desirability) leading to survey error in measures of household finances;
- the sources of Total Survey Error in survey data on household finances (i.e. non-response error, coverage error, measurement error, processing error, etc.);
- methods to reduce Total Survey Error in data collection on household finances (e.g. “gamification” to lower response burden, “indirect questioning techniques” to lower social desirability bias, etc.);
- the use of administrative data to enhance data collection

Paper Details

1. Measuring income in household surveys: evidence from a collection of experiments
Dr Paul Fisher (Institute for Social and Economic Research, University of Essex)
Professor Thomas Crossley (University of Essex and Institute for Fiscal Studies)
Dr Alessandra Gaia (Institute for Social and Economic Research, University of Essex)

Income data collected as part of household surveys is critical for the study of material living standards, however, it is known to suffer from misreporting. We present findings from a series of experiments implemented as part of a UK household panel survey – the Understanding Society Innovation Panel - designed to reduce misreporting.
Under-reporting of receipt of state transfers (welfare benefits) is common in household surveys. This can be due to recall error or social-desirability bias. Where a respondent does report the receipt of a state transfer, they may fail to remember the correct amount or period or make an error during the interview. Our first innovation is to use an end-of-module “summary screen” to allow respondents to review and edit their reports in-interview. We assess the effects of the “summary screen” on under-reporting of benefit receipts and the accuracy of reported amounts (through examining the presence of outliers, rounding, and comparing them to the official benefit rates).
Misclassification of state transfers is also possible. This occurs where a respondent knows they receive a benefit, but they do not know the name, or confuse it with a similar sounding one. eg. in the UK, respondents may confuse “carers allowance” with “attendance allowance”. Our second innovation is to experiment with a series of soft-checks that follow-up where a respondent reports receiving a state transfer but where they are unlikely to meet the eligibility criteria for that transfer.
Third, we also experiment with two questionnaire designs in the collection of state transfers. The first is characterised by the use of filter questions, which are commonly relied on in household surveys to reduce respondent burden. A weakness of filter questions is that reporting errors on the filter question mean that eligible respondents do not pass the filter. In the case of state transfers, this would lead to under-reporting. We compare the design with filter questions to a second simpler design. In the later, respondents are sequentially presented with a series of showcards listing specific types of state transfer without the use of filters.
In addition to assessing the effectiveness of each innovation independently, we analyse the effects of them on the total individual income distribution. We analyse various distributional measures including measures of poverty and inequality.
Finally, we also perform experiments which assess the reliability of before vs. after tax income measures. We conclude by providing guidelines on best practice for survey designers.


2. The Growth and Sensitivity of Wealth Concentration Estimates
Dr Jesse Bricker (Federal Reserve Board)

Income and wealth concentration is increasingly viewed as a potential source of political and macroeconomic instability. A flurry of research using income tax data has shown that income is becoming more concentrated (Piketty, 1999, Piketty and Saez, 2003). Though the same forces that increase income concentration may also increase wealth concentration—for example, through saved income (Saez and Zucman, 2016)—the available wealth data are inconclusive about some of the basic facts, such as how fast wealth concentration is growing.

In the U.S. there are no micro wealth data with coverage comparable to income tax data, and wealth concentration estimates are based on three other data sources. First, household surveys measure wealth, and those with a wealthy oversample—such as the Survey of Consumer Finances (SCF) —can credibly represent the top of the wealth distribution (Bricker et al, 2016, Kennickell, 2011). Second, wealth concentration can be inferred by capitalizing income tax data into a wealth estimate (Saez and Zucman, 2016, Greenwood, 1983). Third, data from so-called “rich lists” (for example, the Forbes 400) are used to estimate parameters from a Pareto distribution and interpolate the top of the wealth distribution with the aid of survey data (Vermeulen, 2015).

But how fast is wealth concentration growing in these data? Purportedly, the level and growth of concentration differs between methods in recent years (Saez and Zucman, 2016). The top 1 percent in 2013 hold 43 percent of wealth when wealth is inferred from income tax data, but hold just 36 percent when wealth is measured in the SCF survey. While much of the difference in levels is due to measurement choices (Bricker et al 2016), there is a clear difference in growth: wealth concentration has grown from 34 to 43 percent over the past decade when it is inferred from income tax data, but has grown much more modestly---from 34 to 36 percent---when measured in the SCF (figure 1).

However, we show in this paper that under reasonable assumptions there may be no difference in wealth concentration growth between these methods. We begin by exploring the degree of uncertainty in wealth concentration estimates from each method and show that concentration measures derived from capitalized income tax data are sensitive to small changes in assumed rates of return, especially when the return is low (Kopczuk, 2015), and using annual rather than permanent income. The ostensible benefit of inferring wealth from income tax records is the strong coverage at the top of the distribution that the tax data provide. But, this benefit is overwhelmed by the costs of using a model used to predict wealth (figure 2).

Over the past decade, in fact, the variability in SCF wealth concentration estimates---measured by sampling and non-sampling variability---is smaller than the variability from inferred wealth, even when including estimation variability from correcting for under-coverage through a Pareto distribution assumption. Overall, a survey with strong oversampling gives more precise estimates of wealth concentration than wealth inferred from income tax data.


3. Extrapolating the income and wealth distributions – a joint distributional approach
Dr Viktor Steiner (Free University, Berlin)
Mr Junyi Zhu (Deutsche Bundesbank)

We use the marginal and bivariate Pareto distributions to extrapolate both income and wealth distributions. Evaluation is performed by comparing the top marginal distributions using either marginal or joint approaches. We further introduce the observed top distributions from income tax statistics and manager magazines as the evaluation information as well as the tail distributions in estimating the Pareto distribution. Two aspects of questions can be answered by using the extrapolated distributions: simulation studies on taxation policies (interaction between capital and labor income taxes as demanded by inflation driven bargaining, capital income or wealth taxation); whether differences in saving rate or rates of return are shaping the wealth inequality.