ESRA 2017 Programme
|ESRA Conference App|
Tuesday 18th July, 11:00 - 12:30 Room: Q4 ANF3
New technologies to improve the measurement of household finances
|Chair||Mr Alexander Wenz (University of Essex )|
|Coordinator 1||Professor Annette Jäckle (University of Essex)|
|Coordinator 2||Professor Mick Couper (University of Michigan)|
Session DetailsThe emergence of new mobile technologies (including smartphones and tablets, activity trackers, barcode scanners, smart meters, etc), as well as the availability of new data sources (for example from online financial management services such as Mint.com), offer new possibilities for survey research. New technologies can potentially improve the scope and quality of data collected in surveys, and have distinct advantages over existing questionnaire-based methods of data collection.
The measurement of household finances is one particular area that could benefit from data collection using new mobile technologies. Existing surveys measure the financial circumstances of households with detailed question lists on income, wealth, and debts, and use diary methods to collect information on household expenditure. These surveys are not only long and burdensome for respondents, but they also rely heavily on the respondent’s ability to recall financial information, decisions, and behaviors, and are hence prone to recall error.
New mobile technologies make it easier for respondents to provide accurate data, by reducing the need to recall information: mobile devices can be used to capture objective measures, for example by asking respondents to scan payslips or till receipts, or to log purchases with an app at the point of sale. Alternatively, mobile devices can be used for ‘in-the-moment’ surveys that are triggered at regular intervals or in pre-specified GPS locations, again reducing the need to recall information. Some types of data can also be captured passively, without involving the respondent in the data collection process. For example, information about financial transactions could be obtained from financial aggregators that respondents use to manage their finances, or information about consumer behavior from store loyalty cards. Despite these opportunities, researchers must keep in mind that mobile technologies introduce new methodological challenges and sources of error to the data collection process.
In this session, we welcome contributions that discuss applications of new technologies for the measurement of household finances.
We encourage proposals from researchers, including academic research, national statistics and market research, investigating for example:
- Which mobile and other new technologies can be used to collect financial data in surveys?
- What are the sources of Total Survey Error in data collected with new technologies? How do errors affecting representation and/or measurement compare to questionnaire-based data?
- Which methods can be used to maximize participation in data collection using new technologies?
Paper Details1. Real Time Measurement of Household Electronic Financial Transactions in a Population Representative Panel
Dr Arie Kapteyn (Center for Economic and Social Research, University of Southern California)
Dr Marco Angrisani (Center for Economic and Social Research, University of Southern California)
Dr Swaroop Samek (Center for Economic and Social Research, University of Southern California)
Real Time Measurement of Household Electronic Financial Transactions in a Population Representative Panel
Center for Economic and Social Research
University of Southern California
In this paper we report on our experiences collecting data on household financial transactions, balances, and expenditures from electronic records in a population-representative household Internet panel (the Understanding America Study, UAS). The data are combined with survey information from the panel and made available to the research community with minimal delay.
To date, the measurement of financial and consumer behavior mainly relies on self-reports, which are burdensome for respondents and tend to suffer from serious measurement and selectivity issues. Since most individual financial and spending behavior (use of payment instruments, investments, assets and liabilities) is recorded in electronic form, direct access to electronic records potentially both improves accuracy of measurement and reduces respondent burden. In this project, we collect financial electronic records directly through the financial institutions of respondents in the household panel. We have designed and implemented a data collection system, using a contract with a major financial information aggregation firm. Participating panel members have consented to share with us their financial information through the aggregator platform.
Fundamental to this project, by relying on the UAS, we can combine survey information with information from the electronic records. In the paper we document various features of the project: consent rates, completeness of electronic records, and comparisons with self-reports. The comparison with self-reports serves two purposes: it yields insight into the differences between self-reports and electronically measured expenditures, thereby providing information on the accuracy of both, and it documents selectivity of the group that consents to participating in the project (about 60% of those who are asked). Panel members are incentivized to participate by monetary rewards depending on the number of accounts they share with the financial aggregation firm and on how well they keep that information current. The size and structure of the incentives is varied experimentally across respondents so that we are able to determine which incentives are most effective.
2. Participation in a Mobile App survey to collect expenditure data as part of a large-scale probability household panel: response rates and response biases
Professor Annette Jackle (University of Essex)
Ms Carli Lessof (University of Southampton)
Dr Jonathan Burton (University of Essex)
Professor Mick P. Couper (University of Michigan)
Smartphones offer new measurement possibilities, because they can deliver short surveys ‘close to the moment of experience’ and can employ in-built features such as cameras, sensors or GPS tracking. Nevertheless, the use of approaches of this kind beyond small-scale, volunteer-based research depends on the level of uptake that can be achieved and an assessment of any biases. In this paper, we examine participation in a supplementary study designed to measure household expenditure using a receipt scanning App as part of a nationally representative household panel study, Understanding Society. We invited 2,432 members of the UK Household Longitudinal Study Innovation Panel to download an App to report all their spending on goods and services for a month. Respondents were asked each day to scan all receipts received, report spending for which they did not have receipts, or report that they had not made any purchases that day. The incentive scheme included an experiment with a conditional incentive of £2 versus £6 for downloading the App, followed by a daily conditional incentive of £0.50 for using the App, and a £10 bonus for using the App every day. The App data collection and its accompanying web surveys will be completed in December 2016.
We first consider the level of participation in the study at different stages of the data collection. What proportion of the sample had the required technology (a smartphone or tablet) to participate in the task? What proportion had said they were hypothetically willing to participate in such a task at a prior wave? What proportion actually downloaded the App? How did participation evolve over the month? What impact did the conditional incentive for downloading the App have? We then examine biases in the types of sample members who participated, considering social-demographic characteristics, financial position and behaviours, mobile device ownership, usage patterns and self-rated skills, and indicators of cooperativeness collected in previous waves of the panel. This paper will contribute to an assessment of whether a Mobile App survey can be used to improve the measurement of household spending and to the broader debate about the potential for innovative data collection approaches of this kind.
3. Data quality from a Mobile App survey to collect expenditure data as part of a large-scale probability household panel survey
Ms Carli Lessof (National Centre for Research Methods, University of Southampton)
Professor Annette Jackle (Institute for Social and Economic Research, University of Essex)
Dr Jon Burton (Institute for Social and Economic Research, University of Essex)
Professor Mick Couper (Institute for Social Research and Joint Program in Survey Methodology, University of Michigan)
Smartphones offer new measurement possibilities. They can be used to capture hard-to-report behaviours through passive monitoring or by delivering short surveys ‘close to the moment of experience’, thereby reducing reliance on respondent recall. At the same time, they can employ in-built features of mobile devices such as cameras, sensors and GPS tracking to capture new kinds of data that go beyond traditional survey responses. As well as depending on reasonable levels of response, these kinds of approaches will only be widely adopted if their promise of better measurement is delivered.
In this paper we examine the quality of the data collected in a supplementary study designed to measure household expenditure using a receipt scanning App as part of a nationally representative household panel study, Understanding Society. Members of the UK Household Longitudinal Study Innovation Panel were invited to download an App and to report all their spending on goods and services for a month. Respondents were asked each day to scan all receipts received, to report spending for which they did not have receipts or to report that they had not made any purchases that day. Scanned receipts were coded to extract information such as category of spending and product codes of itemized purchases. This approach has the potential to increase the accuracy of reports of household expenditure by capturing easily forgotten shopping occasions and by removing the need for respondents to sum their spending over time, as well as providing more detailed information than could be achieved through a standard survey including linked data from nutritional databases.
This paper evaluates the quality of measurement in two ways. Firstly, we consider outcome quality. We examine estimates of monthly individual and household expenditure, comparing the App data with expenditure data from previous waves of the panel, as well as drawing on respondent self-assessments of the proportion of expenditure missed due to non-participant household members, drop-out and under-reporting. Second, we consider process quality. We examine the pattern of receipts returned over time, the proportion of spending occasions verified by a scan, the proportion that are successfully coded and the quality of coding. If time allows we will consider factors which may explain variations in quality of data such as the socio-demographic characteristics of participants, their financial position and behaviours, their mobile usage patterns and self-rated skills, and indicators of cooperativeness collected in previous waves of the panel.
4. Improving diary keeping of daily expenditures with online diaries
Mrs Laura Erhard (Bureau of Labor Statistics)
The U.S. Bureau of Labor Statistics (BLS) is currently working towards a redesigned Consumer Expenditure Survey (CE) in response to increasing evidence in measurement error, declining response rates, and the emergence of new data collection technologies. Moving away from a two-part survey, with independent household diary and interview samples, the redesigned CE proposes to use a single-sample design. Each household will complete two personal interviews, and each household member 15 and older will complete a one-week spending diary. Each respondent will be given the option to keep an online diary or a paper diary to record their daily expenditures. The online diary was incorporated into the redesign plan to allow greater ease of use for the respondent and to increase the potential for “in the moment reporting”. The use of an online diary also provides the opportunity for real-time monitoring by interviewers, allowing them appropriate information to follow-up with the respondents to encourage the diary keeping task and address any potential issues during the diary keeping week.
Prior to implementing this significant design change, many large and small research efforts have been undertaken to evaluate both individual features as well as the design as a whole. The most recent test was part of the 2015 Proof of Concept test, fielded to test the feasibility of the overall redesign plan. This presentation will provide an overview of this test, focusing on the diary keeping protocols, and highlighting results. We will look at online take-up rate of the diary, quality comparisons between the online diaries and paper diaries, and comparisons of expenditure reporting counts and levels between the two modes. Additionally, we will discuss some of the limitations encountered to date with the online diaries and next steps.