Assessing risk of non-response bias and dataset representativeness during survey

1 March 2016

Speaker: Dr Gabriele B. Durrant, Department of Social Statistics and Demography, University of Southampton
Co-authors: Jamie Moore, Solange Correa and Peter Smith 

"Assessing risk of nonresponse bias and dataset representativeness during survey data collection"

Survey methodologists have shifted away from using response rates as an indicator of survey non-response bias. Instead, monitoring of response within and across sample subgroups is recommended, utilising representativeness indicators that measure similarity between respondents and sample in terms of variation in sample response propensities. Such indicators, which estimate propensities with a statistical model and require attribute information on both respondents and non-respondents, can be decomposed to assess impacts of variation associated with different attribute covariates. When coupled with call record paradata detailing interview attempts, they can be used to monitor response during data collection, informing adaptive strategies in which methods may be modified to maximise data quality and minimise costs.

Practitioners are increasingly interested in these methods, but gaps in our knowledge exist. For example, representativeness trajectories during data collection in different surveys have not been compared. Consequently, it is unknown whether it is possible to generalise points beyond which increases in representativeness are minimal and methods should be modified.

This paper utilises R-indicators and Coefficients of Variation of response propensities to describe representativeness trajectories during data collection across several surveys, including covariate level indicators. Furthermore, ‘phase capacity’ points are identified that may inform time points for changing the data collection strategy or the stopping of data collection.

The research makes use of the UK 2011 Census nonresponse link study, which links census attribute data to call record paradata from three UK social surveys. The results have implications for optimising data collection and efficiency savings. If time allows, the presentation will also discuss the assessment of the risk of nonresponse bias using data from Understanding Society.