Affiliations: [a] Vrije Universiteit Amsterdam, The Netherlands | [b] Statistics Netherlands, The Netherlands | [c] Utrecht University, University Medical Center Utrecht, The Netherlands
Correspondence:
[*]
Corresponding author: Paulina Pankowska, Department of Sociology, Faculty of Social Sciences, Vrije Universiteit Amsterdam, de Boelelaan 1105, 1081 HV Amsterdam, The Netherlands. Tel.: +31 20 59 83178; E-mail: p.k.p.pankowska@vu.nl.
Abstract: This paper discusses how National Statistical Institutes (NSI’s) can use hidden Markov models (HMMs) to produce consistent official statistics for categorical, longitudinal variables using inconsistent sources. Two main challenges are addressed: first, the reconciliation of inconsistent sources with multi-indicator HMMs requires linking the sources on the micro level. Such linkage might lead to bias due to linkage error. Second, applying and estimating HMMs regularly is a complicated and expensive procedure. Therefore, it is preferable to use the error parameter estimates as a correction factor for a number of years. However, this might lead to biased structural estimates if measurement error changes over time or if the data collection process changes. Our results on these issues are highly encouraging and imply that the suggested method is appropriate for NSI’s. Specifically, linkage error only leads to (substantial) bias in very extreme scenarios. Moreover, measurement error parameters are largely stable over time if no major changes in the data collection process occur. However, when a substantial change in the data collection process occurs, such as a switch from dependent (DI) to independent (INDI) interviewing, re-using measurement error estimates is not advisable.
Keywords: Data reconciliation, inconsistent data sources, measurement error, linkage error, hidden Markov model, latent class model, dependent interviewing
