Multilayer perceptron models for the estimation of the attained level of education in the Italian Permanent Census

1.Introduction

The Italian National Institute of Statistics (Istat) is moving towards a register-based production system. The new Italian Census is one of the most important outcomes of this statistical program. The Attained Level of Education (ALE) is one of the output figures provided by the Census. Istat released official estimates of the ALE for the 2018 and 2019 resident population by using predicted values for each unit in the Italian Base Register of Individuals (BRI) [1, 2].

ALE is the result of a multisource approach: it makes use of administrative data, the 2011 Italian (traditional) Census, and sample survey data. Log-linear models are applied to estimate ALE. Due to the complexity and heterogeneity of the available information, this approach requires an expensive initial phase of data analysis and treatment to achieve an accurate prediction. Moreover, different imputation procedures must be combined to deal with sub-populations characterized by different amounts of information.

In the last years, machine learning (ML) techniques have been applied in many contexts (including official statistics, see [3]) with the aim of improving predictions, especially when very large collections of data can be leveraged. The advantage of using such techniques is also in their almost automated application to data. These opportunities motivated the study of ML for the ALE prediction task, with the twofold objective of improving estimation accuracy (given the high amount of available data) and reducing human workload (given the efforts needed for data treatment and sequential usage of complex models in the official procedure).

In recent years, Istat has gained considerable experience in the use of neural networks to extract statistical information from extremely large and unstructured data sets generated by non-traditional sources. In particular, the potentialities of deep-learning models like convolutional neural networks (CNN) have been investigated in [4, 5, 6] for the treatment of images and natural language. In this study, we focus instead on the Multilayer Perceptron model (MLP). Early applications of the MLP model in the field of official statistics can be found in [7, 8, 9]. More recently, MLP and other machine learning techniques have been studied within the HLG-MOS group [10].

In this paper, we compare predictions of the ALE variable resulting from MLP models to the official ones. First, predictions are computed within the same informative setting, i.e., the same preliminary data analysis and the resulting data elaboration (variable selection and treatment) used for the official procedure is used for MLP. The same covariates are used, with the same level of aggregation. The aim of this experiment is to evaluate the capacity of MLP to improve the quality of predictions. In a second experiment, the MLP model is used in its most natural context, that is, the data is fed to the MLP almost without any selection and aggregation. This second study is useful to assess the possibility of making predictions of ALE in a more efficient and automated way.

The paper is structured as follows. Sections 2 and 3 describe the available data and the procedure adopted to produce official ALE estimates. Section 4 introduces the MLP model and its application to our problem. The experimental study and the results are illustrated in Section 5. Some conclusions and future studies are discussed in Section 6.

2.Basic information about the attained level of education in the Italian Census

The BRI is a comprehensive statistical register storing individual data gathered from various data sources. Core variables – such as place and date of birth, gender, and citizenship – are associated to each unit of the register.

For the ALE prediction procedure, data of different kinds are jointly used: administrative data, 2011 traditional Census data, and sample survey data.

The three sources of data are characterized by different patterns and amounts of information, that is a different set of variables and different classifications of ALE.

The structure of available information is summarized in Fig. 1. Blue cells indicate that the information is available for the specific subpopulation.

Figure 1.

Structure of available information for mass-imputation of the attained level of education at time t.

More in detail, core information from BRI is available for all individuals: age, gender, citizenship, marital status, place of birth and place of residence.

The different availability of information on ALE from 2011 to t-2 determines the partition of the population of interest into three subgroups:

In all the subgroups, data on ALE were reclassified according to the 8-item classification adopted by Istat for the purpose of disseminating Permanent Census data. The classification is as follows: 1 – Illiterate, 2 – Literate but no formal educational attainment, 3 – Primary education, 4 – Lower secondary education, 5 – Upper secondary education, 6 – Bachelor’s degree or equivalent level, 7 – Master’s degree or equivalent level, 8 – PhD level.

ALE, at reference time t, is only known for people interviewed in the Census sample, which is a representative subset of the population of interest. For the 95% of population not in the Census sample, ALE has to be estimated. Although ALE is estimated for each individual in the population of interest (micro level), the aim of the prediction procedure is to reproduce the frequency distribution observed in the sample. Hence, the first interest of the prediction is to maximize the distributional accuracy. Nevertheless, since we are dealing with registers that are characterized by individual-level information, predictive accuracy (micro level accuracy) should be evaluated as well.

3.The official estimation procedure based on log-linear models

The official procedure adopted by Istat in production ([2]) is based on log-linear imputation. As stated in Singh (1988) [11], this method generalizes hot-deck imputation by choosing suitable predictors for forming “optimal” imputation classes. In fact, the approach is based on modeling the associations between variables.

The objective of log-linear models is the representation of the interdependence of the variables in the contingency table. In the following we will refer to the case of three categorical variables (X1,X2,X3), with categories (i,j,k),i=1,…,I,j=1,…,J,k=1,…,K, such that

A log-linear model is a parameterization of the probabilities θ𝑖𝑗𝑘 in terms of variables’ interactions. The link between these two objects is the following. Let n be the sample size, and let η𝑖𝑗𝑘 be the expected value of the sample counts in cell (i,j,k):

Then, a log-linear model is defined by:

under the constraints

When some of the interaction terms (i.e., the λs) are set to zero, different kinds of dependence relationships are defined.

The idea underlying the approach is that of estimating a model for the prediction of ALE at time t (henceforth It) given the values of known covariates X. In particular, we estimate the conditional probabilities h(It|X) and then impute It by randomly taking a value from this distribution.

The conditional probabilities h(It|X) are estimated by means of log-linear models ([12]) as follows.

First, a log-linear model is applied to the contingency table obtained by cross-classifying the variables (It,X) to estimate their expected counts η^i⁢jIt⁢X, from which we estimate the counts η^jX. The estimated conditional probability distribution h^(It|X) is easily obtained by computing η^i⁢jIt⁢X/η^jX. This approach includes as a special case the random hot-deck when all the interactions between variables are included in the model (saturated log-linear model), but it has the advantage of allowing the use of more parsimonious models as well, by testing the associations among variables. This is an important characteristic especially when the number of variables and contingency table’s cells increase.

It is worthwhile noting that different log-linear models are used within groups A, B and C, mainly because of the different available information. As already remarked, in group A, a log-linear model is estimated by using only administrative data, while for the other groups, log-linear models are estimated by using survey data as well.

For each subpopulation (A, B and C), a step of variable selection was performed to detect the combination of covariates to be included in the model. The best log-linear model is chosen by means of cross-validation. More specifically, log-linear models for each sub-population are built to estimate the following conditional probabilities:

Apr is an auxiliary information on ALE coming from an administrative source and it covers a particular subpopulation of individuals: those who changed their place of residence after 2014. Moreover, it is a self-declared information with a low level of quality, and it comes with a more aggregate classification (4 levels).11 We decided to use ALE from the apr source only in subpopulation C, where we have not any other information on ALE.

Sirea refers to people who were targeted but not surveyed by the 2011 Census and were later detected by post-Census operations carried out in agreement with Italian Municipalities.

An in-depth analysis of the independent variables was necessary to appropriately reclassify the covariates in the model. In particular, suitable age levels were identified by taking into account the structure of the Italian school system and a classification in 14 levels was adopted.22 Citizenship was aggregated into Italian/Not Italian to reduce the number of categories.

4.MLP for the prediction of the attained level of education

As anticipated, this work investigates an alternative approach to the ALE prediction and imputation problem, based on a MLP model.

The MLP (the acronym stands for Multilayer Perceptron) is a supervised machine learning algorithm. When it encompasses more than one hidden layer, the MLP constitutes the simplest example of deep neural network (DNN).

In general, a neural network consists of a network of elementary computing units (artificial neurons) connected according to a specific network topology. In a neural network, each artificial neuron j is fed with an input vector xj and returns an output yj=f⁢(wj⁢xj-bj), where wj is the vector of weights associated with each input and bj is a scalar weight (bias). The function f is the activation function and is a nonlinear function, typically sigmoidal or Rectified Linear Unit (ReLU).

The architecture of an MLP is organized in layers of neurons. The output of the neurons of the previous layer (k-1) is the input of each neuron of the next layer (k).

Therefore a k-th layer neuron returns an output vector hk:

where Wk is the matrix for weights of the neurons of the k-th layer.

Figure 2.

Example of MLP.

In the Fig. 2 we show an example of MLP with two neurons for the input layer, two neurons for the output layer, and 2 hidden layers with four neurons.

The output of an MLP is a composition of the outputs returned by each layer and realizes a nonlinear mapping between the input and output vectors.

According to the universal approximation theorem [13] and subsequent extensions, any mapping between an input vector X and an output vector Y is arbitrarily approximated by an MLP with a sufficiently large number of neurons or a sufficiently deep number of layers.

The algorithm with which the network is trained according to a dataset of examples is the backpropagation algorithm [14].

Given a task, e.g. classification or regression, and a dataset of examples of a mapping ({Xi, Ti}), in general the loss function represents the distance between the mapping performed by the network {Xi, Yi} and the mapping provided in the dataset. The backpropagation is an iterative algorithm that aims to find, in the space of the weights of a neural network, the minimum configuration of the loss function. A loss-function typically used for a classification task is the cross-entropy defined as:

During each iteration of the backpropagation algorithm, the weights are updated by calculating the gradient of the loss function:

The algorithm terminates when it achieves the best performance on external datasets (validation set).

In our approach we use this neural network architecture for its well-known ability to find, after a training phase, a good approximation of the relationship between the input variables and the distribution of the output variable [15].

In order to predict the ALE of each resident unit in BRI, first a MLP is trained, then, analogously to log-linear models, a random extraction of ALE values from the estimated ALE distribution is performed, conditional to observed covariates. Of course, this reduces prediction accuracy but improves the distributional accuracy, which is our main goal.

Our approach aims to be as general as possible, therefore:

To train the MLP, we employ the cross-entropy as loss function to be minimized. The cross-entropy is a measure of the distance between the distribution of the output variable and the distribution of the target variable. The architecture of the network is shown in Fig. 3: it has two hidden layers of 128 neurons each, and an output layer with 8 neurons (one per modality of the target variable). To limit the risk of over-fitting in the learning phase, two dropout layers have been interposed. The best configuration of some hyper-parameters (number of hidden neurons, dropout probability, learning-rate) was explored through a suitable grid-search.

Figure 3.

Architecture of the implemented MLP model.

For each record of the dataset, the model generates a probability distribution on the 8 ALE items. In a conventional ML approach, the imputed value would be the modal value of the distribution. However, in our case study, an important goal is to best reproduce the distribution of the ALE variable in the population of interest. Therefore, as already mentioned, to increase the distributional accuracy, for each record we impute an ALE item that is randomly extracted from the probability distribution of the corresponding pattern of covariates.

For our case study, we use a Linux server with Ubuntu 16.04.5 LTS operating system, deployed on the Azure cloud platform and equipped with a Tesla V100-PCIE-16GB GPU. The GPU is not strictly necessary but reduces the runtime to train the model.

The training phase of our MLP model lasts about one hour. The runtime mainly depends on three factors. The first factor is the complexity of the model: our MLP has about 27,000 free parameters (the neural network weights); the second factor is the number of iterations (epochs) performed by the optimization algorithm: we set it to 500. The third factor entails the way the training set was built: we adopted a k-fold validation approach and generated a dataset of 312,813 trials (this will be better explained in paragraph).

5.Experimental study

The comparison of MLP with the official imputation model is carried out on the Italian region Lombardia and the subset of population for which the target variable is available (see last column of Fig. 1). The target variable is the self-declared ALE in the 2018 sample census, referring to the year 2018, which corresponds approximately to 5% of total population of interest.

Note that people with complete longitudinal information on course attendance from administrative sources (part of subpopulation A in Fig. 1) are excluded from this experimentation, as the knowledge of their schooling history, until scholar year 2017/2018, would make the ALE 2018 prediction task too easy. Further studies will be devoted to this subset of units.

The dataset for the experimentation consists of 312,813 people residents in Lombardia in 2018 with no missing data on ALE 2018 (target variable). This is the sum of sub-populations A-Yes, B-Yes and C-Yes in Fig. 1.

A first experiment is carried out by using the MLP with the same covariates selected for log-linear models. The goal is to minimize confounding factors, therefore allowing for a neat comparison of results in terms of statistical accuracy. In a second experiment, data provided to MLP are not pre-processed: all the variables in the dataset enter the MLP algorithm without any selection or reclassification. In particular, the variables age and citizenship are not aggregated into classes and the variables relating to the type of school attended are used as they are presented from administrative sources, without any type of aggregation. The variables relating to the place of residence and place of birth are also included. Moreover, the information on the data source of the three subpopulations is not considered and the flag variable (ABC_2017) which identifies the three subgroups A, B and C, is not introduced. This second experiment is clearly meant to study the possibility of using a more automated approach for the prediction of the ALE variable in large-scale production settings.

The variables used in the different experiments are described in Table 1.

The results of estimates obtained with MLP are compared with the ones obtained with the official procedure. Quality measures are concerned with predictive accuracy of each unit and accuracy of estimated aggregates (quantities obtained by aggregating the unit predictions). The first measure is generally the one analyzed in ML approaches, while the second is usually taken into account in National Statistical Institutes when evaluating the quality of an estimation procedure. We note that it is not necessarily true that a method with the best predictive accuracy is also the best in terms of accuracy of aggregates. This issue has been extensively investigated in the Machine Learning literature, see for instance [16, 17]. Since the ALE distribution will be published by gender, age classes and citizenship, it is important to evaluate the distributional accuracy in these specific subpopulations. The aggregates considered in this study refer to the main figures that are officially disseminated by Istat. In particular, we report results for the ALE distribution by citizenship.

Accuracy is calculated using the k-fold approach with k = 5. The database is partitioned into 5 subgroups and:

Tasks 1–3 are repeated 5 times so to reconstruct the entire data set. The same approach is used for both ML and log-linear models so that results can be compared.

Figure 4.

Item accuracy: Log-linear vs MLP estimation (test set 2, run 1).

After the implementation of this approach each individual (in each k-fold) has two probability distribution on the 8 ALE items, estimated using ML and log-linear models. The imputation process consists of extracting a random value from the probability distribution. The same imputation process is repeated 100 times to consider the model variability and the resulting indicators are averaged over those repetitions.

6.Conclusions

In Istat, the increasing use of administrative data poses the need to investigate new methods that can efficiently handle large amounts of heterogeneous data and still lead to output statistics of satisfactory accuracy. The statistical procedures used so far in the statistical production system generally require preliminary tasks of data analysis and data treatment that are very expensive in terms of resources. Moreover, even when surveys are repeated in time, these preliminary tasks must be performed for each new round (although perhaps with less effort), at least for confirming the data treatment adopted in previous rounds. Machine learning techniques can be useful to alleviate this problem since they are naturally applied in an automated way. A study on real data is performed to analyze whether a MLP model can improve the official procedure adopted in the Italian Permanent Census for the prediction of the attained level of education of each Italian resident. The evaluation focuses on two quality aspects: accuracy of predictions (and of estimated aggregates computed by directly using the predictions) and efficiency of the procedure. The efficiency assessment is primarily concerned with the automation of the process, which means that resources spent for data analysis and preparation can be minimized. Results are encouraging especially concerning the efficiency. In fact, we do not notice an improvement in terms of accuracy, but the same level of quality is reached by using raw data, that is without resorting to expensive data pre-treatment steps.

There are still some open problems to deal with for the application of the MLP approach to the production of official ALE statistics. The first is concerned with the presence of sampling weights. In this application, survey data are used without taking into account sampling weights. The role of sampling weights is to make the sample representative of the whole population, thereby leading to unbiased estimates. While techniques to incorporate sampling weights in classical statistical models are well developed in the literature, the same cannot be said for machine learning models. Further studies will be devoted to shed light on this important issue.

Another important question is concerned with the evaluation of uncertainty of estimates obtained by using the MLP approach. When official statistics are disseminated, National Statistical Institutes (NSI) must provide a measure of their accuracy. This is a fundamental piece of information that increases the credibility of NSIs. Common measures of accuracy (under the assumption of negligible bias) are estimated coefficients of variation and confidence intervals. MLP outcomes are evaluated by accuracy measures as well, but this mostly happens in contexts that are very different from official statistics, both in terms of data and goals. In fact, machine learning techniques are mostly aimed at micro-level prediction, rather than at estimation of population parameters. This aspect as well deserves further analysis.