On the domain aided performance boosting technique for deep predictive networks: A COVID-19 scenario

2.1LSTM for COVID-19 transmission analysis

LSTM [17] have been used in effectively predicting the COVID-19 transmission dynamics. The recurrent nets are known for their ability to store short term memory of the previous data sequences through recursive neural connections. These LSTM variants have been utilized to learn the interrelationships between any parametric variations to estimate the future transmission dynamics. However, on the downside, the data obtained from the natural surroundings are non-stationary in nature. These are therefore unpredictable and difficult to model. The recurrent neural architectures fail to effectively learn the embedded characteristic features of such non-stationary data. So, a more detailed analysis of the gradually changing slope information can be extracted from such data for more accurate estimations.

2.2LSTM, GRU networks for stock price prediction

Stock market indexes are guided by a number of complicated financial indicators. Researchers have explored LSTM [20] for predicting the stock prices of Nifty 50 by analyzing the historical data of the previous five years. Based on the data collected on multiple financial indicators, the system was able to predict the opening and closing stock prices with reasonable accuracy. Similar attempts have also been made using linear regressions and time series models [21] for predicting rice price volatility in the Sri Lankan market. Projection of the same market indicators on a 2D or a 3D plane and analyzing with CNN based counterparts could possibly reduce the error in predictions.

2.3LSTM networks for wind speed forecasting

Wind speed have been considered as a potential renewable form of energy since centuries. With the wind power gaining more popularity in many countries as a renewable energy source, it is important to find an effective way to harness the power of wind speed for generating electricity. In this context, LSTM networks [22] have been applied to learn from the historical climatic data. A proposed LSTM based model was able to predict the wind speed of Wind Atlas for South Africa (WASA) with reasonable accuracy. If the same data is analyzed based on the changing slopes of the historical climate, then the study might lead to further reduction of the error in estimation.

2.4LSTM networks for sea surface temperature prediction

Global warming has been a serious concern of contemporary scientific interest over the past couple of centuries. Sea surface temperatures (SST) is one of the most important phenomenon having profound effects on global climate. Deep LSTM networks [23] have been proposed to predict daily SST. Researchers have claimed the proposed LSTM based model to be highly promising for short and mid term predictions.

2.5LSTM for photo-voltaic energy prediction

Photo-voltaic (PV) systems convert sunlight into usable source of electricity. PV cells are embedded into power networks all over UK. The output of PV is dependent on local climatic parameters. Researchers have employed LSTM networks [24] to study the dependencies between various climatic conditions and PV energy metrics for a seamless electric grid. LSTM networks have been useful in forecasting the temporal variations of PV power metrics with reasonable accuracy.

2.6GRU for tropical cyclone prediction

Tropical cyclones have devastating effect on life and environment. Meteorologists have analyzed the historical data trends to predict upcoming cyclones, path trajectories, probable damages etc. GRU [25] based networks have been proposed to predict tropical cyclone trajectories as a temporal function of varying climatic parameters. However, the path trajectories of the cyclones also depend on the previous slopes of the path curves in addition to the influence of external climatic parameters. Hence, the use of similar methodology involving CNN networks may yield more accurate results.

2.7Statistical approaches for time series prediction

Various Machine Learning and ARIMA based models [26] have been developed to predict from time series data. The proposed models have proved to be useful in finding meaningful information from large complex datasets with reasonable accuracy.

The present study may have substantial contribution in further enhancing the accuracy of these models by augmenting the non-stationary data with the respective inherent slope patterns. Though, much work have not been done in image domain analysis of time series data, the following subsection (H) highlights a notable work on data transformation to visual domain.

2.8Domain transformation of data for higher accuracy

Transferring learnable features to another domain have yielded better results in cases where the features are not easily interpretable. CNN networks have been well acclaimed [27, 28, 29, 30, 31] for their super-human abilities in analyzing images. The data was transformed [19] from a domain not suitable for learning to a different domain on which CNN networks could be trained. An improvement in classification accuracy was achieved over conventional classification methods by a margin of 10%.

2.9Assisted learning approaches using sparse Bayesian

Assisted learning techniques have been applied using Sparse Bayesian Learning (SBL) [32]. It was possible to achieve less than 6% NMAE in traffic state estimation by applying appropriate kernel matrices based on spatio-temporal correlations, and further pruned using Kalman filters.

3.1Deep learning based prediction models for COVID-19 transmission

Deep neural networks have been utilized for analyzing different types of data acquired from various sources. Pearson correlation coefficient [33] have been considered for analyzing the inter-relationship between the different components of the data and discard irrelevant components. The historical data related to the pandemic have been analyzed to predict the data trends for the future. Such predicted results can be particularly useful for decision makers in taking appropriate decisions.

Figure 1.

Recursive LSTM architecture.

Figure 2.

Recursive GRU architecture.

Figure 3.

Recursive CNN architecture.

Traditional artificial neural networks are capable of learning from labeled datasets with a fixed feature size but these traditional feed forward networks suffer from forgetfulness with respect to the data trends learned in the past. RNNs are uniquely designed sequence modelling algorithms with recursive feedback loops that takes the output of the previous time-step to the input of the current time step. RNNs are capable of taking decisions influenced by the past data trends. Specialized recurrent networks such as LSTM [34] and GRU have been selected for analyzing the COVID-19 pandemic datasets to predict the future consequences.

A spatio-temporal LSTM (ST-LSTM) architecture comprising of two LSTM layers having 50 neurons each, with return sequences and 20% dropout have been considered in the present study. The first two ST-LSTM layers with return sequences are followed by a third, non-recurrent ST-LSTM layer consisting of 50 neurons with 20% dropout. The third layer connects to a fully-connected dense layer with a single output neuron. The ST-LSTM architecture can accept the infected count data of the last six days along with upto fifth order derivatives i.e. slope information of the pandemic trend over the last six days to predict the pandemic spread for the following day. A recursive ST-LSTM architecture can pick the predicted data from the previous time steps and recursively feed back to the input of the network for the upcoming prediction step and so on. Figure 1 represents the proposed recursive deep ST-LSTM network.

GRU [35] are a type of advanced RNN with specialized gating mechanisms but has fewer parameters compared to a traditional LSTM architecture. A spatio-temporal GRU (ST-GRU) architecture is comprised of two GRU layers having 50 neurons each with hyperbolic tangent (tanh) activation function and 20% dropout, and return sequences with sigmoid activation function without any dropouts. The first two ST-GRU layers with return sequences are followed by a third, non-recurrent ST-GRU layer consisting of 50 neurons with a softmax activation and 20% dropout. It is connected to a fully-connected dense layer with a single output neuron. The ST-GRU architecture (Fig. 2) can accept the infected count data of the last six days along with upto fifth order derivatives of the pandemic trend to predict the spread for the next day. A similar recursive architecture can accept the predicted data from the previous time steps and recursively feed back to the network for the upcoming prediction steps.

Another family of neural networks, CNN [36], have been acclaimed for their adeptness in pattern analysis. The temporal pandemic data can be transformed into an image pattern by projecting on a 2D plane. A CNN architecture comprising of four convolution layers (9-36-36-81) with a fixed kernel size 6, followed by a fully-connected flattened dense layer having 360 neurons at 10% dropout. The flattened dense layer fully connects to a single output neuron. The CNN model under consideration accepts the infected count of the last six days along with upto fifth order derivatives of the pandemic trend to recursively predict the pandemic spread on a future timeline. Figure 3 represents the recursive deep CNN network under consideration.

3.2Data augmentation to a learnable domain

In the background of the COVID-19 pandemic, the rate of spread in a geographical location depends on the number of people infected. Additionally, the rate of spread of pandemic also depends on other regional parameters such as climatic factors, population density, elderly ratio etc. Pandemic trend datasets having similar characteristics as the COVID-19 are non-stationary in nature due to the temporal variations of one or more of these related characteristics.

Considering the temporal spread of pandemic, let the data be represented by a hypothetical equation h⁢(t) over a temporal scale. Now, with time, variations in one or more related parameters can gradually influence the rate of progression of the pandemic. The extreme granular level of parametric changes can be captured from slopes, rate of change of slopes, variations of the rate of changes etc. by computing the derivatives of h⁢(t).

The slope information obtained from the pandemic data along the temporal axis (t) have been augmented for transformation from a less learnable temporal domain to a learning efficient derivative domain for analyzing the domain spectral patterns.

In the present scenario, since the contagious agent is transmittable, higher the population density in a particular region, greater will be the chances of a pandemic spread. Also, due to the genetic variations between the male and female, their individual immunity levels vary greatly. So, sex ratio of a region can also bring about a big difference in the rate of spread of the pandemic. Similar decline of the immunity levels affect both males and females with age and therefore the elderly dependency ratio can impact the spread as well. Additionally, since the pandemic is caused due to a bio-agent, the infectivity as well as the life span of the contagion may vary with the change in climate across the globe. Therefore, historical weather data have also been acquired to investigate any patterns in the pandemic data due to climate change.

3.3Algorithm optimization

Training any deep neural network is data dependent and the training efficiency relies largely on the training time and the resources used for training. The performance and efficiency of a trained neural network is an optimization problem. There are several hyperparameters based on the individual mathematical formulations and the respective deep learning algorithms can be optimized accordingly. In the current study, optimizers such as the Stochastic gradient descent (SGD), AdaGrad, AdaDelta, RMSProp and Adam have been employed along with a suitable choice of hyperparameters such as the learning rate (LR), decay rate, momentum, regularization, hidden layer size, number of hidden layers etc.

3.4Error calculation

Standard Error calculation methodologies such as Root Mean Square Error (RMSE) [37], Mean Absolute Error (MAE) [38], and Mean Absolute Percentage Error (MAPE) [39] have been selected for error computation.

4.1Data collection and pre-processing

Data collected from the different sources [40, 41, 42, 43, 44, 45, 46, 47] comprised of inconsistencies such as missing values and noisy spikes. Since the actual population data, infected count, weather data etc., do not have random fluctuations in reality, the data has been smoothened using a moving average filter of length 3.

Figure 4.

Comparative time series prediction accuracy (100% – MAPE%) of LSTM and GRU.

Sequence models like LSTM and GRU are generally used for analyzing time sequence data. Following the same idea, initially, vanilla LSTM and vanilla GRU networks have been employed to analyze the pandemic spreads. These specialized recurrent networks are good at retaining information from historical data. The vanilla networks performed well for countries where the temporal rate of progression of the pandemic is uniform. However, for countries where the underlying features suffered from variations of the embedded characteristics, the prediction accuracy was on the decline for both LSTM and GRU vanilla networks. Figure 4 shows a comparative study of the declining prediction accuracy for several countries using the LSTM and GRU networks.

Figure 5.

Domain converted of time series data.

4.2Domain transformation for augmented learning

Since the rate of spread of infection is related to the multiple regional parameters, the daily infection count and the derivatives of the data trend has been derived upto the fifth order to capture the slope variations for further analysis.

Figure 5 shows the domain converted time series data in a derivative domain. The domain transformation in Fig. 5 illustrates how the embedded characteristic features of the pandemic data changes over time. The spectral trends of the preceding six days (Fig. 5) would be analyzed using ST-LSTM, ST-GRU and CNN networks to predict the seventh day infection count which in turn is fed through a recursive loop to the input for the upcoming eighth day and so on. The most optimum performance of the proposed data model can be obtained for short term prediction (upto 2 weeks in advance). Based on the distribution of the pandemic spread, a combination of high, moderate and low impact countries have been selected for data analysis, prediction and validation of the results.

4.3Correlation study

The correlation data in Table 1 shows that in the months from January to April, 2020, the highest temperature in the epicentric regions of India varied from 19∘C to 39∘C while the average temperature varied between 12∘C to 31∘C. During the same period, the highest humidity varied from 53% to 100% with the average humidity ranging between 41% to 94%. The correlation study reveals that similarity of trends of the pandemic spread cannot be observed with any of the captured population or climatic parameters over time and across locations. All the identified features that did not add much information were discarded to lower the prediction error. Pandemic data upto the fifth order derivative have been considered for data modeling.

4.4Algorithm optimization

The proposed CNN, ST-LSTM and ST-GRU algorithms have been optimized using multiple optimizers as presented in Table 2 (data of Spain).

Analyzing the quantitative outcomes in Table 2, it can be observed that each algorithm performs relatively better under some specific optimizer configurations. These configurations have been used for further processing.

4.5Predictions with best optimizer configurations

The algorithms with their individual best optimizer configurations have been employed to study the pandemic spread. Figure 6 shows a qualitative analysis of the CNN, ST-LSTM and ST-GRU prediction accuracies (100% – MAPE%) for some of the selected countries. Experimental evidences (Table 3) show that the proposed methodologies using CNN algorithms give the best results.

The recurrent neural architectures such as the ST-LSTM, ST-GRU retain short term memories of the past trends of the data and the error encountered in these predictions depend on the past experiences. The transmission dynamics of the pandemic being a real life scenario are extremely complicated and changes with time depending on different parametric variations. Therefore studying the past trends using these algorithms, achieved slightly higher prediction errors compared to the more efficient RRCNN which could analyze the latent features embedded in the spectral patterns.

CNN based architectures being comparatively more computation efficient, analyzes the spectral patterns corresponding to the rate of change of these variations over time. It is experimentally observed that the accuracy of these algorithms increased when the first derivative of the pandemic data was added to the data model. Higher order derivatives (upto 5th order) were included to enhance the accuracy further. Additionally, the accuracy of a CNN algorithm depends on the size of the convolution kernel. In this background, a kernel size 6 (which is closest to the length of the spectral bandwidth) resulted in the lowest prediction error.

4.6Restricted recursive CNN

CNN network had been most successful in analyzing the changes in the slopes of different characteristics in the pandemic patterns. However, certain predicted changes in slopes resulted in negative slopes for the transmission dynamics. In reality, the cumulative number of infected can never decrease. Hence, a Restricted Recursive CNN (RRCNN) architecture has been proposed where the predicted values of the network is controlled within an allowed limit. In the present study, the predictions made by the RRCNN is limited to non-negative values only. Over a couple of tests, it is observed that this modified version of the CNN model can minimize the overall error in estimation.

4.7Critical analysis

In the current study, the selection of the most suitable algorithm for the prediction model has also been based on the time of training required for achieving the maximum accuracy. The LSTM and the GRU networks are

Figure 6.

Comparison of RRCNN, ST-LSTM, ST-GRU and CNN predictions across countries [legends: training data (black), validation data (red), predicted data (blue)].

Figure 6.

Continued.

Figure 7a.

Comparative accuracy (100% – MAPE%).

Figure 7a.

Efficiency plot.

computationally expensive owing to the complexity of the mathematical formulations of their respective cell states.

On the contrary, CNN networks are computationally much cheaper and are capable of analyzing complex image patterns in lesser time. Table 3 quantitatively compares the prediction accuracies (100% – MAPE%) achieved by the chosen algorithms. It can be observed from the given results that the RRCNN model has achieved the closest estimation of the transmission dynamics on an average in comparison to the other models. The prediction accuracies for the chosen algorithms have been plotted under the Fig. 7.

As per the existing literature, RNN variants such as an LSTM or a GRU based prediction model has given far better predictions in analyzing time sequenced data. The same has been corroborated by various scientists that the short term memory preserved in the LSTM and GRU cell states help these networks in generating predictions under the influence of recently observed data trends. On the other hand, CNN networks have been reported to perform extremely well on image patterns due to the presence of convolution kernels which can particularly focus on smaller regions within the image and relay the edge related information for a specific spatial distribution through sampling layers.

The models have been trained on the pandemic data and the prediction of the trained models were validated against the following 1 day’s pandemic count. Table 3a compares the validation and predicted errors (MAPE%). As per literature, the LSTM model performs best (lowest error) compared to the other algorithms on the validation data i.e. in predicting 1 day following the training data. However, as the prediction horizon increases, changes in the embedded characteristics causes variations of the transmission dynamics. The LSTM and the GRU algorithms gradually end up with higher prediction errors over larger intervals. However, the CNN algorithms could be observed to analyze the changes in the embedded patterns reasonably well and achieved lower prediction error in the longer term. Table 3b compares the experimental results of regular vanilla LSTM and GRU with the proposed methodology using ST-LSTM, ST-GRU and CNN algorithms. As per experimental observations, the CNN algorithm was able to outperform the LSTM and GRU variants.

In the present scenario, the pandemic data has been acquired as a time sequenced data as in Fig. 8. Now, let the original time sequenced data be represented by Ni. Further, this time sequenced data when plotted on a 2D plane, generates a pattern as shown by the first vertical block (data = N). As the values of Ni varies over time Ti, a first order derivative (Fi) of the pattern has been computed as 𝑠𝑙𝑜𝑝𝑒=Δ⁢NΔ⁢T for a very small change of Δ⁢T on the time axis. This concept of slope has been illustrated in a small rectangular magnified projection within Fig. 8. Similarly, a second order derivative (Si) has been computed from the first order derivative pattern and so on up to the fifth order derivative (Hi) as in Fig. 8. Thus, a time sequenced data can be converted into a set of spectral patterns similar to an image dataset and can be analyzed using CNN networks. The RRCNN variant has performed better than the ST-LSTM and ST-GRU by achieving lower prediction errors in analyzing these patterns.

Figure 8.

Sequence to pattern.

Depending on their individual computational capacity, each algorithm has its own time requirements for analyzing and training depending on the size of the training data. For the current study, each of the chosen algorithms were exposed to a bootstrapped training dataset consisting of 10k records. Figure 7 shows a comparative study of the accuracy (100% – MAPE%) achieved on the given training dataset against the training time necessities of the respective algorithms.

The histograms in Fig. 7b represent the highest percentage accuracy that could be achieved by an individual algorithm while a plotted line drawn over the histograms demonstrates the respective time efficiencies. It can also be inferred from Fig. 7b, that the CNN based algorithms have the maximum computational capacity and are capable of handling large training datasets in almost 60% lesser time taken by GRU units and nearly 50% lesser time consumed by LSTM cells.

Optimizers such as Adam, Rmsprop, Adadelta, Adagrad and Adamax have been utilized for optimizing the hyperparameters for maximum accuracy with minimum effort. Structurally, the LSTM and GRU models have been tuned for the optimum number of recurrent layers, dense layers with fully connected units, the allowable dropouts and activation functions at each layer. Similarly, the CNN based algorithm have been fine tuned for the most optimum number of Convolutional layers, count of filters, size of kernels, stride size of the filters, max-pooling layers, dense layers with fully connected units, the allowable dropouts and activation at each layer.

The stochastic gradient descent (SGD) is the most commonly used optimizer where a learning rate can be optimized for evaluating the optimum step-size for adjusting the internal neural weights. A momentum hyperparameter accelerates or dampens the vanilla gradient descent in the relevant directions. However, a fixed step-size limits the overall efficiency leading to the essence of more advanced optimizers.

A computationally efficient Adam optimizer built on the adaptive estimation of moments have been utilized for fine tuning the decay rates upto the first and second order moments. A modified version of the Adam optimizer, i.e. an Adamax, known for occasionally delivering superior outcomes over Adam have also been tested in the current scenario. Another prominent Rmsprop optimizer has been utilized for computing a square-root of the moving average of the gradients and then divide the gradient with the square root to control the step-size of the oscillations more precisely along the entire training process. A different adaptive gradient descent optimizer, AdaGrad, takes an initial learning rate and suitably adapts it based on the frequency of the parameter updates during the training process. Each of these optimizers achieved limited accuracy improvements on the chosen algorithms. However, one of the main reasons behind the limitations could be related to the continual decay of the learning rates throughout the training. To overcome the limitations, a more robust AdaDelta optimizer have been utilized for limiting the change of learning rates upto a small window of gradient updates immediately preceding adaptation instance. Since the rate of change of the variations of the different factors influencing the pandemic spread changes with time, the imposed restriction upto the immediately preceding time window proved to be most effective.