Performance of LSTM neural networks in nowcasting global trade during the COVID-19 crisis

2.1Nowcasting in the context of the COVID-19 pandemic

Nowcasting is the forecasting of the current or near-current value of a variable, often using information that is published or made available more quickly than the variable of interest. Some commonly nowcasted series include GDP [9, 10, 11] and international trade [12, 13]. These types of aggregated macroeconomic variables lend themselves well to the nowcasting paradigm, as they are often published later than some other economic indicators while still being of great interest to policymakers, investors, and firms. Some common methodologies to perform economic nowcasting include mixed data sampling (MIDAS) [14, 15], dynamic factor models (DFM) [13, 16], mixed-frequency vector autoregression (VAR) [14, 17], and Bayesian vector autoregressions [18]. Hopp [3] and Loermann and Maas [19] examined neural networks’ suitability to the application, more specifically long short-term memory (LSTM) networks in the case of the former. The LSTM methodology is explained further in Section 2.2. For more information on nowcasting, including commentary on common data issues encountered in the field, see Hopp [3] or Cimadomo et al. [18].

Nowcasting became more relevant than ever in the wake of the economic fallout from the COVID-19 pandemic. Since March 2020, when many governments around the world began shutting down businesses and other forms of economic activity in response to the virus, the rate of change in the economy has been truly unprecedented [20]. Furthermore, the epidemiological nature of the crisis and successive COVID-19 waves have meant that the economic recovery has not been one of monotonic recovery, as governments have often had to roll back and reinstate openings in response to the severity of local and national outbreaks. This has simultaneously increased the need for accurate, timely assessments of the economic situation to inform policy and mitigate economic impact on citizens, while making those assessments harder to acquire.

However, crisis often creates opportunity and breeds innovation, and the field of nowcasting has been no different. A wealth of papers relating to nowcasting during the COVID-19 pandemic have been published since March 2020. Many geographies are represented, including Canada [21], Sub-Saharan Africa [22], the United States [23], Mexico [16], and the Euro area [17], among others. Perhaps more interestingly, novel data sources have additionally been explored, for instance Google mobility data [24], retail payment system data [21], Google search trends, and mobile payment data [22]. Unfortunately, the longevity of the COVID-19 crisis to this point ensures that nowcasting its effects on the economy will remain fertile ground for new research in the coming months and years.

2.2Long short-term memory neural networks and dynamic factor models

Having established the context in which the nowcasting exercise outlined in this paper takes place, this section will give a short background on the methodologies employed. DFMs are a very popular technique employed in nowcasting, used by prominent institutions such as the New York Federal Reserve for their nowcasts [25], as well as having over 171 results in Google Scholar for the term ‘nowcast “dynamic factor model”’ in the domain of economics. Compare this with 140 results for ‘nowcast “midas”’ and 144 results for ‘nowcast “vector autoregressive”’, two other commonly used nowcasting techniques [26]. The models operate under the assumption that there are one or more underlying latent factors common to all variables in the model, which, together with each variable’s idiosyncratic component, explain the observations. The models have been extended to address the specific requirements of nowcasting, for instance, the handling of mixed frequency data, e.g., predicting a quarterly variable with monthly variables [12]. Their simplicity, flexibility, and strong predictive performance in empirical analyses have contributed to their adoption in the field. It is for this reason the methodology was chosen to benchmark the performance of the LSTM against. For more information on DFMs, see Bańbura and Rünstler [27] and Bok et al. [25].

Artificial neural networks (ANNs) have risen in prominence in recent years due to their impressive performance in a variety of applications, including tasks like image classification and natural language processing. However, traditional feed-forward networks lack a temporal component; information flows unidirectionally through the network. The recurrent neural network (RNN) architecture and long short-term memory network architecture (LSTM), a derivative of the former, add a temporal component in the form of feedback loops, where information can flow either forwards or backwards in the network. This allows them to better handle time series and renders them more suitable for application in the nowcasting context [28]. For more information on how ANNs and LSTMs work, see: Hopp [3], Singh and Prajneshu [29], Sazli [30], or Loermann and Maas [19]. For more detailed information on LSTMs’ properties which make it suitable for nowcasting, see Hopp [3], Section 3.2.

3.1Description of data and models

Hopp [3] examined the LSTM’s performance versus that of DFMs in nowcasting global merchandise and services trade. In that case, LSTMs were found to produce superior predictions. However, the test period was the fourth quarter of 2016 to the fourth quarter of 2019, a period when the target series’ movements were much more muted than compared with 2020 and 2021. Furthermore, test performance was found using artificially simulated data vintages based on historical publication lags. For context, a data vintage refers to how the model input data would have appeared at different points in the past. A nowcasting model should be evaluated at different points in time, for instance early on, when not much data for the nowcasted period is available, and later on, when most or all input variables’ time series are complete for the nowcasted period. In the absence of actual snapshots of how the data appeared in the past, artificial data vintages can be used, which simulate missing values in the data based on the publication lag of each variable. For instance, consider a dataset consisting of two monthly indicators which are used to nowcast a quarterly indicator. The first indicator is published with a one-month lag, so May’s data comes out in July, and the second indicator is published with a zero-month lag, so May’s data comes out already in June. If the second quarter is being nowcast, i.e., June’s figure, a simulated data vintage of one month prior to the target period simulates the data as it would have appeared in May, with the first indicator having data until March available, and the second having data until April available. All data after those dates would be set to missing. The analysis performed in this paper seeks to build on the previous paper’s findings and further validate and stress test them with: (A) a much more volatile and difficult to predict in context, and (B) actual data vintages collected over the course of 2020 and 2021.

In this analysis, three target variables were again nowcast: global merchandise exports in both value [31] and volume [32], and global services exports [32]. These are the same series examined in Hopp [3]. All target series were expressed in seasonally adjusted quarter over quarter growth rates. In total, 45 independent variables were used as inputs to estimate both a DFM and LSTM model for each target series: 17 for merchandise exports values, 17 for merchandise exports volumes, and 21 for services exports. Variables were sometimes used to estimate more than one target series. Input variables included things such as industrial production indices, manufacturing export order books, and retail trade indices, among others. See appendix 1 for a full list of input variables, including their geographies, frequencies, sources, and for which target series they were used. The same variables were used in estimating both the DFM and LSTM models to ensure maximum comparability. Input variables were a mix of monthly and quarterly frequencies expressed in period over period seasonally adjusted growth rates.

The DFM and LSTM models were trained on data dating from the second quarter of 2005 to the fourth quarter of 2019, representing the maximum extent of information a forecaster or policymaker would have had in the run up to the COVID-19 pandemic. Actual data vintages collected over the period from March 2020 to October 2021 were then used to assess model performance in nowcasting the target series from the second quarter of 2020 to the second quarter of 2021, an exceptionally volatile and difficult period to nowcast due to the unprecedented impacts on the global economy of the COVID-19 pandemic. Actual data vintages were collected on a monthly basis from March to July 2020, then on a weekly basis from August 2020 to October 2021.

The LSTM model used was the same examined in Hopp [3], using the averaged output of 10 networks. For the logic of using the average of multiple networks’ outputs, see Hopp [3] Sections 4.1 and 5, or Stock and Watson [33]. Hyperparameters were found by using the period from the second quarter of 2005 to the third quarter of 2016 as a training period, and the fourth quarter of 2016 to the fourth quarter of 2019 as a test period. Ragged edges were filled using the mean of each series, see Hopp [3] Section 3.2 for more information. “Ragged edges” refers to missing values at the end of each variable’s time series. They are called “ragged” because different variables may have a different quantity of missing values at their ends due to differing publication lags and schedules.

The DFM model used was that described in Cantú [12], where a state-space representation is used to model the DFM under the assumption that the target and independent variables share a common underlying factor, as well as containing their own idiosyncratic component. Subsequently, the Kalman filter is applied and maximum likelihood estimation used to obtain parameter estimates. The Kalman filter is a means of estimating values of an unknown variable using that variable’s own measurement history coupled with statistical noise. The process is recursive and occurs in two stages; a predict stage and an update stage. In the predict stage, an initial estimate of the latest value in the series is obtained via a linear model. In the update stage, the eventual recorded value is combined with its a priori estimated value via weights to obtain a refined estimate of the latest value in the series, and the process repeats. For more information on this specific DFM methodology and the Kalman filter, see Bańbura and Rünstler [27] and Bok et al. [25].

Once DFM and LSTM models were trained for each target series with data up until the fourth quarter of 2019, predictions could be obtained on actual monthly and weekly data vintages to see how the models’ forecasts would have developed over time as the pandemic unfolded and its economic repercussions began to appear in the data. In this way, we can see what narratives and guidance the nowcasts would have provided to policy makers and analysts as well as assess their errors over time and final performance.

Predictions were made for each quarter on data vintages dating 100 days either forwards or backwards in time, to assess performance both early on, when little data for the period was available, and later on, when data on most independent series had been published.

3.2Results

Figure 1 shows the development of the two models’ predictions over time for the period from the second quarter of 2020 to the second quarter of 2021. The X axis shows the days difference from the target period. E.g., 0 days difference for 2020 Q2 refers to 1 June 2020, to 1 September for 2020 Q3, etc. The Y axis displays the quarter over quarter growth rate. The red line displays the actual observed growth rate, while the blue and green lines represent the predictions of the LSTM and DFM models, respectively. Each point making up the blue and green lines represents what the two models predicted the growth rate of the target series would be given the data available at that point in time. Generally, the predictions should move closer to the actuals line as time goes on and more data is released.

Figure 1.

Nowcast evolution over time. Note: For brevity, “Values” refers to global merchandise exports in values, “Volumes” refers to global merchandise exports in volumes, and “Services” refers to global services exports.

3.3Comparison with dynamic factor model

Before further comment on the performance and prediction qualities of the LSTM versus the DFM is made, it should be reiterated that the series mean was used to fill in missing values in ragged edges for the results obtained in Fig. 1 and Tables 1–3. Using the mean as opposed to ARMA estimations for filling ragged edges, the other option available in the nowcast_lstm library, naturally has some impact on the development of the LSTM’s predictions, especially early in the prediction period. For insight on how using ARMA would have impacted the LSTM’s predictions, refer to appendix 2, which shows the same information as Fig. 1 with the addition of LSTM ARMA ragged edge filling predictions. Using ARMA increases the reactiveness of the LSTM forecast, most evident early in the 2020 Q3 prediction period, but the findings and conclusions drawn from the previous section and below largely hold true for both the mean and ARMA ragged edge filling approaches.

Observation of the development of the two methodologies’ predictions during the COVID-19 pandemic leads to two broad conclusions of these methodologies on this dataset. The first, that the DFM is generally more reactive, and second, that the DFM is much more influenced by previous values of the target variable, especially early on in the prediction period. It could be argued that both of these observations make the LSTM’s predictions more suitable for nowcasting during economic disruptions, like those caused by the COVID-19 pandemic, or any other exceptionally volatile context.

Figure 2.

Share of weeks with a bigger revision.

The first observation, regarding the DFM’s increased responsiveness, means that the DFM will respond more quickly and to a greater degree to changes in the data. At first glance, this may seem to be an advantage in the COVID-19 context. Indeed, considering only the second quarter of 2020, the DFM did generally perform better over the whole prediction period. It responded to the negative indications in the data significantly more quickly than the LSTM. However, this behavior is more beneficial when strong signals in the data remain consistent throughout the period. Even within the second quarter, extremely negative signals in the data were not distributed evenly, and this is visible in the development of the two models’ predictions over the period. Extreme contractions recorded in April and the first half of May 2020 were followed by strong growth in June, as restrictions were lifted. As a result, both models had to revise their forecasts upwards in the latter half of the prediction period. In all three cases, the DFM, as the more reactive model, had to revise its predictions more heavily than the LSTM. For instance, in the case of merchandise export values, by July 2020 the DFM had significantly overshot the degree of contraction, predicting growth 7.6 percentage points lower than the actual. At the same time period, the LSTM was predicting growth just 1.7 percentage points lower than the actual. The LSTM, not having reacted as strongly or quickly to April and May signals, had more slack than the DFM to correct course in later weeks without the need for strong revisions.

This behavior is apparent in subsequent quarters as well, where the DFM reacts to the same data releases with much stronger revisions. Figure 2 quantifies this observation, displaying the share of weeks or data releases in each quarter where either the DFM or LSTM had a bigger revision to its forecast.

In every quarter-target series combination but one, the DFM more often had the bigger week to week revision than the LSTM, sometimes drastically so, as in services in the third quarter of 2020, where the DFM had a bigger week to week revision during 89 per cent of the prediction period. The two models only achieved parity in 2021 Q2 for services. Figure 3 further reinforces the observation, displaying the average weekly revision’s absolute value for the two methodologies by target variable and quarter. The LSTM had smaller revisions on average for every target-quarter combination, often drastically so. By this metric, even target-periods which look favorable to the DFM in Fig. 2, such as 2021 Q1 values or 2021 Q2 services, do not compare well. The average weekly revision for the LSTM in these two periods is just one fifth and one third that of the DFM, respectively. This suggests that in weeks where the DFM has a smaller revision, it is smaller than an also small LSTM revision, whereas the DFM often has very large revisions which are not matched by the LSTM. In short, in this analysis, with a given data release, the DFM is more likely to revise its predictions much more heavily than the LSTM.

Figure 3.

Average weekly revision.

If a methodology produces predictions that are liable to be drastically altered not only over the course of the prediction period, but even from week to week, that limits its usefulness and the degree to which decisions can be made from its outputs. Services in the fourth quarter of 2020 are a good illustration of the contrasting nature of the two methods’ prediction development. In the first week of the prediction period, the DFM started out forecasting very strong growth of 8.4 per cent. The next week, it revised its predictions upwards to 13.6 per cent. It continued to revise itself even higher the next four weeks, before reversing trend and continually revising downwards before plateauing at around 3 per cent. Then, one week in December, it revised itself down by 3 percentage points, beginning to forecast a contraction of about 1 per cent until the end of the prediction period. Over the same period, the LSTM began by forecasting modest growth of 1.1 per cent. It slowly revised itself upwards for the next two months, hitting 4 per cent by the beginning of November, before remaining at that level for the rest of the prediction period. This relationship and pattern are broadly generalizable to all of the quarter-target series combinations examined in the analysis.

This leads to the second main observation, that the DFM is more heavily influenced by previous values of the target variable. The LSTM does not take into account previous values of the target variable when making its predictions. The DFM does, however, due to its architecture, where the target variable is also used in estimating the latent underlying factor(s) [25]. Especially early in the prediction period, we can see how the DFM’s predictions are heavily influenced by either the previous value or the previous prediction of the target variable, observable in Fig. 1 as its predictions from one quarter appear to “flow into” those for the next. The LSTM does not display this behavior. Under normal circumstances, this characteristic is not particularly detrimental, as the difference between growth rates quarter to quarter does not widely differ historically, and early in a prediction period the prior observation would have been a sensible place for a forecast to start. For instance, between 2011 and 2019, the median absolute value of the difference between a quarter’s growth rate and the previous quarter’s was just 1.6, 0.8, and 1.5 percentage points for values, volumes, and services, respectively. Under the extraordinarily volatile conditions during the COVID-19 pandemic, however, where these quarter-to-quarter differences are more than an order of magnitude higher, this severely hampers the usefulness and accuracy of the DFM’s early predictions. Because there was a sharp contraction in the second quarter of 2020, the DFM predicted similar results early in the prediction period for the third quarter. It was only able to generate more accurate predictions once more data were released signaling a strong reversal in the trend. Similarly, early in the fourth quarter prediction period for merchandise values, it again predicted extremely strong growth, as that was observed in the third. However, it was highly unlikely that such an exceptional growth rate would be maintained in the fourth quarter, something it could only reflect much later in the prediction period. The LSTM in all cases started with much more conservative estimates, which then gradually built towards a final prediction as more data were released. This behavior could be desirable in a nowcast during normal periods, but perhaps especially so in a volatile context such as the COVID-19 pandemic, where early estimates are ideally conservative to leave room for uncertainty, slowly building towards a final prediction as more data are released and confidence grows. This initial conservatism is enhanced in the LSTM by using mean-filling for ragged edges as opposed to ARMA, which displays prediction evolution slightly closer to the DFM’s (see appendix 2). Even in the ARMA case, however, initial predictions remain much more conservative than the DFM’s and later revisions resemble the mean-filling approach’s very closely.