Country statistical capacity: A recent assessment tool and further reflections on the way forward

1.Introduction

A country’s statistical capacity takes a central role in its development. Strong statistical capacity results in accurate measurement of economic activities (which provides timely inputs for policy response) and better information flows among various stakeholders (which enhances governance and efficiency). For poorer countries that often have weaker capacity, strong statistical capacity is particularly important, since it helps with monitoring poverty reduction and transparent uses of international aid.

The Statistical Capacity Index (SCI) is a tool developed by the World Bank in 2004 to assess improvements in country statistical capacity [1]. The SCI has been widely employed by different international and national agencies to measure progress with various development indicators including development trends [2], or areas of statistical improvement in member countries [3], or tracking the Sustainable Development Goal (SDG) for child development [4]. It also strongly contributes to the academic literature. We offer in Table 1 a brief overview of some selected academic studies in the past decade that employ the SCI. These studies highlight the importance of statistical capacity building [5, 6] and its useful values for measuring economic growth and government institutions and governance [7, 8, 9, 10, 11, 12, 13] as well as country potential success with achieving the Millennium Development Goals (MDGs) [14].11

Yet, the SCI has several key limitations [15]. First, the various aspects of the capacity of a national statistical system (NSS) that the SCI measures have fast become outdated. Since its launch, the SCI’s methodology and coverage have remained the same, while technological advances with computing and data storage capacity have enabled NSSs to make significant advancements with data collection methods and better dissemination practices. While the international community’s adoption of the SDGs raised the bar for NSSs regarding their capacity to produce higher-quality and more (frequent) data, the SCI includes no indicators of some important surveys (including the labor force surveys and establishment surveys). Second, the SCI focuses on poorer countries, which limits its relevance and application in an increasingly globalized world. Third, the conceptual principles and mathematical properties of the SCI leave much room for improvement. For example, key concepts underlying data production and data usage are not clearly defined, which may contribute to the misconceptions that all the stakeholders use similar standards in safe-guarding data quality and make similar efforts to ensure open data access. Technical concerns were also raised over the lack of a solid foundation behind the index’s aggregation method [18]. These concerns have practical relevance. For example, the overall SCI scores of Cameroon and Sudan rose from 55.6 and 51.1 in 2015 to 68.9 and 63.3 in 2016, respectively, indicating a 24-percent improvement of statistical capacity over one year. This stands in sharp contrast with the common knowledge that a country’s statistical capacity often improves incrementally.

The Statistical Performance Indicators and Index (SPI) represents an effort to address these limitations. While it was just recently introduced [19, 15], the SPI has been adopted for measuring country statistical capacity in various policy reports on progress with the Sustainable Development Goals (SDGs) [20, 21, 22]. SPI indicators have been formally employed to measure country statistical capacity under the SDG monitoring framework (SDG indicator 17.18.1). Researchers have also started employing the SPI to study country statistical capacity, ranging from assessing and improving data openness and accessibility, the quality of NSSs, government use of data, and future official statistics [23, 24, 25, 26] to better understanding national statistical offices’ (NSOs) response to the Covid-19 pandemic [27]. The SPI was also used to study sector-specific topics such as food and agriculture statistics [28] and forecasting GDP growth [29].

We make several new contributions in this paper. First, we offer detailed comparison for the SPI with its predecessor, the SCI, regarding different conceptual and empirical aspects between the two indexes, including their coverage for the years, countries, indicators, data sources and dimensions (categories), conceptual framework, and development focus. Second, we further examine the relationships of the SPI and the SCI with some key agriculture development indicators such as food security, food sustainability and productivity as well as headcount poverty, GDP per capita, and an SDG progress index [22]. To our knowledge, while previous studies briefly compare these two indexes [5, 30, 31], we offer the most comprehensive comparison between the SPI and the SCI in this paper.22 Our analysis employs the latest SPI data update in 2022, which were not available in previous studies. Finally, moving forward we propose clear guidelines on how the SPI can be maintained and updated to ensure that the process is transparent, replicable, safeguarded with high quality, and provides data consistency over time.

We find that the SPI is built on clear conceptual and mathematical foundations, which distinguishes it from the SCI. The method used to aggregate the SPI is based on three-level nested weighting approach, compared to simple arithmetic weighting for the SCI. It is supported with data on up to 186 countries, for both poorer and richer countries, while the SCI covers only 145 poorer countries. The SPI offers a shorter time series but more recent data for the period 2016–2022, while the SCI covers the period 2004–2020. The SPI offers more than twice the number of indicators as that of the SCI, covers more data dimensions (including more agricultural data), and is more closely linked to the SDGs. The SPI also has a stronger correlation with most agricultural development indicators as well as other key indicators.

This paper consists of six sections. We provide in the next section an overview of the SPI, including its conceptual framework (Section 2.1) and a description of country SPI scores (Section 2.3). We subsequently compare the SPI and the SCI in Section 3 before further examining their relationships with key (agriculture) development indicators, such as undernourishment, food insecurity, food sustainability, agricultural productivity, headcount poverty, GDP per capita, and an SDG progress index. We offer some reflections on guidelines for maintaining and updating the SPI in policy discussion Section 5 and finally conclude in Section 6. Supplementary materials for further analysis is provided in Appendix A.

Figure 1.

The Pillars and Dimensions that Construct the New SPI. Source: Dang et al. (2023).

2.Overview of SPI

2.2SPI country scores

Figure 2.

Country SPI Score. Panel A. SPI Overall Score, 2022; Panel B. Changes in SPI Overall Score, 2016–2022.

We map in Fig. 2, Panel A the SPI scores in 2022 for all countries. To provide a visual aid with interpretations, the countries are color-coded into five groups based on their performance. Figure 2 shows much heterogeneity for countries in different geographical regions or at different income levels.

Consequently, we examine the SPI in more detail by region and income levels in Table 2.66 Panel A of this table shows that there are large differences across regions. North America is the region with the strongest average SPI (93), which is followed by Europe and Central Asia (85), South Asia (67), Latin America and the Caribbean (65), East Asia and the Pacific (64), Middle East and North Africa (64), and Sub-Saharan Africa (58).

While Panel A notes the countries with the minimum and maximum scores for each region in parentheses, significant variation in the SPI overall score exists within regions. For instance, Fig. A.1, Panel A (Appendix A) shows that in the Latin America and Caribbean region, Costa Rica is the country with the highest SPI score of 89, while Haiti, one of the lowest-scoring country in the region, earns a far lower score of 39.6. In Sub-Saharan Africa, the highest-scoring country is South Africa with a score of 82.4, while the lowest-scoring country is South Sudan with a score of 33.8. In the East Asia and Pacific region, the top-scoring country is Australia with a score of 89.9, while the lowest-scoring country is Nauru with a score of 32.6.

Table 2

SPI overall score by region and Income Level

Group	Mean	Min	Max
Panel A: By region
East Asia and Pacific	64.3	32.6 (Nauru)	89.9 (Australia)
Europe and Central Asia	84.6	31.4 (Turkmenistan)	93.6 (Finland)
Latin America and Caribbean	65.5	39.6 (Haiti)	89.9 (Costa Rica)
Middle East and North Africa	64.3	24.4 (Libya)	83.4 (West Bank and Gaza)
North America	92.9	92.8 (United States)	92.9 (Canada)
South Asia	66.9	58 (Afghanistan)	79.1 (Sri Lanka)
Sub-Saharan Africa	58.4	33.8 (South Sudan)	82.4 (South Africa)
Panel B: By income level
Low income	54.4	31.9 (Syrian Arab Republic)	70.7 (Uganda)
Lower middle income	62.6	35.3 (Micronesia, Fed. Sts.)	84 (Mongolia)
Upper middle income	68.5	24.4 (Libya)	90.7 (Georgia)
High income	81.1	32.6 (Nauru)	93.6 (Finland)
Not classified	52.3	52.3 (Venezuela, RB)	52.3 (Venezuela, RB)

Note: countries with the minimum and maximum scores are shown in parentheses next to their scores.

We further examine the SPI score by income levels in Table 2, Panel B. This panel shows that countries with a higher income level have a higher SPI score. In particular, high income countries have an average SPI of 81, which is followed by upper middle income countries (69), lower middle income countries (63), and low income countries (54). In terms of relative differences, the SPI score for high income countries is 19 percent higher than that of upper middle income, 29 percent higher than that of lower middle income countries, and 50 percent higher than that of low income countries. Overall, the Pearson correlation in 2022 between (logged) GDP per capita and the SPI overall score is 0.58.77

Figure 2, Panel B further plots the changes in the SPI scores over the period 2016–2022. Most countries improved their score over this period. Countries that improve the most (more than 10 points) spread across different regions and include Chile, India, Indonesia, and Russia. On the other hand, certain countries such as Yemen perform worse, perhaps due to its ongoing conflicts. More details on changes in the SPI scores by region and income levels are shown in Appendix A, Table A.2.

We provide full overall scores and pillar scores for all countries in 2022 in Table A.1 (Appendix A). Further analysis shows that all the SPI pillar scores are positively correlated with one another, but no perfect correlation exists, suggesting that each pillar provides additional information on a country’s statistical performance. Compared to richer countries, low-income countries perform worse regarding data infrastructure and data sources [15].88 The SPI is publicly available at www.worldbank.org/spi.99

Table 3

Comparing the SPI and the SCI

No.	Characteristics	SPI	SCI
1	Years covered	2016–2022	2004–2020
2	Number of covered countries for overlapping years (unique scores)
	2016	167 (167)	145 (83)
	2017	174 (174)	145 (87)
	2018	174 (174)	145 (86)
	2019	174 (174)	145 (83)
	2020	181 (181)	145 (83)
	2021	181 (181)	N/A
	2022	186 (186)	N/A
3	Number of indicators	51	25
	Annually collected	44	25
	Non-annually collected	7	0
4	Data sources (%)
	Public International Databases	86	80
	NSO website	14	20
	Total	100	100
5	Aggregation method	3-level nested weight	Simple arithmetic weight
6	Conceptual framework	Yes	Not clear
7	Mathematical foundation	Yes	No
8	Dimensions	Covers 5 data dimensions (data use, data services, data products, data sources, and data infrastructure)	Covers 3 data dimensions (data products, data sources, and data infrastructure)
9	Agricultural data	More	Less
10	Focus	Sustainable Development Goals	Millennium Development Goals

Note: All the numbers are for the latest overlap year (2020) between the two indexes, unless otherwise noted. The numbers of unique scores are shown in parentheses next to the number of covered countries in each year. “N/A” denotes “not available”. All indicators are annually collected in the SPI, except for indicators produced by Open Data Watch, which are collected on a two-year cycle (indicators in dimension 2.2 and 4.3).

3.SPI and SCI

We turn next to comparing the SPI and the SCI. The SPI has several advantages over the SCI on both the conceptual and empirical fronts. In general, it better responds to the modern data landscape as well as countries’ current operational needs to monitor and improve their statistical capacity. In particular, the SPI explicitly offers standard desiderata for a statistical index (i.e., simple, coherent, motivated, rigorous, implementable, replicable, incentive consistent) [32].

Table 3 compares various features of the SPI against those of the SCI. The SPI offers data from 2016 onwards, while the SCI covers the period 2004–2020 (row 1). Further examining the overlap period 2016–2020 for the two indexes, Table 3 shows that the SPI covers between 167 and 186 countries (which include both low-income and high-income countries and more countries over time due to better data sources), while the SCI focuses on 145 non-high-income countries only (row 2). The importance of the new addition of high-income countries should not be underemphasized, since for the first time, the SPI allows for comparison of the strengths of well-regarded NSSs with the less-developed institutions in lower- and middle-income countries as well as measurement of the differences between low- and high-income countries regarding their capability to produce and disseminate statistical products and services [30].1010

Besides more country coverage, the SPI has better measurement power over distinguishing country scores. The SPI offers unique scores for all the countries (numbers in parentheses in row 2) during 2016–2020, which implies that the SPI can distinguish each and every country’s statistical performance. In contrast, the SCI offers a unique value for just more than half of the countries it covers in the same period, implying that it can distinguish only around half of these countries regarding their statistical capacity.1111 Furthermore, the SPI offers 51 indicators, which is more than twice the 25 indicators provided by the SCI (row 3). The majority of the SPI indicators are collected on an annual basis (44 out of 51 indicators) and the SPI has a greater number of such indicators (44 indicators collected annually in the SPI versus 25 for the SCI), although as a percentage the SCI had a higher share collected annually (86% for the SPI versus 100% for the SCI).

The SPI and the SCI both follow the guiding principles that the source data should be publicly available and meet certain quality standards (e.g., as provided by the curators of the international databases). Consequently, the two indexes largely collected data from publicly available, international databases (around 80 percent) and collect the remaining data from NSO websites (row 4).

Conceptually, the SPI has much clearer conceptual framework and it is built on a clear mathematical foundation with three-level nested weighting structure that offers desirable properties for an index such as symmetry, monotonicity, and subgroup decomposability [32]. These features are not available with the SCI (rows 5–7). Specifically, the SPI consists of five pillars of data use, data services, data products, data sources, and data infrastructure, which provide an updated characterization of a modern NSS. In contrast, the SCI offers zero coverage for the first two dimension (i.e., data use, data services) and only covers the three latter dimensions of the SPI (i.e., data products, data sources, and data infrastructure) (row 8; also see [31] for more discussion).

Regarding agricultural data, the SPI has more indicators related to agriculture than the SCI. In particular, the SCI only looks at the availability of an agriculture census and the availability of child stunting indicators. The SPI includes those, but also covers the availability of agricultural surveys. Furthermore, it additionally covers all the indicators related to agriculture, food, and nutrition under SDG number 2 (row 9).

Finally, the SPI cover indicators related to the SDGs, while the SCI cover indicators related to the (older) MDGs (row 10). Specifically, the SPI offers 22 indicators related to the SDGs, with two indicators provided by the World Bank. We provide a mapping of the SPI indicators to the SDG indicators in Table A.3 (Appendix A).

Figure 3.

Scatter plot and Trends of SPI and SCI Overall Scores for 2016–2020.

As a further check, we compare the volatility of the SPI and the SCI over time. Since country statistical capacity typically takes time to build up and so tends to change gradually, rather than abruptly [18, 32], an index is generally preferable if it has less volatility.1212 For the overlapping years, 2016 to 2020, the SCI has slightly more volatility during this period with an average standard deviation of 4.2, while the corresponding figure for the SPI is 3.9. For a visual illustration, Fig. 3 plots the relationship between the overall scores in 2016 and 2020 for the two indexes and some randomly selected countries. Regressing the 2020 SPI scores on the 2016 SPI scores provides an R² of 0.88, which is stronger than that of 0.73 for the SCI (top left corner panel). Some countries are shown to have more fluctuations with their SCI scores than the SPI scores in this period. In particular, El Salvador, Ethiopia, Mexico, Mongolia, and Tunisia stand out. Some countries including Chile, Kazakhstan, Mexico, Nicaragua, and Senegal even show opposite trends for the two indexes.

While in-depth analysis is necessary to offer further insights into these countries, Mexico offers an illustrative case study. The country’s National Institute of Statistics and Geography (INEGI) offers high-quality data and statistics, contributing to the country’s maximal score on data use and excellent SPI scores on data services, data products and data sources (Appendix A, Table A.1). Various factors are regarded as conducive to INEGI’s success, such as its full technical and management autonomy from the government, democratic political environment, and the willingness to adopt international standards [38, 39]. This is consistent with the finding that an NSO’s independence has a positive correlation with its SPI score [19].

4.Relationship with key agricultural development indexes

Development practitioners are typically interested in how well the statistical tools can help them evaluate a country’s performance in specific sectors. We thus examine the correlations between the SPI, the SCI and five common agriculture-related indexes produced by different organizations. The aim in this section is to examine whether the indicators of statistical capacity and performance are correlated with important agricultural development outcomes. And, if both are correlated, to examine if one is consistently more correlated with outcomes of interest. The outcomes include Food and Agriculture Organization’s prevalence rates of undernourishment and severe food insecurity [40], the Economist’s global food security index [41] and food sustainability index [42], and the United States Department of Agriculture’s international agricultural productivity index [43]. We also add three other key development indicators, including the headcount poverty rate (using the daily $2.15 poverty line), GDP per capita, and an SDG progress index [22].

Figure 4.

Absolute Value of Pearson Correlations between the SPI and the SCI and Key Agriculture Indexes. Note: The Pearson correlation coefficients are fully shown with the statistically significant levels in Table A.7. The correlations are shown for the averaged value of the index over the most recent two consecutive years (to reduce volatility). The years used for the indicators are: $2.15 poverty headcount rate (2021–2022 or latest two years available), GDP per capita (2021–2022 or latest two years available), SDR: SDG Index Score (2021–2022), Prevalence of Undernourishment (2019–2020 or latest two years available), Prevalence of Severe Food Insecurity (2020–2021 or latest two years available), EIU: Global Food Security Index (2021–2022), EIU: Food Sustainability Index (2021), USDA International Agriculture Productivity Index (2019–2020).

The results, plotted in Fig. 4, show that the SPI has a stronger Pearson correlation with all the indexes except for the USDA’s agricultural productivity index where its correlation is almost the same as that of the SCI. Notably, the SPI’s correlations with six indexes are statistically significant different from those of the SCI (Appendix A, Table A.4).

5.Further reflections on the way forward

We propose a set of rules that will be followed by the World Bank’s SPI team to update the SPI over time. These rules cover the quality control of new releases, the frequency and timing of data releases, the processes regarding updating and adding of indicators, versions of the SPI, and countries’ recourse. While these rules could be further improved in the future, setting up these rules can offer both transparency (regarding how the overall index scores can be expected to evolve as new data becomes available) and data comparability over time.

We also follow the guiding principle that data gaps affecting various dimensions of the SPI framework will be addressed in partnership with the international community. This will also involve developing new methodologies and constructing new data sets for measuring areas not yet measured. The result will be to produce a new set of indicators to include as part of the SPI, including in the SPI overall score, pillar scores, and dimension scores.

The adaptable and dynamic nature of the SPI framework is one of key features and innovations over existing approaches. However, any changes to or additions of the indicators is expected to affect the comparability of the SPI overall score, pillar scores, and dimension scores from one vintage of the data to the next. If these changes are not implemented well, this could potentially diminish the utility of the index scores.

We discuss next the governance of the SPI and the proposed rules to update the SPI over time.

6.Conclusion

We provide a comprehensive comparison between the World Bank’s recently launched Statistical Performance Indicators and Index (SPI) and its predecessor, the Statistical Capacity Index (SCI). We find that the SPI is built on clear conceptual and mathematical foundations, with more sophisticated aggregation method than that of the SCI. We find that the SPI is more strongly correlated with several measures of food security, sustainability of food systems, and a broader measure of health linked in part to food systems and food security (i.e., undernourishment).

The SPI offers a shorter time series but more recent data for the period 2016–2022, while the SCI covers the period 2004–2020. The SPI provides data on up to 186 countries, covering both poorer and richer countries, while the SCI covers only 145 poorer countries. It offers more than twice the number of indicators as that of the SCI, covers more (agriculture) data dimensions, and is more relevant to the SDGs. The SPI also has a stronger correlation with most agricultural development indicators as well as other key indicators. We also propose a set of rules for maintaining and updating the SPI.

Beyond its contributions to monitoring and building country statistical capacity, the SPI offers various promising research directions. One potential venue is to better understand how it compares with other (open) data indexes that are also used to gauge country statistical capacity. Another is to further study whether, and the extent to which, it helps countries improve their progress with the SDGs.