“You say you want a [data] revolution”: A proposal to use unofficial statistics for the SDG Global Indicator Framework

2.1Measuring the SDGs

From a statistical perspective the measurement challenges arising from the 2030 Agenda are huge. To assess progress, each of the 169 complex, multi-faceted targets requires a statistical indicator. In fact, many of the targets are so complex, 232 indicators have been agreed. In truth, if all aspects of the targets were covered properly, then arguably twice that number would be required [14].

The MDG requirements were modest in comparison with the SDGs. Nevertheless, at the end of the MDG lifecycle in 2015, countries could populate on average, only 68 per cent of MDG indicators [2]. In their final MDG progress report, the United Nations [15, p. 10] warned that ‘Large data gaps remain in several development areas. Poor data quality, lack of timely data and unavailability of disaggregated data on important dimensions are among the major challenges. As a result, many national and local governments continue to rely on outdated data or data of insufficient quality to make planning and decisions.’

The far reaching ambition of the 2030 Agenda has led to development targets that are well ahead of available official statistics and statistical concepts. In many cases, appropriate statistical methodologies do not yet exist from which to generate indicators. To elaborate this problem and facilitate the population of the GIF the IAEG-SDG has classified all SDG indicators in to three tiers on the basis of their conceptual development and availability of data [16]. The tiers are:

In April 2019, the IAEG-SDG reported that 44 per cent of the selected indicators were classified as Tier 1 (see Table 1). Furthermore, they reported that 15 per cent of the indicators remained classified as Tier 3. While Table 1 shows the not inconsiderable improvements in conceptual development and data availability that has been made since 2016, it also highlights the magnitude of the task still facing the global statistical community. The pace of transition of indicators through the tiers to reach Tier 1 is likely to slow as presumably the low hanging fruit has now been picked. Table 1 suggests this is indeed the case, as the conversion rate to Tier 1 was slower between December 2017–2018 than between December 2016–2017. There were no additional conversions between December 2018 and April 2019.

2.2Who measures?

Countries understandably guard and protect their reputations preciously. Consequently, countries can be quite sensitive about what is measured and who does the measurement. This sensitivity has often led to tensions between official national statistics compilers and external compliers. For example, IOs may from time to time, alter national statistics to ensure they align with international standards to facilitate international comparisons – the main purpose of international statistics. Understandably this can lead to disagreements and tensions. Tensions can also arise between governments and Non-Governmental Organisations (NGOs), universities or other countries for a variety of reasons. In the context of the 2030 Agenda, this has led to some tensions as to whose data should be used in the compilation of the SDG indicators.

Countries, perhaps not unreasonably, are anxious that only official national data are used to populate the SDG indicators. Equally, IOs are anxious that national statistics are internationally comparable and have therefore put forward arguments as to why their data should be used. Kapto [17, p. 135] notes, ‘A tense debate is taking place on data flows from national to regional to global levels, and on custodian agencies’ role in harmonizing national data for global comparability, as countries assert their sovereignty over national data.’ What he describes as a ‘cease-fire’ between countries and IOs; an agreement on data flows has been brokered by the IAEG-SDG [18]. But there are some circumstances where this approach may be sub-optimal. In thinking about this, it is useful to remember that the primary purpose of the GIF is to produce global and internationally comparable indicators.

The first reason to query the ‘country data’ approach is where specific national official statistics do not exist. Unfortunately, this is not an uncommon problem. It makes perfect sense to use good quality national official statistics when they exist but if they don’t, and there is insufficient data to populate global indicators (see Tier 2 – Table 1), then other approaches must be found. The demand for SDG indicators has exacerbated this problem as many of the targets (and consequent indicators) fall far outside the scope of traditional official statistics and thus are not guided by agreed international measurement standards (see Tier 3 – Table 1).

A second problem with the ‘country data’ approach is where problems with the national official statistics exist. Problems could mean anything from incompleteness, errors or inaccuracies, non-adherence to international standards, inconsistencies over time, or imbalances. A good example of where this might arise is the asymmetries that frequently exist between bilateral trade datasets. From a global perspective, unbalanced trade data are not especially useful, and so steps are taken by IOs to remove these asymmetries. This may lead to a mismatch between official national statistics and official international statistics. This issue is not unique to international trade, problems with national data exist across all statistical domains. Despite the best efforts of NSOs and IOs, internationally comparable data will be a real challenge for the GIF.

A third and more delicate issue is that of impartiality. Targets, such as for example, 16.5 or 16.6 which deal with corruption, bribery and institutional accountability provide perfect examples of why it might make sense to use external data sources other than official statistics provided by a NSS. There have clearly been cases where official national statistics could not be trusted to provide an independent or impartial picture. This is not to say, that all national data are untrustworthy. On the contrary – most national official statistics are trustworthy. But there are cases (either indicator or country specific) where arguments can be made that more independent and trustworthy data may exist.

Figure 1.

Net official development assistance (total) as a % of gross national income, 2002–2016. Source: OECD DAC: https://data.oecd.org/ oda/net-oda.htm and authors own calculations.

Another exception to the ‘country data’ approach arises from what can be termed the data revolution. Today our day-to-day dependence on technology is leaving a bewildering array of ‘digital footprints’. This in turn has created a deluge of digital data. Some of these new digital datasets are global in scope offering the possibility of compiling genuinely harmonised global statistics. In such cases, where a single data source might provide more consistent and timely data than the amalgamation of multiple individual country datasets, it would seem insensible to discount their use for the purpose of global reporting. This might be applicable to targets such as 15.1 that deal with forest, drylands, wetlands and mountain regions governed by international agreements. Arguably superior quality and internationally comparable data could be derived from satellite imagery than from multiple national datasets of which many will be based on irregular sample surveys of varying quality. For other examples, see the mapping of projects in the UN Big Data Inventory and the UN Global Pulse to SDG goals undertaken by MacFeely [9].

2.3The cost of measurement

Unlike the MDGS, the SDGs are universal. The SDGs are also much broader in scope, far beyond simply reducing extreme poverty, to encompass the survival of our planet, improving equity and freedom in our societies and trying to develop a more stable and sustainable economic model. One of the implications of such a broad and ambitious development agenda is the price tag. Estimates vary, but Ambassador Macharia Kamau of Kenya, who co-chaired the SDG intergovernmental consultative process, anticipates that implementing the 2030 Agenda could cost somewhere between $3.5 and $5 trillion per year [19]. Ibrahim Thiaw [20], United Nations Assistant Secretary-General and Deputy Executive Director of the United Nations Environment Programme, estimates it will cost at least an additional US$1.5 trillion annually over the Millennium Development Goals. The intergovernmental committee of experts on sustainable development financing [21] estimated the value of investment in infrastructure required to achieve the eradication of poverty alone at between $5 and $7 trillion annually.

To put these numbers in perspective, total Official Development Assistance (ODA) contributions from the OECD Development Assistance Committee members’, averages about $113 billion per year (in current prices).22 Since Monterrey in 2002, when the wealthier nations of the world, promised to contribute 0.7% of their Gross National Income to ODA [22], the cumulative shortfall in ODA (2002–2016) has risen to $2.4 trillion (current prices) or $2.9 trillion in 2016 constant prices. Since 2015, and the commencement of Agenda 2030, the average country effort hasn’t changed appreciably (see Fig. 1) and remains well short of the 0.7% target.

Developing the statistical concepts and collecting the data required for the GIF will not be inexpensive either. The Global Partnership for Sustainable Development Data estimates around $650 million per year is needed to collect data to support the 2030 Agenda, of which only $250 million is currently funded [23]. PARIS21 [24, p. 11] has estimated that ‘funding for statistics needs to be increased from current commitments of between US$300 million and 500 million to between US$1 billion and 1.25 billion by 2020’. The PARIS21 estimates cover a wider remit than just SDG indicators, and presumably this explains some of the difference in scale between the two estimates. But irrespective of which estimate of costs is used; all estimates clearly exceed existing funding. More recently PARIS21 [25] has estimated that ODA devoted to data and statistics ($591 million in 2015 and $623 million in 2016) is only one third of 1% of ODA.

2.4Summary of challenges

Part 1 has provided some of the background and context that are relevant to the proposals put forward in Part 2. The universal and broad scope of the 2030 Agenda present real measurement challenges for the global statistical community. Populating the GIF will be a challenging and complex task with enormous resource implications, even for developed countries with sophisticated statistical systems. History suggests that it is highly unlikely that by 2030 all of the 232 indicators will be populated. Today, only 44% of the SDG indicators can be populated. One of the risks with the Tier system is that it has created a vacuum, and as the saying goes: ‘nature abhors a vacuum’. Who will fill that vacuum and how? At a time when multilateralism is faltering, when funding is not matching ambition, and where the ‘data revolution’ has brought new competition, we see countries clinging to an anachronistic view, prioritising ‘country data’ or even abandoning the use of statistics as a policy support instrument, and international organisations jealously laying claim to indicators to attract or safeguard funding. If the development and statistical communities are serious about populating the GIF then it is time to consider alternative approaches.

3.1A proposal

An agreed recognized and mandated body, with the authority and competence to certify statistics as ‘fit for purpose’, would review unofficial statistics to see whether they can be certified as ‘official’ for the purposes of populating the SDG GIF. Statistics certified ‘fit for purpose’ could be accredited and used as official statistics. For the purposes of this discussion ‘Fit for purpose’ means that an indicator or statistic meets pre-defined quality and metadata standards and has been compiled in an impartial and independent manner. Those pre-defined standards and criterion must be open and transparent to all. The term quality can be interpreted in the broadest sense, encompassing all aspects of how well statistical processes and outputs fulfil expectations as a SDG indicator. In more concrete terms, ‘fit for purpose’ would mean that any statistic must be relevant, accurate, reliable, coherent, timely, accessible, and interpretable. The statistic must be produced using sound methodologies, concepts and reliable systems. The statistic must also be compiled within an institutional environment that recognises the need for objectivity, impartiality and transparency. This last point is important. For a statistic to be designated official, neither the input data nor the methodologies can be proprietary but must be available to all and open to scrutiny (subject to obvious confidentiality constraints).

Figure 2.

A proposed future: Using unofficial data and statistics to compile SDG indictors.

This proposal envisages the SDG GIF being populated from a combination of official statistics and unofficial (but certified as official) statistics. By pooling all available indicators an improved completion rate will be achieved. To ensure a level playing field and maintain quality standards a formal accreditation system is required. By combining official and accredited unofficial sources into a single high-quality ‘pool’ the chances of successfully populating the GIF will increase (See Fig. 2).

In this new regime the indicator pool would comprise of:

This supplementary approach would only be used when particular conditions apply. Firstly, it should be a measure of last resort, and only considered when all other official options have been exhausted. Specifically, when:

Secondly, compilers of unofficial indicators hoping to secure accreditation must demonstrate their adherence to the principles of official statistics. For national accreditation this means observance of the UN Fundamental Principles of Official Statistics [3]. In particular, principles 1 (impartiality), 2 (professionalism), 3 (scientific standards), 6 (confidentiality), 9 (international classifications) are of special relevance and should be rigorously tested. Principle 5 (quality and other aspects of data) is also extremely important. For global accreditation it would mean adherence to the Principles Governing International Statistical Activities [4].

Thirdly, unofficial indicators will be required to meet a defined set of quality standards. For national accreditation, the indicator would be required to meet the same standards and conditions as set out in the national code of practice or national statistical quality framework. For international accreditation, the indicator will be expected to meet the quality standards as defined in the UN Statistical Quality Assurance Framework [27]. Furthermore, clear metadata standards should be set for accreditation. In cases where standards don’t yet exist, the UNECE Common Metadata Framework [28] sets out suitable generic standards that could be used as criteria for accreditation.

Finally, prospective compilers of official SDG indicators must be able to guarantee that they can supply those indicators for, at least, the lifetime of Agenda 2030. In practical terms, this means being able to supply, at a minimum, the statistic on an annual basis for the years 2010–2030. While sufficient funding is important, in line with the UN Fundamental Principles of Official Statistics [3], funding must be free of any political or ideological conditions or influence. Access to the indicator itself must also be open and constraint free.

One could view the conditions outlined above as overly rigid and with too many criteria. The counter argument might be to just let compilers bid against each other and whichever indicator or statistic can be demonstrated to have the best quality (however defined) would be selected. While such an egalitarian approach might be intuitively logical and attractive, it might be seen as contravening the spirit of the ‘country first’ principle agreed by the IAEG-SDG in 2019 [16]. Hence, for the purposes of this paper, a set of reasonable, albeit conservative, conditions are envisaged. It also means that the results of the homologation would be binary i.e. only two possible outcomes are envisaged – a statistic is either accredited or it is not. One could also make the argument that such a binary result is itself too rigid, and one could perhaps envisage a wider set of choices, for example, an intermediate or experimental certification might be possible. If the scope of the discussion were broadened to official statistics more generally, this indeed would be a worthwhile consideration. However, in the context of the SDGs, which already have the complication of Tiers, an experimental category might muddy the waters too much.

3.2How does this differ from the current situation?

Unofficial or ‘non-official’ data sources are already being used as inputs in the compilation of official statistics all around the world, both at national level and international level. At national level, for example, unofficial data are frequently used to supplement official survey data in the derivation of consumer price index expenditure weights, retail sales index trading day weights, and in many aspects of compiling national accounts. Typically, at national level, there are no official guidelines or accreditation systems used in these processes. Depending on the quality and detail of the metadata, the reliance of an individual statistic on unofficial data may or may not be clear. As noted above, NSOs will be guided by their own national codes of practice and the UN Fundamental Principles of Official Statistics [3], in particular, principle 5 which states that ‘Data for statistical purposes may be drawn from all types of sources, be they statistical surveys or administrative records…’

The same is true at international level, except that IOs are directed by the Committee for the Coordination of Statistical Activities guidelines on the use of unofficial data. Those guidelines, Recommended Practices on the use of Non-Official Sources in International Statistics [5], provide direction on the use of unofficial source data. No formal accreditation system is necessary when using unofficial data as they are effectively subsumed into official aggregates and thus are covered by the formal ‘official’ label applied to the derived indicator. In other words, accreditation of unofficial data is implicit. The guidelines however stay silent on the use of fully developed indicators.

Both NSOs and IOs already regularly use unofficial source data to compile official statistics. This practice is expected to grow as statistical agencies are now looking beyond survey data and administrative records to investigate whether big data is a useful source of data for compiling official statistics. In 2018, 34 NSSs from around the world had registered 109 separate big data projects on the Big Data Project Inventory33 compiled by the UN Global Working Group on Big Data. IOs had logged a further 91 projects [9]. NSOs and IOs are investigating a wide range of big data sources, from satellite imagery to mobile phone CDR records to augment or supplant existing data sources or generate completely new statistics. The question now is how all of this activity will be integrated with the compilation of official statistics more generally.

This proposal goes a step further than existing practices and frameworks, in that it anticipates using, in the specific cases outlined in Section 2.1, unofficial statistics to create a larger ‘pool’ from which SDG indicators can be selected. This pool would comprise of not only unofficial source data to derive official statistics, but also using already developed unofficial indicators or statistics (but reclassified as official) – see Fig. 2. Now compilers of statistics (official and unofficial) would submit bids (proposals) to the IAEG-SDG for consideration. Bids would only be considered if they adhere to agreed quality and metadata standards and broader principles of official statistics.

3.3Risks associated with adopting this proposal

No doubt persuasive counter-arguments can be made against implementing this proposal. After all such a move will introduce risks. But not adapting to the modern data world runs the risk of achieving only a partially populated GIF, which in turn risks tarnishing the reputation of the global statistical community. A business as usual approach also puts NSSs, particularly those in developing countries, under unnecessary pressure to compile a range of new statistics.

This section outlines some of the most obvious risks in adopting this proposal. There are legal concerns, reputational risks and practical implementation issues, such as costs, to be considered. Some of these issues are discussed briefly.

3.4The risks associated with not adopting the proposal

Some of the risks associated with implementing the proposed approach have been outlined above. But there are also risks in not considering such an approach. It is also important to carefully consider these. The main risks would appear to be those arising from unaddressed competition.

3.5Ideological arguments

It is clear there is resistance in many countries to governments collecting more data. The argument underlying this resistance is supposedly fears of a Big Brother state [51, 52]. Despite statistical legislation and the UN fundamental principles, respondents, but most particularly firms, don’t always trust NSOs to safeguard their data from other arms of government or not to use their data for non-statistical purposes. As an aside, MacFeely [9] notes the incongruity of these concerns and the lack of concern regarding the emergence of a corporate or private sector Big Brother. But there is ideology at play here. The neo-liberal agenda aims to minimise the role of the public sector. Landefeld [7] warns, even in the data sphere, there will be resistance by industry to expanded government oversight.

Thus, one can anticipate ideological arguments against accreditation, along the lines that this is an expansion of the role of government. But as Reich [53, p. 5] correctly points out ‘Government doesn’t “intrud” on the “free market”. It creates the market.’ Polanyi [54, p. 61] too notes the importance of the ‘deus ex machina of state intervention’ for the formation of markets. The UN or national government must set normative standards, whether, defining post codes, tax numbers, personal identification numbers or statistical classifications – these are all part of a nations data infrastructure [55] and essential to the efficient functioning of a market. Even Hayak [56], the godfather of modern liberal economics, understood this, explaining that in line with liberal principles, the State should exercise control of weights, measures and statistics.

In any event, challenging the establishment of national or global accreditation mechanisms on the grounds of such ideology is a specious argument. An accreditation system will facilitate wider participation of the private sector, along with academia, NGOs and civil society in the 2030 Agenda. It opens a doorway, for indicators that have traditionally been excluded from consideration, to compete for recognition as official SDG indicators.

3.6Consistent philosophy

The 2030 Agenda emerged from a globally inclusive, open and democratic process. In line with this philosophy, contributions on the compilation of SDG indicators could also be open and inclusive. To an extent they already are, in that anyone can propose indicators, or comment on existing proposals. But to date, it has been envisaged that compilation will be the exclusive permit of official statisticians (either national or international). But what if the power and knowledge of unofficial data and unofficial statisticians could be harnessed? This indeed would be a data revolution.

The idea of an accreditation system is not inconsistent with the philosophy underlying the UN Fundamental Principles of Official Statistics [3]. In particular principle 5 which states:

In other words, statistical agencies should in principle use the widest variety of data sources possible to compile official statistics provided the quality of those data are sufficiently good and the costs are not prohibitive. Why not go one step further, and argue that statistical agencies should in principle use, not only the widest variety of data, but also the widest variety of statistics for the purposes of providing official statistics to feed the SDG GIF?

The idea is also broadly consistent with the spirit of the 2030 Agenda itself, which states ‘Data and information from existing reporting mechanisms should be used where possible’ [57, p. 48]. So, like the fundamental principles, the 2030 Agenda recognises the importance of reusing existing data and information from other official systems. Again, one could argue that what we are proposing is simply an extension or relaxation of this condition – in particular, a relaxation of the term ‘existing reporting mechanisms.’ The 2030 Agenda also noted that any ‘global review will be primarily based on national official data sources’ (our emphasis) [57, p. 74a]. Thus, it was recognized from the start the GIF might require data from outside national official sources. The document wisely didn’t set any conditions or limitations on what these sources might be.

This proposal is also consistent with the broad philosophy or vision put forward by the Independent Expert Advisory Group on a Data Revolution for Sustainable Development in their report ‘A World That Counts’. In this report, they state ‘New institutions, new actors, new ideas and new partnerships are needed, and all have something to offer the data revolution. National statistical offices, the traditional guardians of public data for the public good, will remain central to the whole of government efforts to harness the data revolution for sustainable development. To fill this role, however, they will need to change…and strong collaboration between public institutions and the private sector’ [58, p. 9]. The report stresses the need to create incentives for private sector participation and comes tantalizingly close a number of times to proposing something quite radical, but it never quite does1010 – it highlights the importance of data sharing but never statistics. In short, they advocate a vibrant ‘global data ecosystem’ [58, p. 17] and an extended concept of statistical systems. We interpret (global or national) data ecosystems as something much broader than (global or national) statistical systems – See Fig. 3.

Figure 3.

(National/Global) statistical and data ecosystems.

The NSOs mapped in Fig. 3 don’t require any explanation. A NSS is the collection of statistical institutions or units within a country that collects, compiles and disseminates official statistics on behalf of national government. For the purposes of this argument, we understand a data ecosystem to be the amalgam of all data and statistical actors in a country, including official statistics and holders of public sector or administrative data, private and commercial sector data holdings and indicators, research data, civil society and non-governmental data holdings. We acknowledge that in an era of globalising data imposing a distinction between a national and a global data ecosystem is perhaps somewhat archaic. The idea of constraining global digital data to a ‘country’ or that data will respect national borders is anachronistic. Thus, we acknowledge that data ecosystems may need to be international or global by default. The important point is that data ecosystems are much broader than official statistical systems.

The official statistical system, whether national, regional or global, should retain control of the process for standards and certification. Thereafter, there is no reason why NSOs or NSSs could not accredit unofficial statistics or indicators for the purposes of compiling SDG indicators. Furthermore, with the evolution of modern, globalised data sources, there is no reason why IOs or the UN could not establish regional or global accreditation systems to facilitate the use of good quality unofficial statistics.

3.7Lessons from history

Scientific discovery has always relied on amateur inventors or scientists. Many important contributions were made ‘by men with minimal scientific education’ [59, p. 201]. John Harrison, a clock maker, invented the famous H⁢1 ships chronometer used to estimate longitude; Michael Faraday discovered diamagnetism, electrolysis, and electromagnetic induction; Gregor Mendal, a Czech Augustinian monk, pioneered experiments on dominant/recessive qualities of genes in peas; William Herschel, an amateur astronomer forged the development of telescopic lenses and discovered the planet Uranus; and Charles Darwin was the legendary amateur naturalist famous for his contribution to the theories of evolution. Weinberger [60] points out, the reason that amateurs such as these could make such important contributions and have them recognised was that there were bodies, such as the Royal Society, the Royal Astronomical Society or the Académie des Sciences to test and validate their work.

There are lessons we can learn from this approach. Just as professional scientists did not have the monopoly on scientific wisdom in the past, official statisticians do not have the monopoly on information today. In fact, when it comes to mining new forms of digital data, official statisticians are for the most part far behind their unofficial counterparts. Today, many unofficial statistics are produced by a wide variety of compilers, ranging from: journalists; researchers; social media outlets; civil society; academia; commercial enterprises; lobby groups; and NGOs. The quality of these statistics varies enormously, from one end of the quality spectrum to the other. In many cases the quality is hard to determine, as the underlying data and methodologies are proprietary and shrouded in mystery. In other situations, the statistics are clearly of good quality and are accompanied by supporting metadata. It seems unwise therefore to tar all unofficial statistics and indicators with the same brush.

Is there a way to sift and sort this effort in such a way as to harness it? Could NSOs (at country level) or the UN (at the global level) provide mechanisms that could test and validate unofficial statistics and accredit them for the purposes of the SDG GIF? That is the question posed in this paper. Without such a system, new statistics will emerge daily, leaving the public unclear as to their quality and utility. But by providing a quality assurance stamp, NSOs at country level and the UN at the global level could say which statistics are ‘facts’. The UN could become today, what the Académie des Sciences was to the Victorian era, in terms of validation. Winning such recognition might provide the necessary incentive for many compilers to become less proprietary with their data, methodologies and algorithms.

3.8Lessons from the data ecosystem

Official statisticians are not the only experts grappling with the challenges of homologation. Across the wider data ecosystem, data scientists and researchers too have been developing their own principles,1111 certification standards and accreditation procedures. Some examples include: The Network of Expertise in long-term Storage and Accessibility of Digital Resources (NESTOR) seal;1212 the Trusted Digital Repository checklist (ISO standard 16363) https:// www.crl.edu/archiving-preservation/digital-archives/ metrics-assessing-and-certifying/iso16363; and most recently (2016) the CoreTrustSeal or Core Trustworthy Data Repositories Requirements,1313 launched jointly by the International Council for Science World Data System (ICSU-WDS) and the Data Seal of Approval. While the focus of these standards has been primarily the certification of digital data repositories rather than on the compilation of statistics, they nevertheless offer examples of how communities have developed governance structures and certification standards to ensure that data can be trusted. No doubt there are lessons to be learned from these scientific communities.

The fact that certification standards and accreditation procedures are being developed by communities in the wider data ecosystem suggests that perhaps, our concerns and thinking inside the national and global statistical system are not so different from those outside. In other words, official statisticians, data scientists, researchers, and other scientists are all concerned with the quality of data, statistics and analyses. Perhaps optimistically, it suggests, that compilers of unofficial statistics will not find adherence to accreditation standards an alien or overly challenging concept.