Bridging the gap: Gallup’s role supporting the official statistics ecosystem
Abstract
The global data ecosystem is changing rapidly. New demands are increasingly being placed on National Statistical Offices (NSOs) worldwide to collect data to track a growing array of indicators. However, many NSOs lack the capacity to collect frequent, representative and high-quality data on even core metrics of national progress, such as food security. In response to this, there has been a growing number of partnerships between NSOs, international organisations such as the United Nations, and private sector organisations to address the data gap, which was only accelerated by the COVID-19 pandemic. Using Gallup, the global research and analytics firm, as an example, this paper highlights a number of areas where the private sector can provide value to the realm of official statistics. By adhering to globally recognised statistical protocols with a firm commitment to the principles of rigour, transparency, and respondent confidentiality, organisations such as Gallup play an important role in supporting the collection of official statistics. They can also bridge key data gaps related to the most pressing challenges of our time, and drive accountability on key issues of national and global development.
1.Introduction
Statistics provide the backbone of modern evidencebased policy making. The increased demand for reliable, quality and timely data is driving changes in the way governments, international organisations (IOs), non-governmental organisations (NGOs) and private sector organisations produce statistics that help track progress on societal issues.
The role of international official statistics has become more significant in the years since the United Nations’ 2015 adoption of the Sustainable Development Goals (SDGs). The SDGs represent an ambitious plan to shape global progress toward closing development gaps and resolving the biggest challenges facing humankind, including poverty, planetary degradation and conflict. The UN SDG 2030 Agenda is built on 17 goals, 169 targets and 231 unique indicators, measured at the national level. While United Nations system agencies are custodians of SDG indicators, National Statistical Offices (NSOs) are tasked with providing data inputs to monitor progress at the country level.
Some of these metrics have a long history of being gathered by NSOs, especially key sociodemographic and economic indicators such as statistics on employment. In recent times, the role of national statistical systems has been significantly expanded: the SDGs call for NSOs to collect a number of innovative indicators that were not included within their traditional suite of national official statistics.
A 2020 UN report noted that “The very comprehensiveness of the 2030 Agenda creates the need for an unprecedented range of statistics at different levels, including those derived from official statistical systems and from administrative and non-traditional data sources” [1]. Further, as a 2016 OECD brief noted, “national statistical systems will face trouble in certain indicators or will lack the incentives to measure them at all” [2]. For example, indicators related to a given topic – such as environmental preservation – were not traditionally part of the purview of NSOs.
One of the main aims of the UN’s 2030 Agenda is to drive country statistical capacity development and coordination. Stronger statistical systems enable countries to track the health of a country’s developmental course and capture important metrics that inform decision making, policies, and resource allocation toward building resilient, cohesive societies. This is important for low- and middle-income and high-income economies alike, as they seek to address people’s fundamental needs. However, NSOs particularly in low- and middle-income countries face a multitude of challenges including capacity and human capital constraints, technical or time constrictions, competing priorities, and financial limitations, that affect their ability to deliver frequent, timely and quality data. As such, acquiring relevant data for the range of indicators to be measured for the SDGs has created challenges for NSOs, as well as issues of standardisation and coordination across countries.
In many cases, national capacity has been enhanced and data gaps for several SDGs have been overcome through NSO collaboration on initiatives such as the USAID-funded Demographic and Health Surveys (DHS) which provide data for an array of indicators including access to safe water, early marriage and birth registrations [3]. Nonetheless, given that DHS surveys have finite capacity and are relatively infrequently implemented – once approximately every five years – data gaps and data timeliness are still pervasive issues for many SDG indicators.
In some situations, data constraints can be overcome by taking advantage of advances in data science to bring a massive volume of secondary or non-traditional data to bear on the issue. Bringing the best of this secondary data together with new analytical techniques can be a significant aid to policymakers in forming evidence-based decisions [4]. Examples include crowdsourcing or tracking online food prices to help monitor food price inflation, social media monitoring to support disaster management, and using GPS services to help predict the spread of infectious diseases [5]. Despite the promise of ‘big data’, UNESCAP reported that “No country examples are found of the use of mobile phone data, scanner data, smart meter data or social media data for the compilation of the SDG indicators” [6].
The lack of available data for feeding into SDG indicators is most evident in the case of sensitive or hard-to-measure constructs such as modern slavery. In these cases, international organisations must rely on their own data-collection efforts within countries, often in partnership with private organisations [7]. As advocated by SDG 17.17 on establishing “effective public, public-private and civil society partnerships, building on the experience and resourcing strategies of such partnerships”, the UN and other public interest institutions are increasingly exploring the complementarity offered by private sector capabilities in supporting monitoring of progress on major societal issues. Despite the ancillary benefits derived from building a wider statistical ecosystem within countries, tasking international private sector organisations with collecting data for national indicators of progress carries several challenges, especially where the commissioning entity is not the country’s NSO. For example, on topics where multiple parallel measurement frameworks exist, the use of non-NSOs for implementation means that governments lack the final say in determining which measurement framework is used, creating the possibility that the indicators selected for implementation do not align with national priorities or frameworks for development [2]. In addition to this by bypassing NSOs when collecting data, private sector organisations have little impact on building NSO national capacity whether gathering data or compiling official statistics [1].
Another perceived shortfall is that national governments may not own the data that are collected. In many cases, the collected data belong to the commissioning entity who makes the ultimate decision on whether to make the data publicly available, or not. For projects where data is used for SDG monitoring such an arrangement has rapidly become the norm, as practised by the World Bank [8] and FAO [9].
When appropriately implemented, public-private partnerships can create significant opportunities for the international statistical community if they are rooted in the principles of trust, usability and sustainability [4]. Private sector organisations can work with international organisations on intentional approaches to developing novel indicators that are not typically or consistently measured by NSOs, but which are critical to addressing key developmental barriers. Centralising and consolidating data collection allows for the international harmonisation of methods, such that results are comparable across countries as well as over time. It also offers opportunities to benefit from economies of scale, thereby increasing the financial efficiency of – and time taken to undertake – multi-country or global data collection.
One such example is Gallup’s partnership with United Nations specialised agencies, which are often supported by philanthropic organisations to collect data on specific SDG indicators. Using Gallup’s global survey infrastructure, the Gallup World Poll, international organisations have been able to gather high-quality, timely and reliable data disaggregated by income, gender, age, ethnicity, migrant status, degree of urbanisation and other characteristics.
Since 2005, through the World Poll, Gallup has conducted nationally representative surveys annually in over 140 countries and territories and in over 145 languages, representing more than 98% of the world’s adult population. The World Poll provides an infrastructure for surveying the world’s aged 15 and older population on a variety of societal and development topics through a consistent probability-based sampling methodology, using either Computer-Assisted Personal Interviewing (CAPI) or Computer-Assisted Telephone Interviewing (CATI), with a typical sample size of
By way of the breadth of country coverage, it is the largest nationally representative study in the world. As such, the World Poll infrastructure has been used by institutions including the United Nations and World Bank to support with the monitoring of progress on SDG indicators: i) Financial Inclusion (SDG 8.10.2) on behalf of the World Bank; ii) Food Insecurity (SDG 2.1.2) on behalf of the Food and Agriculture Organization of the United Nations (FAO); and iii) Modern Slavery (SDG 8.7.) on behalf of the International Labour Organization (ILO). Gallup supported these organisations in the development and testing of the measurement tools and indices that have helped quantify these key global societal challenges. These organisations also own the data and they are proprietary.
The rest of this paper will further elaborate on how Gallup has helped bridge the gap with NSOs and contributed to the wider health of the global data ecosystem. First, it will outline Gallup’s role in developing and collecting global statistics on topics from global nutrition to financial inclusion. In the following section, the paper will outline Gallup’s work surveying hard-to-reach populations on highly sensitive topics, such as modern slavery, before outlining some of the most common challenges and opportunities presented by collaborations with international organisations. Sections five and six respectively contain more details on the methodological standards that Gallup adopts in its global polling, and the role these standards play in helping to establish sustainable statistical practices around the world, before a final discussion about how Gallup’s experience partnering with international organisations demonstrates the ability of public-private partnerships to fill gaps in collecting official statistics.
2.Gallup’s role in developing and collecting global statistics
As an instrument widely used and scrutinised by NSOs and United Nations agencies alike, the Gallup World Poll provides a case study in how private sector data producers can partner with official statisticians at the inter-governmental level to explore new measurement methods and to fill vital data gaps while maintaining high levels of data quality, transparency, trust and professional ethics.
Gallup World Poll researchers routinely partner with experts across the international development community, academia and philanthropy, combining Gallup’s institutional expertise in conducting multi-national surveys with partner organisation representatives’ depth of knowledge regarding the specific issues addressed by their organisations. Gallup maintains strict criteria for vetting potential partnerships to identify any potential conflicts of interest and ensure that the work will remain unbiased.
When initially developing the World Poll’s core question module in 2004, Gallup engaged with leading social scientists, such as psychologist Daniel Kahneman and economist Sir Angus Deaton, to develop a set of indicators that would identify non-economic measures of societal progress such as wellbeing, in contrast to commonly used statistics such as GDP. The salience of filling these data gaps with input from the academic community was confirmed by the release of the Stiglitz-Sen-Fitoussi Commission’s report in 2008, which called for new indicators that could shed light on people’s “reported or experienced” wellbeing [10]. In addition to advancing the academic profile and foundation of the science of wellbeing, since 2012 the data has also underpinned the “World Happiness Report” – a partnership between Oxford University, Gallup, and the Sustainable Development Solutions Network (SDSN) – which has driven international public and policymaker attention to the discussion around metrics “Beyond GDP” and helped shape thinking around potential frameworks for a post-growth economy. Gallup also works directly with international organisations to develop new survey modules, creating and mainstreaming new indicators and helping to fill data gaps for existing indicators in years when NSO implementation does not take place. An example of Gallup’s role in supporting the validation of novel indicators and filling data gaps for existing indicators is its partnership with the Food and Agriculture Organization of the United Nations (FAO) through the “Voices of the Hungry” (VOH) project.
2.1Voices of the hungry
In 2012, in tandem with the FAO, Gallup helped establish an expert group to identify an innovative approach to measuring the severity of food insecurity experienced by individuals in the population, later known as the “Voices of the Hungry” project. After discussing the feasibility of the initiative from a conceptual point of view, the group drafted a preliminary version of the questionnaire for piloting, informed by insights from the use of experience-based food security scales such as the US Household Food Security Survey Module, the Household Food Insecurity Access Scale and the Latin American and Caribbean Food Security Scale.
Following a public procurement process, Gallup was selected to pilot the preliminary version of the 15-item scale in the World Poll in Germany, the Democratic Republic of Congo, India and China. A technical subcommittee met in January 2013 to review the pilot results leading to a refinement of the questionnaire, reducing the module from fifteen questions to eight, reframing the scale to focus on the individual rather than the household and renaming it as the Food Insecurity Experience Scale (FIES).
Following consultation with a panel of experts including national statistics institutes, the FAO undertook sixty-three focus groups [11] in four sub-Saharan countries (Angola, Ethiopia, Malawi and Niger) which explored the respondents’ understanding of the questions and suggested alternative phrasings. Subsequently, the final linguistically adapted version of the module was pre-tested by Gallup prior to survey implementation, leading to minor refinements in the wording of a few items.
After the successful pilot testing of the FIES in the four countries, the FAO partnered with Gallup in 2014 for the first global iteration of data collection. Utilising the World Poll, the FIES questionnaire was translated into more than 150 languages and dialects and administered in over 140 countries, enabling the FAO to create a common reference tool, allowing similar experience-based food insecurity scales to be adjusted to the common global standard and be comparable across countries.
The importance of this process was recognised in the U.N. Statistical System Organizations’ statement on Goal 2 at the second IAEG-SDG meeting in Bangkok, 2015, when the FIES indicator was reviewed for suitability for inclusion within the SDG indicator framework: “The collection of FIES data in more than 140 countries in 2014, through a private data collection service provider, has been the necessary step to validate the application of the scale in virtually every country in the world and to establish a global baseline for target 2.1. Having collected data from 146 countries has in fact allowed us to develop the analytic procedures that are necessary to ensure that the measures obtained in different languages, culture and livelihood conditions could be calibrated against a common standard reference metric, so that indicators would be truly comparable across countries” [12].
Despite the acceptance of the FIES as an SDG indicator there was recognition that “in the short to medium term, [national statistical systems] are simply not sufficiently equipped to collect the necessary real-time and high-frequency data, which also needs to be comparable internationally” [13] to feed into the international organisation reporting system. As such, when official national data are not available, Gallup World Poll-collected FIES data are annually used to help assess global food insecurity as reported in the UN RomeBased Agencies and WHO’s flagship publication, “The State of Food Security and Nutrition in the World report” [14] (SOFI) and track progress on SDG indicator 2.1.2. Annually, the national statistical authorities of over 70 countries accept the use of Gallup’s data in lieu of their own official national accounts of food insecurity. Such a quick turnaround of reporting would not be possible solely relying on official national sources [13]. This demonstrates the value that private organisations like Gallup can bring to collecting official statistics as NSOs come under increasing pressure and demands. The Gallup World Poll provides the additional advantage of enabling disaggregation of data using individual demographic characteristics. As such, collecting FIES data through the Gallup World Poll has enabled the authors of SOFI to annually report the global food insecurity gender gap, in alignment with the SDG’s mandate to “leave no-one behind.”
2.2Global Findex
Paralleling Gallup’s collaboration on food insecurity, in 2011 The World Bank partnered with Gallup to implement a multi-country survey project administered through the World Poll on individuals’ access to and use of banking services. The Global Financial Inclusion Index (Global Findex) utilises the Gallup World Poll to field a module of questions that provides comparable indicators showing how people around the world save and borrow money, make payments and manage financial risk. Administered every three years, the most recent iteration was conducted in 2021 during the COVID-19 pandemic, with results released in 2022.
The World Bank’s collaboration with Gallup also includes qualitative work, through additional in-depth interviews that enable a deeper, more contextual understanding of barriers, enablers and opportunities for greater financial inclusion of communities around the world.
The Global Findex Database offers insights and facilitates analysis by providing key demographic and socio-economic indicators, magnifying influence on policy and decision-making. The Global Findex provides policymakers and financial institutions worldwide with data on the economic opportunities available to people in their communities, as well as their capacities to save and to access loans for education or entrepreneurship. It also monitors new developments in the expansion of banking services and mobile money, as well as the effects of other country-level financial inclusion reforms.
The Global Findex dataset is available for public download on the World Bank Microdata Library website. With over 150,000 downloads to date, the creation of a high-quality and accurate database on financial inclusion has driven private sector investment, academic research and policymaker attention towards ways of enhancing financial inclusion.
G20 leaders have adopted the Global Financial Inclusion results as part of their suite of G20 Basic Set of Financial Inclusion Indicators. In 2015, the United Nations formally adopted The Global Findex data to underpin SDG indicator 8.10.2, the proportion of adults (15 years and older) with an account at a bank or other financial institution or with a mobile-money-service provider. Following the inception of Global Findex, the World Bank’s former President Jim Yong Kim prioritised financial inclusion and incorporated it within the World Bank’s key strategic agenda.
2.3Global diet quality project
A further example of Gallup’s role in collecting global statistics is its work on the Global Diet Quality Project (GDQP), a collaboration with the Global Alliance for Improved Nutrition (GAIN) and Harvard T. Chan School of Public Health. To address global malnutrition – and the related health and economic impacts of poor diets – policymakers need high quality data that are credible, affordable and timely. Yet despite the importance of a good diet to a good life, there has hitherto been no single data source responsible for tracking what the world eats [15].
The GDQP is the first global initiative to fill this data gap. It is the world’s first routinely collected, globally implemented, internationally comparable source of diet quality data. At the heart of the GDQP lies the Diet Quality Questionnaire (DQQ). The DQQ has two primary aims: to investigate diet adequacy and understand diet components that protect against – or increase the risk of – noncommunicable diseases. It was expressly designed for monitoring purposes.
Traditional methods of collecting dietary intake information are time and resource intensive. They typically rely on past 24-hour recall records of every food item consumed in terms of grams and calories, relying on several probes. Interviewers need to be highly equipped with dietary expertise in order to conduct these detailed surveys, and such training is often limited in lower income countries. As such, national scale dietary intake surveys cost millions of dollars to implement and require a high degree of training and specialism to undertake. Many higher income countries do collect nationally representative dietary intake data, but these face other limitations beyond cost. Different methods exist at a national level, meaning global comparability of data is unachievable. Sporadic data collection every five to ten years creates significant data gaps over time. Time-consuming data analysis also means that in some cases, findings are not released for years after data collection.
To address these challenges, the DQQ is designed to take five minutes to implement and measures consumption of 29 main food groups that have been selected for their relationship to nutrition and health, sustainability, and alignment with United Nations indicators and recommendations. Using food groups to measure diet quality requires minimal specialised knowledge from the enumerator – historically a barrier to data collection – making it widely accessible. Because of the speed and simplicity of the DQQ, it is far more cost efficient than traditional dietary intake methods. Each nationally representative survey costs less than 1% of the cost of many national surveys that measure quantitative dietary intake in terms of grams and calories.
The GDQP is a good example of how a private sector organisation, such as Gallup, can support initiatives that are not necessarily the highest priority or visibility for NSOs. The simplicity of the survey, publicly available tools, resources and training guides, mean that the GDQP is a robust mechanism that countries and organisations can use to fill gaps in global diet quality data. For example, Switzerland runs a highly granular and robust assessment of national diet quality – the National Nutrition Survey – that is costly and time-consuming to conduct, so it is not frequently implemented. The DQQ was recently integrated into Switzerland’s nutrition strategy, as a way of measuring the state of diet quality in the nation between iterations of the National Nutrition Survey – of heightened importance following external factors that impact dietary changes, such as post-Covid food inflation, or disruption to global supply chains following conflict in Ukraine. As such, the DQQ is not a substitute for Switzerland’s main dietary measurement system, but a highly effective, low-cost interim solution.
As evidenced above with the examples of FIES, Global Findex and GDQP, Gallup’s support has played an important role in shaping and developing the measurement frameworks used for quantifying issues central to national and global development. By working in tandem with leading global experts, custodian UN agencies and international donors, the private sector has collaboratively facilitated the generation of additional insights to tackle key global challenges.
3.Measuring highly sensitive topics and capturing hard-to-reach populations
A look at Gallup’s partnership with Walk Free, an Australian international human rights group, provides an example of how the private sector can work closely with experts to address gaps on critical but hard-to-measure topics, including modern slavery.
In 2012, Walk Free was established by philanthropists with the aim of “ending modern slavery within our generation” [16]. Since 2014, Gallup has been helping Walk Free, and subsequently the International Labour Organization (ILO) who joined the project as a technical partner and co-funder in 2015, to answer the question “how many people are living in modern slavery?”
In 2014 Gallup researchers worked closely with Walk Free and the Global Slavery Index Expert Working Group to develop an initial set of questions designed to capture a range of scenarios that could be classified as modern slavery. To partly address the limitations of a household-based sample when the target population is largely hidden, the survey instrument incorporated a network sampling approach, using the “family” rather than the “household” as the reference group, to increase the likelihood of identifying victims in a random sample survey.
In late 2014, the draft questions were cognitively tested in six countries, including on 36 adults who had been in modern slavery in Indonesia, Nigeria, Pakistan and Ethiopia. The results of cognitive testing were positive: Respondents largely understood the questions, recalled the information being sought, wanted to provide the information, and could respond in the format required. The exercise also identified the need for some adjustments in item wording and the working definition of the family network.
The validated questionnaire was first implemented in seven nationally representative household surveys in 2015 and has since expanded to 71 national surveys, with a total sample of 92,568 individual interviews. By using a hierarchical Bayes modelling approach, Gallup and Walk Free researchers have been able to estimate the rates of forced labour and forced marriage at the individual level, as well as the average prevalence of all forms of modern slavey at the national level [17].
The surveys are a core element of the methodology used to estimate the prevalence of modern slavery in Walk Free’s Global Slavery Index (GSI). The same Gallup-collected data are also foundational to the Global Estimate of Modern Slavery, a statistical partnership between Walk Free, the International Organization for Migration (IOM) and the ILO. In 2017, the estimate was adopted by the UN General Assembly to track global progress toward SDG 8.7 (eradicate forced labour, child labour, modern slavery and human trafficking).
Gallup’s adherence to international standards for independence, transparency in methodological and analytical procedures and high ethical standards with regard to respondent safety and confidentiality has enabled the success of these partnerships.
4.Challenges and opportunities: Partnering with international organisations
Partnerships between private organisations likeGallup and international organisations inevitably throw up challenges at all stages of the research process. At the outset stage of partnerships, insufficient financial resources can be a major challenge in collecting global statistics. In many cases, bringing together a range of like-minded, influential actors is therefore crucial in driving progress toward collecting global statistics. It is possible to overcome this challenge, but it can be a time-consuming process.
Challenges also arise during data collection and analysis. The aforementioned Food Insecurity Experience Scale (FIES) is one of two frameworks used by the FAO to measure malnutrition alongside the Prevalence of Undernourishment (PoU). An independent evaluation of the FIES [18] identified some discrepancies between FIES and PoU data at the moderate food insecurity level in a small handful of countries. These discrepancies risked blocking the publication of national level data. The most effective mechanism to address such concerns and enhance credibility was to make the FIES data publicly available to the broader academic community and independent researchers, highlighting the importance of private-sector transparency in collecting official statistics.
The final stage of any effort to collect global statistics – communicating and disseminating the findings – can pose challenges, primarily in getting the data into the hands of relevant organisations and stakeholders. Data of the highest quality and relevance can be painstakingly collected, but if it fails to reach those able to use it for informed decisionmaking, its utility will be limited. To overcome the challenge of relevant dissemination, successful partnerships have convened events that bring together key actors around new global statistics. Gallup hosts regular events with international organisations to launch new research and reports. For example, as part of the ILO’s centenary initiatives, Gallup partnered with ILO on the Global Women at Work Project and released a flagship report “Towards a better future for women and work: Voices of women and men” on International Women’s Day in 2017 [19]. The event brought together influential panellists and attendees to coalesce around the first ever global account of attitudes towards women and work.
Beyond hosting events, Gallup also draws upon wider communications strategies to disseminate global statistics as widely as possible. The Gallup News site regularly features articles about global statistics that come from its partnerships with international organisations, as well as sharing reports on alternative channels including social media to distribute findings to as broad a range of stakeholders as possible.
Ultimately, the most effective mechanism for increasing use of data is for the data to be made publicly available. Gallup works with, and supports, client organisations making their commissioned data publicly available. Examples of this include creating landing pages for project specific work (for instance, [20]); putting the data in a format which enables it to be accessed on academic archives (e.g. [21]); or by creating generic landing pages, such as Gallup’s Global Datasets for Public Use page [22].
5.Quality standards: Maintaining the highest levels of methodological quality
To preserve the credibility of SDG statistics gathered via public-private partnerships, assurances of data quality and adherence to ethical standards are critical. Methodologies used and the data collected should fully comply with the same internationally recognised quality guidelines that apply to NSOs, such as the UN Fundamental Principles of Official Statistics [23] and International Statistical Institute Declaration on Professional Ethics [24]. In particular, private sector data providers must comply with international standards for protecting respondents, consistent with human rights norms and principles. According to the UN Office of the High Commissioner for Human Rights, such principles include the capacity for data disaggregation to identify vulnerable groups, self-identification to give subjects the option to disclose personal characteristics, transparency in research design and data collection methods, and protection of subjects’ confidentiality [25].
Given the increasing importance of third-party data to address national and global data gaps, it is necessary that best practices are consistently adhered to by non-traditional data providers of official statistics across the data ecosystem. As a major private sector organisation with over 80 years of experience in survey research, Gallup has developed consistent processes, protocols and systems based on internationally recognised guidelines to ensure quality, transparency and reliability in the data it collects.
Gallup provides a full methodology document with every release of World Poll data describing each phase of the research process. This document includes details about survey design, sampling and weighting procedures, as well as specific question wordings and information detailing how Gallup standardises key variables like income and education. An accompanying Dataset Details document lists complete methodological information for each country in which data has been collected annually since 2005. This information includes: interview dates, sample size, design effect, margin of error, mode of interviewing, languages and areas of exclusion.
Gallup’s commitment to preserving transparency in how the World Poll is conducted requires maintaining high methodological standards at every stage of the research process, from the development of new question items to the reporting of results.
5.1Development of new metrics
Over the years, Gallup has partnered with some of the world’s largest development organisations and charitable foundations to create additional World Poll question modules that fill gaps in national statistics. Such statistics are critical to advancing the wellbeing of populations and include partnerships with the World Bank and the UN Food and Agriculture Organization described above. Institutions like the World Bank and the UN have their own rigorous standards for data quality, which World Poll procedures must meet or exceed. A number of World Poll metrics are now used to track progress on societal domains, including law and order (SDG 16) and subjective wellbeing.
In developing these modules, and all World Poll questions, comparability of the results across countries and territories is of central importance as this allows development experts to track progress and assess the effectiveness of interventions. Ensuring World Poll questions are interpreted consistently across more than 140 populations is one of the most challenging aspects of data collection. Gallup takes great care through cognitive interview and translation processes to harmonise how respondents around the world interpret the questions.
5.2Cognitive interviews
One important threat to the validity of survey data collection is item ambiguity and related concerns in the quality of item development. This threat arises when items are poorly constructed, but even when an item is well-conceived in English (or another language), it may fail to yield meaningful data when administered globally due to cultural differences or translation difficulties. Gallup’s cognitive interview process provides one means to identify and redress these problems before the survey is fielded. Gallup applies cognitive testing to every new World Poll question or module to improve its quality and translations, thereby making the final questionnaire far more robust and reliable in yielding actionable data as well as defensible against criticism.
More specifically, the purpose of cognitive testing is to understand what respondents can comprehend with a reasonable degree of validity. Respondents go through four basic stages when answering a question: 1) comprehension, 2) retrieval, 3) judgment and 4) response. The cognitive interview is used to identify these problems using a variety of methodologies (e.g., “think aloud” versus probing) and probes (e.g., concurrent versus retrospective; standardised versus active). Cognitive testing gathers feedback from respondents that helps researchers identify problem areas.
Cognitive interviews were indispensable in refining questions used for the World Bank’s Global Findex module. Gallup conducted in-person cognitive interviews in more than 20 lower-middle-income countries. The cognitive interviews were designed to test respondents’ understanding, comprehension and resonance of complex personal finance topics and account for local/cultural specificities in how people access banking services.
5.3Translation quality
As evidence of Gallup’s efforts to continually improve the global comparability of new question items, World Poll researchers modified the survey’s translation process in 2019. Back translations, in which content is re-translated from the target language back into its original language in literal terms, had previously been the standard process for checking disparities in meaning between the original and translated questions. This method was discontinued in favour of a dual translation process, a new approach consistent with best practices for conducting “3MC” (Multicultural, Multinational, and Multiregional Contexts) surveys developed as part of the University of Michigan’s Comparative Survey Design and Implementation (CSDI) Guidelines initiative [26].
Recognition of the pitfalls associated with back translations led to the development of the TRAPD model first developed for the European Social Survey [27]. TRAPD stands for translation, review, adjudication, pretesting, and documentation.
5.4Survey programming
All World Poll surveying is computer-assisted, using Computer-Assisted Personal Interviewing (CAPI) on handheld devices for face-to-face interviewing and Computer-Assisted Telephone Interviewing (CATI) software for telephone interviewing. Gallup’s CAPI and CATI partners possess the technological capabilities and software to implement all programming features that are crucial to the success of any project, including reliable randomisation.
Using a team of data analysts, Gallup has developed an automated processing system where all features of the data can be efficiently tested. Gallup’s quality assurance team reviews the output and finalises programming in preparation for pilots. Testing survey scripts before fieldwork is a crucial aspect of the success of instruments with complex skip logic and built-in randomisation. At the pilot phase, Gallup’s quality assurance team works closely with local partners to ensure the survey instrument is being executed correctly. In particular, the quality assurance team monitors local partner performance in:
• Consistency in module implementation
• Translation of the survey into local languages
• Data delivery in the required formats and timelines.
5.5Sampling
Gallup uses probability-based sampling methods for the Gallup World Poll, ensuring representative samples in each country – typically with sample sizes of
Gallup has well-established protocols to ensure that samples collected are of the highest quality. Specific methods used to enhance sampling quality include the following:
• Gallup collects extensive demographic information on participant households and individuals, which it uses to ensure the collected data is representative of expected populations. While Gallup does not use quota sampling in the field, it does compare participation by age and gender to known characteristics of a sampled region and can employ additional quality assurance (QA) processes if completed interviews do not generally match expected profiles.
• In face-to-face interviewing, interviewers make up to three attempts in case of non-contact either with a household or a selected respondent, in order to decrease non-response bias against residents who may not be home at a particular time of day. Similarly, in telephone interviewing Gallup makes up to five contact attempts to a single phone number before fully resolving the number. Additional attempts may be made if it appears they are likely to be successful.
• In a dual-frame telephone interviewing approach, some members of the population are dual users and have a chance to be selected from both landline and mobile frames, thus giving them a higher probability of sample selection. To account for this, Gallup identifies dual users by asking whether they own both types of phones. It adjusts their weights accordingly to address or correct for the over representation. For respondents sampled through landline, within household selection also takes place as part of the randomisation process.
5.6Data collection
During fieldwork, Gallup analysts ensure there are no missing or inconsistent data such as timestamps out of chronological order, incorrectly coded randomisation variables or failed skip logic. On discovery of a discrepancy or error, Gallup’s data analyst and regional Research Director will investigate the precise cause of the issue and provide a resolution.
Interviewer supervision is another key element to successful data collection. To ensure interviewers are following the methodology and executing the questionnaire properly, Gallup requires local partners to conduct validations for 20–30% of each interviewer’s output, across both face-to-face and telephone interviews. Interviews are validated in one of four ways: by supervisor accompaniment; in-person re-contact to verify random route and respondent selection protocols are implemented correctly; phone re-contact (back checks); or listening to recorded interviews.
Once fieldwork has been completed, the data collected goes through a rigorous quality control process that combines automated flags with qualitative expertise, including:
• Duration metrics: Gallup uses a variety of duration metrics to identify interviews, interviewers and Primary Sampling Units (PSUs) with anomalous data. Typically, Gallup looks for cases when item/section/interview length is much shorter than the average. This can indicate data falsification, interviewing shortcuts (e.g., skipping words in longer questions) or non-compliance with data collection protocols (e.g., letting an in-person respondent fill in the survey on the tablet).
• GPS metrics: For face-to-face field interviews, GPS coordinates are used to confirm interviewers are conducting interviews at the correct locations. Additionally, GPS coordinates are collected multiple times during the questionnaire administration – occasionally revealing that an interview began in one location but was completed elsewhere. Specified flag notification systems have been incorporated in the system to alert the supervisors.
• Respondent selection metrics: Respondent selection metrics let Gallup know an interview was conducted with the correct randomly selected respondent. There is a temptation for many interviewers to interview someone else in the household when the selected respondent is not available. To identify suspicious data coming from a given interviewer, Gallup has defined and actively reviews a specific subset of interview metrics. These metrics include the percentage of one-adult households per interviewer, mismatches between age/gender of a selected respondent in the enumeration screens versus in the questionnaire’s demographic section, overall response rate per interviewer and interviewer productivity per day.
After the basic structure and coding of a dataset is checked, each dataset passes through a set of in-depth data quality checks by a designated quality control (QC) analyst. Gallup’s QC analysts typically specialise in one or two geographic regions as they possess an in-depth understanding of regional peculiarities and risk areas. The QC analyst reviews the automated flags and investigates any interviewer-level anomalies. Additionally, at this stage the designated QC analyst and the regional Research Director review the content of the data to confirm that it makes sense for the country context and that there are no highly unusual substantive results, as this may occasionally indicate a coding error or implementation challenges with a survey question. Results are also examined for anomalous patterns, such as frequent straightlining on batteries or unusually high amounts of item nonresponse. At the end of the data processing cycle, Gallup can ensure the validity, reliability and accuracy of the collected data.
5.7Weighting
Data weighting is used to minimise bias in survey-based estimates to ensure samples are nationally representative for each country and is intended to be used for generating estimates within a country. The weighting procedure for the World Poll is formulated based on the sample design and performed in multiple stages.
In countries where data are collected face-to-face, first Gallup constructs sampling weights to account for any disproportionality in selection of primary and subsequent levels of sampling within each stratum. Sampling weights are calculated to account for any disproportionalities in allocation, selection probabilities of PSUs, SSUs and households within the ultimate cluster. Next, within selected households, weighting by household size (number of residents aged 15 and older) is used to adjust for the probability of selecting a single adult in each selected household, as residents in larger households will have a disproportionately lower probability of being selected for the sample. The product of these two steps constitutes the base weight. In countries where data are collected via telephone, Gallup constructs a probability weight factor (base weight) to account for selection of telephone numbers from the respective frames and correct for unequal selection probabilities as a result of selecting one adult in landline households and for dual users coming from both the landline and mobile frame.
Next, the base weights are post-stratified to adjust for non-response and to match the weighted sample totals to known target population totals obtained from country level census data. Gallup makes non-response adjustments to gender, age, and, where reliable data are available, education or socioeconomic status, and also trims weights to avoid extremes and reduce the variance of weighted estimates. Finally, approximate study design effect and margin of error are calculated using Kish’s formula. The overall design effect calculation reflects the influence of data weighting.
5.8Analysis and reporting
Possessing the technical and infrastructural capability to collect robust global data alone is insufficient; Gallup recognises that it must also be trusted to do so by other members of the global research community.
Transparency is critical in global research work. In addition to publishing the methodology documents described earlier, Gallup is committed to sharing data collected for the purpose of being released as a public good through a microsite where all such partnership data is housed [28], as well as through licenses to World Poll data in a public-facing portal called Gallup Analytics. These licenses enable Gallup to retain the commercial viability of the World Poll, while still providing wider access to the data.
Gallup also maintains strict public release guidelines for any survey results that enter the public domain – whether those results are released by Gallup or a partner organisation – that enable the research in question to be independently assessed. These include, inter alia:
1. When a survey conducted by Gallup for a client is publicly released, all survey results must be made available to anyone on request.
2. All survey releases must include the exact question wording, interview dates, interviewing meth-od, sample size, definition of the survey population, and size of sampling error.
3. Gallup must review and may request revisions to conform to Gallup’s methodological and analytical standards on press releases and other documents prepared to help in the public dissemination of the survey data.
4. The Gallup name may not be used in any paid advertising or similar promotional materials in support of a particular product, service or point of view.
5. Gallup will have final approval on all question wording and methodology for any survey designated for public release using the Gallup name.
Several different types of analysis can be conducted using World Poll data, such as cross-sectional estimates at the country and sub-group levels and estimates of change over time.
5.9Guaranteeing respondent safety
Gallup’s commitment to guaranteeing respondent safety is reflected in internal processes for maintaining data confidentiality and protecting respondent anonymity through data collection, data transfer and storage, and reporting processes. The organisation’s requirements for protecting respondent confidentiality are in accordance with the Code of Professional Ethics and Practices of the Worldwide Association of Public Opinion Research (WAPOR), the European Society for Opinion and Marketing Research (ESOMAR), the American Association for Public Opinion Research (AAPOR), the Council of American Survey Research Organizations (CASRO) and the American Statistical Association (ASA).
For World Poll research, Gallup’s data collection partners and field teams in each region are trained according to an ethics protocol aligned with the standards for the organisations listed above. Each person interacting with respondents or working with respondent data must sign a confidentiality pledge.
Gallup’s in-house Institutional Review Board (IRB) is another cornerstone of this commitment to protecting respondents from potentially harmful effects of participating in research. Gallup’s IRB is registered with the U.S. Department of Health and Human Services (HHS) and has 11 members, including two external reviewers. To ensure respondent safety, all research projects must receive approval from the IRB. The aim of the IRB is to ensure the design (particularly in the areas of consent, recording, transfer of personally identifiable information and use of GPS) minimises respondent burden and protects the rights and welfare of human subjects. It also ensures that the research project adheres to regional and country-specific guidelines, such as the EU General Data Protection Regulation (GDPR).
The IRB’s approach to the protection of human subjects’ welfare is rooted in Gallup’s adherence to the ethical principles and guidelines set forth in the Belmont Report disseminated by the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research [29]. In application, this means that Gallup’s IRB carefully reviews all research protocols to make certain they respect and incorporate the following principles into proposed research procedures:
1. Informed consent: Gallup’s IRB reviews informed consent statements to ensure they clearly articulate the privacy and confidentiality protections participants can expect.
2. Beneficence: The IRB conducts a risk and benefit analysis to ensure risks are minimised and that all risks are justified by the expected benefits of the research.
3. Justice: The IRB reviews all selection criteria and selection procedures to ensure subject selection is equitable.
5.10Experimentation to advance the science of survey research
Since the World Poll’s inception in 2005, Gallup has invested in experiments with novel data collection methods to address interesting research questions. Some examples include:
• Automated data collection using SMS and online tools to determine factors that drive perceived food insecurity.
• High-frequency data used as a vehicle for testing the uptake among the general population of alternative approaches to gathering data that would typically be captured in a census for a national statistical organisation.
In addition, Gallup has supported several client-funded experiments including non-response bias studies to understand the impact of design limitations on representativeness, the effect of incentives (both monetary and non-monetary) on participation as well as various elements of questionnaire design, such as the placement of items in a questionnaire, the effect of filtering on response distribution and the use of different reference periods for respondent recall.
Gallup has also helped mainstream measurement frameworks through their implementation in the World Poll. Among the most recent is the Degree of Urbanisation. Historically, measuring urbanisation has been challenging due to the absence of a common, internationally comparable definition of ‘urban’. In response to this critical measurement gap, the European Union, Food and Agricultural Organization, International Labour Organization, Organisation for Economic Cooperation and Development, UN-Habitat and the World Bank agreed a harmonised definition applicable to all countries worldwide. The Degree of Urbanisation was officially endorsed by the United Nations Statistical Commission in 2020 [30] and is available to GWP data subscribers and users of Gallup Analytics to enable further analyses. By integrating the Degree of Urbanisation into the World Poll, Gallup has helped promote the adoption of international best practices by reinforcing its use in the international system of official statistics.
6.Capacity building: Gallup’s role in building sustainable processes around the world
As discussed above, private organisations such as Gallup can build fruitful partnerships with international organisations and NSOs to develop and collect official statistics and research hardtoreach populations. In the process of bridging the gap to official statistics, Gallup’s role also brings a secondary benefit: building capacity among the broader global statistical ecosystem.
Gallup partners with a wide array of local research partners around the world to implement its global data collection efforts. Gallup regularly conducts in-depth training sessions with research partners, covering all stages of the data collection process from questionnaire design to fieldwork implementation and enumerator instructions. These training sessions are conducted by Gallup’s Regional Directors, who alongside their in-depth knowledge of their countries, also possess decades of experience conducting survey research. Training is particularly valuable in countries with less experience conducting nationally representative surveys.
The impact of the Gallup World Poll on the wider statistics ecosystem in sub-Saharan Africa has been particularly notable. Working with a range of local agencies since 2005, Gallup training sessions have guided the adoption of Computer Assisted Personal Interviews (CAPI) across the continent, greatly improving the quality of data collection. The result has been an enhanced knowledge about different survey aspects, such as neutral probing, research ethics, effective field supervision and response rates, to name a few. As local agencies have grown increasingly familiar with the Gallup World Poll, they have also developed criteria to help them recruit enumerators of suitable talent for survey administration. The more well-trained enumerators on the ground across the world, the better for the broader statistical environment. Gallup-trained enumerators in sub-Saharan Africa have been known to subsequently gain employment in NSOs, demonstrating the quality of their training and interviewing skills.
The result of such partnerships between Gallup and local research agencies is a broad dissemination of best-practice methods, survey infrastructure and quality control processes. These relationships are built on strong methodological foundations and help buttress successful partnerships between NSOs and private organisations.
7.Conclusion
The ability of many private sector organisations to collect highfrequency data, sustained over time, for delivery into the UN system is valuable at a time when SDG-related initiatives require a plethora of official indicators, heightened by the call to mobilise on the SDGs as part of the Decade of Action. In supplementing their efforts, Gallup demonstrates how the private sector can help in measuring some of the most pressing issues of our time and driving accountability to global organisations seeking to change the world for the better. In abiding by high ethical and methodological standards Gallup and other private sector organisations can also play a vital role in promoting high standards of quality for international measurement frameworks, thereby strengthening the wider research ecosystem on the ground across the world.
This article has demonstrated Gallup’s role in supporting the creation of such diverse and sustainable statistics. This role is wide in scope, from developing survey instruments, validating existing measurement frameworks and running experimentation to optimise data collection, all while ensuring the highest degree of accuracy, reliability and transparency.
References
[1] | United Nations Economic Commission for Europe. Measuring and monitoring progress towards sustainable development goals. (2020) . Available from: https://unece.org/sites/default/files/2021-04/2012761_E_web.pdf. |
[2] | Jüttig J [Internet]. SDG data discussion: what next? 2016. Available from: https://web-archive.oecd.org/2016-04-12/396871-development-posts-sdg-data-discussion-what-next.htm [Accessed 9 November 2023]. |
[3] | The DHS program [Internet]. Measuring the SDGs: the role of household surveys. (2017) . Available from: https://blog.dhsprogram.com/measuring-sdgs/. [Accessed 9 November 2023]. |
[4] | Biancotti C, Borgogno O, Veronese GF. Principled Data Access: Building Public-private Data Partnerships for Better Official Statistics. SSRN Electronic Journal. (2021) . Available from: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3896309. [Accessed 14 March 2024]. |
[5] | United Nations [Internet]. Big data & the SDGs. 2017. Available from: https://www.un.org/en/pdfs/Bigdata_SDGs_single_spread_2017.pdf. [Accessed 9 November 2023]. |
[6] | UNESCAP Statistics Division. Big data for the SDGs, country examples in compiling SDG indicators using non-traditional data sources. 2021. Available from: https://www.unescap.org/sites/default/d8files/knowledge-products/SD_Working_Paper_no12_Jan2021_Big_data_for_SDG_indicators.pdf. |
[7] | Campbell J, Neuner J, See L, Fritz S, Fraisl D, Espey J, et al. The role of combining national official statistics with global monitoring to close the data gaps in the environmental SDGs. Statistical Journal of the IAOS. (2020) ; 36: : 443–453. doi: 10.3233/SJI-200648. |
[8] | Demirgüç-Kunt A, Klapper L, Singer D, Ansar S. The Global Findex Database 2021: financial inclusion, digital payments, and resilience in the age of COVID-19. Washington, DC: World Bank; (2022) . |
[9] | Food and Agriculture Organisation of the United Nations [Internet]. Food and agriculture microdata catalogue 2023. Available from: https://microdata.fao.org/index.php/catalog/Food-Security/?page=1&sort_by=popularity&sort_order=desc&ps=15&repo=Food-Security. [Accessed 9 November 2023] |
[10] | Stiglitz J, Sen A, Fitoussi J. Report by the Commission on the Measurement of Economic Performance and Social Progress. (2009) . Available from: https://ec.europa.eu/eurostat/documents/8131721/8131772/Stiglitz-Sen-Fitoussi-Commission-report.pdf. [Accessed 11 May 2023]. |
[11] | Ballad T, Kepple A, Cafiero C. The food insecurity experience scale: developing a global standard for monitoring hunger worldwide. (2013) . Available from: https://www.fao.org/3/as583e/as583e.pdf. |
[12] | United Nations. UN statistical system organizations statement on goal 2. (2015) . Available from: https://unstats.un.org/sdgs/files/meetings/iaeg-sdgs-meeting-02/Statements/UNSSO%20statement_Goal%202%20-%20Oct%202015_rev.pdf. [Accessed 9 November 2023]. |
[13] | Gennari P, Katz S, Cafiero C. Food and Agriculture Organization of the United Nations (FAO). The use of non-official sources for international food security and agricultural statistics. In special issue paper of the Committee for the Coordination of Statistical Activities (CCSA) Case studies*: using non-official sources in international statistics. (2016) . Available from: https://unstats.un.org/unsd/ccsa/documents/E-publication.pdf. |
[14] | FAO, IFAD, UNICEF, WFP, WHO. The state of food security and nutrition in the world 2023. Repurposing food and agricultural policies to make healthy diets more affordable. Urbanization, agrifood systems transformation and healthy diets across the rural–urban continuum. Rome; Italy; (2023) . Available from: doi: 10.4060/cc0639en. |
[15] | Global Report 2022 – Measuring what the world eats. GAIN. Available from: https://www.gainhealth.org/resources/reports-and-publications/global-report-2022-measuring-what-world-eats. [Accessed 14 March 2024]. |
[16] | Walk Free [Webpage on the Internet]. Australia; Walk Free; (2023) . Available from: https://report.minderoo.org/2022/initiative-walk-free#:∼:text=Walk%20Free’s%20mountaintop%20%E2%80%94%20to%20eradicate,well%20as%20global%20legislative%20reform. [Accessed 9 November 2023]. |
[17] | Diego-Rosell P, Joudo Larsen J. Modelling the risk of modern slavery [Internet]. papers.ssrn.com. Rochester, NY; (2018) . Available from: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3215368. |
[18] | FAO. Evaluation of the Project “Voices of the Hungry”. [Internet] Available from: https://www.fao.org/publications/card/en/c/I9282EN/ [Accessed 14 March 2024]. |
[19] | ILO and Gallup launch landmark report on women and work [Internet]. www.ilo.org. 2017. Available from: https://www.ilo.org/washington/events/WCMS_549457/lang–en/index.htm [Accessed 14 March 2024]. |
[20] | Gallup. Degree of Urbanisation Variable [Internet]. Available from: https://www.gallup.com/analytics/315497/urbanization-data-variable.aspx [Accessed 14 March 2024]. |
[21] | UK Data Service. World Risk Poll, 2019 [Internet]. beta.ukdataservice.ac.uk. Available from: https://beta.ukdataservice.ac.uk/datacatalogue/studies/study?id=8739 [Accessed 14 March 2024]. |
[22] | Gallup. Global Datasets for Public Use. Available from: https://www.gallup.com/analytics/318923/world-poll-public-datasets.aspx [Accessed 14 March 2024]. |
[23] | United Nations. United Nations fundamental principles of official statistics, implementation guidelines [Internet]. (2015) . Available from: https://unstats.un.org/unsd/dnss/gp/Implementation_Guidelines_FINAL_without_edit.pdf. [Accessed 9 November 2023]. |
[24] | International Statistical Institute. Declaration on professional ethics adopted by the ISI council [Internet]. 2010. Available from: https://www.scb.se/contentassets/db09cdb81aae41dd8a153bb366b00a36/isi-declaration-on-professional-ethics.pdf. [Accessed 9 November 2023]. |
[25] | United Nations [Internet]. A human rights-based approach to data – leaving no one behind in the 2030 agenda for sustainable development. (2018) . Available from: https://www.ohchr.org/sites/default/files/Documents/Issues/HRIndicators/GuidanceNoteonApproachtoData.pdf. [Accessed 9 November 2023]. |
[26] | Cross-Cultural Survey Guidelines [Internet]. Guidelines for best practice in cross-cultural surveys. (2016) . Available from: https://ccsg.isr.umich.edu/. [Accessed 1 July 2023]. |
[27] | Behr D. Translating questionnaires for cross-national surveys: a description of a genre and its particularities based on the ISO 17100 categorization of translator competences. Translation & Interpreting. (2018) ; 10: (2). doi: 10.12807/ti.110202.2018.a02. |
[28] | Gallup [Internet]. Global datasets for public use. Available from: https://www.gallup.com/analytics/318923/world-poll-public-datasets.aspx. [Accessed 9 November 2023]. |
[29] | Office for Human Research Protections [Internet]. The Belmont report. (1979) . Available from: https://www.hhs.gov/ohrp/regulations-and-policy/belmont-report/read-the-belmont-report/index.html. [Accessed 9 November 2023]. |
[30] | European Commission, ILO, FAO, OECD, UN-Habitat, World Bank [Internet]. A recommendation on the method to delineate cities, urban and rural areas for international statistical comparisons. 2020. Available from: https://unstats.un.org/unsd/statcom/51st-session/documents/BG-Item3j-Recommendation-E.pdf. [Accessed 9 November 2023]. |
