LiteRate: A Shiny R app for improving data literacy in Indonesia¹

1.Introduction

Nowadays, statistical literacy has become a concern of every National Statistics Office (NSO) around the globe. The ability for the public to understand statistics is a prerequisite for successful communication. Amidst the present-day hoax and fake news, the need to reinforce the ability for the public users to understand figures and evaluate information are deemed increasingly urgent.

Statistical literacy is generally defined as the ability of an individual or a group to understand and comprehend statistics [1]. It includes the ability to read and communicate the meaning of data. The lack of statistical understanding can lead to numerous misinterpretations of official data which can be observed in media reports, daily newspaper articles, and in direct contact with our users.

As the leading portal for trusted data in Indonesia, the BPS-Statistics Indonesia (BPS) website has been providing extensive data that cover various subjects and areas. However, these data are presented in the form of tables and static graphs that are monotonous and lack interactivity. The static nature of information from the BPS website discourages users from exploring the data more. As the volume of quantitative information increases, tables need to be carefully designed so that they can still be understood. The more knowledge people have on evaluating figures and the reliability of official statistics; the easier they can assess the meaning and quality of the data. Hans Rosling’s Gapminder tool [2], for example, has changed the way data can be presented, in which international statistics is converted into motion and interactive graphics.

There are already numerous products from different sources that provide innovative tools for data communication. However, few are from official trusted sources. Questions emerge around how the official statistics community should best invest in this endeavor [3]. One way to achieve this is by creating a digital tool that can explain statistics in a clearer and easier way [4]. Organizations such as Eurostat [5] and International Association for Statistical Education (IASE) have developed a range of initiatives to promote statistical literacy. The IASE is leading the work on the International Statistical Literacy Project (ISLP) [6]. Inspired by these breakthroughs, we developed liteRate, an interactive web-based tool for rapid visualization of data published on the BPS website.

LiteRate is an excellent way of showcasing data, which unveils the beauty of data by converting uninteresting figures into attractive and interactive visualizations. LiteRate targets users of various knowledge levels including students, academics, journalists, and policymakers. The users can intuitively engage themselves in customizing the visual using the provided advanced widgets, such as slider, dropdown, and checkbox. There are numerous visualization options available, from scatter plot to boxplot, and from heatmap to word cloud. The user can also do further exploration by changing colors, adding/removing legend, faceting, zooming, panning, and annotating/highlighting important features. Moreover, users can perform simple descriptive computations and download the resulting outputs into various formats. These combinations give users the ability to gain a simple overview of complicated subjects as well as spot quick facts and any interesting features in a graph. These features and functionality of LiteRate are seen to increase data literacy among users of BPS online data.

2.Literature and open source software options review

R is a free and open-source language and environment that provides a wide variety of statistical and graphical techniques [7]. It compiles and runs on a variety of platforms (Windows, UNIX, and MacOS). We choose R for this project because it is a well-developed and effective programming language that provides services such as effective data handling, integrated collection of intermediate tools for data analysis, graphical facilities, and is highly extensible [8]. Some of the R library products used to build liteRate are ggplot [9] and rplotly [10], rpivotTable [11], and esquisse [12].

Shiny is an R package that allows the development of interactive web apps entirely in R, providing a powerful framework for disseminating official statistics [13]. Shiny is a combination of a dashboard, apps, and interactive documents, which are commonly used for data exploration and analysis. The main important features of Shiny are the user interactivity and widget control. Furthermore, a Shiny app can also be extended with themes [14], widgets [15], and JavaScript actions [16]. R Shiny [17] can be used to build a service that enables users to create on-the-fly data visualization which can be presented in a dashboard format.

RSelenium [18], on the other hand, is an R wrapper or a function which wraps another function for the Selenium 2.0 Remote WebDriver [19]. It provides a range of tools and libraries that enable and support the automation of web browsers locally or remotely. It allows developers to simulate common activities performed by end-users such as entering text into fields, selecting drop-down values, checking boxes, and clicking links in documents. It also provides many other controls such as mouse movement and arbitrary JavaScript (JS) execution [20]. JS execution is conducted inside the web browser. This step involves a series of additional processes including interaction with Document Object Model (DOM) [21] and making a Hypertext Transfer Protocol (HTTP) request [22] by utilizing JQuery. JQuery is a fast, small, and feature-rich JavaScript library that simplifies HTML document traversal, manipulation, and event handling [23]. Crawling data with RSelenium requires the server to run execution in parallel. To handle this challenge, we utilize containers provided by Docker. Docker is a full development platform to build, run, and share containerized applications [24]. It provides a way to run applications securely isolated in a container, packaged with all its dependencies and libraries.

RSQLite is an R package that embeds the SQLite [25] database engine in R [26]. We use SQLite in this project because it is a simple, single-user, embedded (serverless), and very light-weight database engine that implements table creation, updating, insertion, and selection operations, plus transaction management.

Figure 1.

The data section on BPS website.

3.Methodology

3.1Data crawling and extract, transform, load (ETL)

LiteRate works by parsing the data available on the BPS website and making them ready-to-use for further visualization and analysis. It takes the data published on BPS website as input by scanning all pages containing relevant data and scraping (extracting) the data (including the metadata) from the webpage. Pages containing data can be accessed from the menu located on the page’s left-hand sidebar (Fig. 1). This data is classified into three groups, which are “Social and Population”, “Economy and Trade”, and “Agriculture and Mining”. Each group has several related subjects that are presented on different pages and tables.

Each subject page has six tabs, which are concept, methodology, basic statistics, sectoral statistics, external link, and dynamic table. The tabs concept, methodology, and external link are not considered for ETL because these tabs are only intended to display explanations regarding definitions and methodologies. On the other hand, the tabs’ basic statistics and sectoral statistics, have two types of table format available: static and dynamic. The static table has an irregular format with non-standardized row and column names. Hence, we also excluded it from this research. Meanwhile, the dynamic table has consistent structures (rows and columns) and standardized metadata that enabled us to automate content and variable extraction. Furthermore, we defined that tables presented in the tabs basic statistics and sectoral statistics are just subsets of the tab dynamic table. Therefore, for purposes of this study, we crawled data from the tab dynamic table only.

As shown on Fig. 2, the tab dynamic table comprises several sections: subject, indicator, characteristic, time, and area. Each section is dynamically loaded using an Asynchronous JavaScript and XML (AJAX) request. However, the BPS website is protected from Cross-Site Request Forgery (CSRF) [27] that prevents outside access via an Application Programming Interface (API). We solved this challenge using Selenium in Fig. 3. Selenium opens the dynamic table page, gets the CSRF code, and makes an AJAX request by including the CSRF code in the payload request. Since each subject in dynamic tables has a different collection of variables (indicators, characteristics, time, and area), it was necessary to create a request for each variable combination using the AJAX request. The result is stored in the RSQLite database. Figure 4 shows the data crawling flow chart.

Figure 2.

The dynamic table page.

Figure 3.

AJAX request dengan CSRF.

Figure 4.

Flowchart data crawling.

Figure 5.

Example of metadata variable provided on SIRUSA.

Figure 6.

Automation testing using selenium IDE.

4.Result

The result of this research is a comprehensive Shiny app named liteRate. In general, there are three major features offered by liteRate:

Figure 7.

Pivot table in liteRate.

Figure 8.

An example of visualization options in liteRate.

Figures 7 and 8 provide examples of a pivot table and the visualization options that can be created using liteRate. The testing result (Table 1) shows that all the three scenarios successfully passed the 32 testing commands with a total time of 24 seconds.

5.Conclusion

We provide a new way to investigate data by providing a non-conventional tool to visualize BPS-Statistics Indonesia data that enables users to understand the information clearly and directly through some previously underexplored points of view. Although currently still in the prototype phase, we believe that liteRate could be a breakthrough for BPS in creating a user appetite for their data and helping to develop citizens’ understanding on how to use data.

The prototype requires further development in order to provide better customer support and improved functionality. To keep improving liteRate, we are encouraging users to provide feedback to us. Future work following this study would focus on (1) handling static-format tables, (2) providing a feature for conducting analyses such as descriptive tables and regressions, (3) collecting logs to analyze user behavior, (4) publishing liteRate on The Comprehensive R Archive Network (CRAN) to reach a broader audience, and (5) incorporating metadata to assist users in data interpretation.

We realize that the use of metadata in liteRate is still limited and only includes a few variables and indicators. It is because there is no integration between the BPS website (bps.go.id) with the BPS metadata website (sirusa.bps.go.id) or API that allows liteRate to get direct access to related metadata at SIRUSA. To empower users to make more informed use of the data, we plan to incorporate metadata into liteRate in a more comprehensive way by adding a flag (‘i’) that indicates there is metadata available for the data. By clicking on the flag, a user can obtain information which includes the statistical methods used for the data collection, data items and definitions, and the classifications used for these data items. We will also provide video tutorials about using liteRate to support users in learning how to use liteRate.