Optimising port arrival statistics: Enhancing timeliness through Automatic Identification System (AIS) data

1.Introduction

National Statistical Institutions (NSIs) are trusted to produce high quality statistics to generate independent insight for all. Traditionally NSIs have relied on surveys to produce official statistics but more recently the trend is to leverage administrative data in Government databases to meet the ever increasing demand for statistical information on a broader range of subjects, with increased timeliness and at greater granularity [1]. Today, in a world where global data creation is projected to surpass 175 zettabytes by 2025 [2], novel data sources are providing new opportunities for NSIs to meet this rising demand with constrained budgets [1, 3, 4].

To exploit the value of these novel data sources (including big data), NSIs, and international statistical organisations such as Eurostat and the United Nations Statistical Division (UNSD) are working together and collaborating to explore how the potential of these new data sources for official statistics can be exploited [5, 6]. Studies to date, including mobile network data, smart city data [7] and web scraping for labour market statistics [8] have shown the potential of these new data sources in official statistics.

Ocean shipping is the main mode of transport for international trade activities [9] making up 80% of global trade in terms of volume [10]. The production of maritime indicators is vital for the understanding of trade activity. Automatic Identification System (AIS) is the international system for tracking ship movements which was originally developed by the International Maritime Organization (IMO) [11]. Ships of a certain size and type are required to carry an AIS transponder that transmit location and other key information in real-time as signals. This paper focuses on near real-time shipping data generated by AIS that can enable the faster compilation of economic and maritime indicators [4].

Data generated by AIS is an example of big data due to the large amount of signal data generated by ships globally. It offers potential for statistical analysis going beyond its original goal of maritime safety [12]. The data can be used to generate more timely maritime statistics on port calls, berth calls, carbon emissions, trade routes or port congestion. The UNSD has set up an AIS task team to evaluate the potential of this data [6]. The European Statistical System also has a similar initiative as part of its ESSnet Big data project [5].

This paper focuses on calculating port visits in Ireland using AIS data. The article resulted from a collaboration between the Central Statistics Office (CSO) and UNSD. This partnership was formed to explore whether AIS data could be used to generate official maritime statistics. This is the first time the CSO has used AIS data to produce faster maritime indicators. Ireland, as an island relying heavily on sea transportation, is ideally placed to capitalise on the potential offered by faster AIS-based maritime indicators.

Several NSIs have published experimental statistics with this data source in order to reduce the time lag between the reference period and the publication of official maritime statistics. The Office for National Statistics in the UK has published more timely shipping indicators [13, 14]. Statistics Denmark publishes regular experimental statistics on port calls to Danish Sea ports [15]. Statistics Netherlands considers the data source as having significant potential [16]. The UNSD has also produced a handbook providing a snapshot of case studies from around the world including Ireland [12]. The case studies include port calls, tracking of fishing fleets and CO2 shipping emissions which were carried out by the CSO, other NSIs, the United Nations and Eurostat. However, overall, the usage of this data source by NSIs is still limited and the above-mentioned projects remain classified as experimental in nature.

There is a lack of studies that have compared outputs from this data source with official statistics to assess its suitability for official production [4]. This paper addresses this gap by comparing port call statistics compiled from AIS sourced data with CSO’s official statistics compiled using administrative data sources.

This paper is laid out as follows. The next section is a literature review on the usage of big data in official statistics. Section 3 covers a description of the data sources and the deployment of the H3 spatial index on AIS records and the data processing environment and includes an overview of the United Nations Global Platform. Section 4 presents a new method, Stationary Marine Broadcast Method (SMBM), that can be used to efficiently identify stopped ships that can subsequently be used to identify port visits. In Section 5 the results are presented. The paper is then concluded with a discussion of results and some final remarks.

2.Literature review

In recent years, the usage of big data in official statistics has been debated across NSIs and has presented both challenges and opportunities for the field of official statistics. NSIs acknowledge that big data cannot be ignored in this modern world. Ignoring the potential of big data can question the relevance of NSIs and risks pushing them out of the information market [17].

A common view among researchers is that traditional surveys and big data can and should be used together to maximise the value of each [21]. For example, big data can complement existing surveys and can fill gaps in coverage, providing insights into hard-to-reach populations. For example, NSIs have started using big data to complement traditional statistics, including on migration [22]. One key benefit is the ability to enhance the timeliness of data, providing more up-to-date information. Traditional statistical methods often involve time-consuming data collection and processing. The usage of big data, like AIS, offer real-time or near-real-time data to produce official statistics. Harchaoui and Janssen [18] have shown how the U.S consumer price index could be produced in a timelier manner using prices available online, combined with web scraping technology.

Big data sources enable statisticians to generate analysis quickly to provide more up-to-date information for decision-makers. Gallego and Font [19] successfully developed a methodology for the early detection of reactivation of tourist markets to help mitigate the effects of the COVID-19 crisis, using Skyscanner data. With its increasing availability and completeness, AIS data can provide more detailed and timely port statistics compared with administrative data sources [20]. The richness and variety of big data sources enable NSIs to gain a more comprehensive understanding of complex phenomena, leading to more detailed and granular analyses [23]. Analysis of big data can produce new findings about variables that have not been previously captured by official sources [21].

However, despite the potential benefits, the integration of big data into official statistics is not without challenges. Methodological issues, such as ensuring representativeness and addressing selection biases, pose significant hurdles [24]. Understanding the methods behind the sampling, measuring, and assembling of big data is critical to extracting value from big data [25]. Many big data sources, including AIS data, are not collected with statistical purposes in mind, leading to issues of reliability and comparability.

The quality of big data sources always needs to be critically assessed. For example, Bähr and his colleagues [26] have developed a framework to identify the possible sources of error when dealing with sensor data.

Privacy concerns are another critical consideration. Big data often contains personally identifiable information, raising ethical questions about the responsible use of such data [27]. Striking a balance between harnessing the power of big data and protecting individual privacy is a complex challenge that NSIs must navigate.

This paper presents a novel methodology for producing official statistics based on geospatial big data and proves it reliability and accuracy by comparing it against administrative data sources in Ireland.

3.Data sources and processing

In this section, we describe the AIS system, the data it generates and how it is collected and enhanced for statistical purposes. We then describe the UNGP before describing the H3 spatial index and how it is deployed on the data records to enable fast and efficient searching and sub setting.

The official statistics used for comparison purposes are the ‘Official Statistics of Port Traffic’ published by the CSO, Ireland. These statistics are compiled quarterly based on returns made by harbour authorities [28].

The reference years for the analysis are 2021 and 2022.

3.3The H3 spatial index

The data is further enriched by adding a spatial index to every AIS message record to enable detailed and efficient geographical analysis. The spatial index system used is H3, a system originally developed by Uber, containing sixteen resolution levels, each level comprising cells that cover the earth’s surface with a corresponding H3 Index [37, 38].

Figure 1.

H3 – Parent H3 Cell with Seven Children.

H3 Index, a hierarchical geospatial index, is based on cells shaped as hexagons. Every hexagonal cell, up to the maximum resolution supported by H3, has seven children cells below it in this hierarchy. The hexagons have the property of expanding rings of neighbours approximating circles (see Fig. 1).

Table 1

Indicative average H3 cell area by resolution

H3 index resolution	Average hexagon cell area hectare (Ha)
	(1 Ha = 10,000 SqM)
0	435,745,000
1	60,979,000
2	8,680,000
3	1,239,000
4	177,000
5	25,300
6	3,610
7	516
8	73.7
9	10.5
10	1.5
11	0.2
12	0.0307
13	0.0044
14	0.0006
15	0.0001

By way of illustration, Brazil is about the size of two H3 cells of resolution zero. The indicative cell size of H3 hexagons is shown in Table 1.

On the UNGP the H3 Index is added to each AIS message for all sixteen resolution levels. The UNGP uses this index to store the data in parquet files. This allows for efficient querying and updating of AIS data within the platform.

The main benefit of the H3 Indexing systems is that any position on the earth, and by extension any AIS message, can be assigned to a H3 cell at a given resolution via the cell’s corresponding H3 Index. Using the H3 Index allows analysts to go beyond the ships actual GPS location to a more generalised location. The H3 Index also allows areas, such as ports and ship neighbourhoods, to be defined in terms of H3 indices.

4.Stationary Marine Broadcast Method (SMBM)

The project developed a novel method, “Stationary Marine Broadcast Method” (SMBM), to estimate the number of port visits in Ireland using AIS data. The Python code and pseudo code for the newly developed method is available on the United Nations Global Platform Git Hub which can be accessed on request [12].

The basic idea behind the SMBM is that a ship needs to be stationary for an extended period of time in a port in order to load and unload. If a ship is stationary within a port area for long enough and has multiple AIS messages around the same zone, then it is highly likely to be a port visit. In a nutshell, under the SMBM a port visit was estimated from stopped ship events that occurred within a port area or polygon. A stopped ship event was based on a ship’s speed being reduced to zero within the port or area of interest followed by multiple AIS messages within a neighbourhood of the initial zero speed observation.

This method takes advantage of the H3 Index system to define an area or ‘neighbourhood’ around a stationary ship that can be used to identify when the ship leaves the neighbourhood.

Once a ship was identified as stationary or stopped (the triggering event) subsequent AIS signals were assessed to determine if the ship remains within the ‘neighbourhood’ or has left the ‘neighbourhood’ defined by the triggering event (the escape event).

A ship was determined to no longer be in the neighbourhood (i.e. the escape condition was met), if its H3 Index was not within the set of H3 Indexes that defined the neighbourhood of the original triggering event (stopping). To define the neighbourhood of a stopped ship, we used a H3 k-ring (griddisk) function that returns a set of H3 Indexes that surround a ship’s current H3 Index location. The term “k-depth” refers to the number of rings used to define the neighbourhood for testing the escape condition.

Depth 0 was defined as the origin index, k-ring 1 was defined as k-ring 0 and all neighbouring indices, and so on, an arithmetic series (see Table 2).

This function introduced two hyper-parameters: k-depth of the k-ring and the H3 resolution to use for the k-ring.

Figure 2.

H3 Neighbourhood: K-depth 3 and H3 Resolution 10 (example for Waterford Port).

For the SMBM a H3 Index resolution of level 10 was used where the width of a cell is approximately 150 m. This width also approximates the size of ships being studied in this article. A K-depth of three was used as this takes account of errors in reporting position, minor changes in position due to tide and current, and small operational movements of the ship (see Fig. 2).

The choice of k-depth and H3 index was based on a sensitivity analysis alongside visual assessments of the spatial mapping. The choice represents a balanced trade-off between k-depth and H3 index, reflecting typical ship and port characteristics.

All in all, the methodology and data processing for the SMBM can be broken down into eight steps which are summarised below in detail.

4.2Step 2: Data reduction

Figure 3.

Study Area: 10 km Bounding Box Buffer.

The data source contains a large volume of information not relevant for analysis. Therefore, it is useful to discard this information. To reduce the volume of data to be analysed, a study area was first defined based on a 10 km box of port areas (see Fig. 3). The AIS data was filtered to only include signals within the study area, reducing the amount of data to be processed by about 80% from the initial fetched AIS data.

5.Results

In this section, the AIS based estimates are presented and compared with the official statistics compiled from administrative data sources for six Irish ports.

Figure 4.

Comparison of official versus AIS-based Port Calls, Q1 2021 to Q4 2021.

Figure 5.

Comparison of official versus AIS-based Port Calls, Q1 2022 to Q4 2022.

In 2021, a total of 11,552 vessels arrived in the six ports according to CSO’s official statistics. In comparison, the AIS based results estimate 11,889 vessels called to the same six ports, a difference of 2.9%. In 2022 11,399 vessels arrived in Ireland based on CSO’s official statistics compared to 12,652 based on AIS data. Figures 4 and 5 compare the two sets of estimates over 4 quarters in 2021 and 2022 and indicate strong alignment in the trend across the quarters. They show that the AIS-based estimates follow the same trend as the official statistics, but the AIS-based port call figure exceeds official statistics in most quarters. This is in line with findings from Arslanalp et al. [4].

Two plausible explanations for this overestimate could be i) AIS based estimates offer better coverage than the official data compiled from administrative data and/or ii) AIS based estimates may include an element of overcounting. The Stationary Marine Broadcast Method risks over-counting arrivals when a ship stops more than once within a port area. A vessel that arrives and stays for several days within a port may visit several locations within the port polygon and a count will occur for each stopping event, for example, a vessel arriving in port, unloading its cargo, and then moving to another location to take on fuel or stores. Another scenario may be navigation within the port area and waiting for a berthing location to become available. Deduplicating overcounted observations using IMO/MMSI poses challenges due to the diversity in ship journey patterns and schedules, as well as differences in port characteristics. Implementing blanket deduplication rules might exclude valid visits, especially considering variations among ship types, such as high-frequency ferries vs container ships. In this context, we opted to retain the overcount to ensure a more accurate representation. While there is a risk of redundancy, maintaining the data allows for more comprehensive analysis.

The overall trends are very similar when looking at individual ports. However, the range of differences vary between the AIS based estimates and official statistics when looking at individual ports, see Table 3. Only small differences are observed for Bantry Bay. However, larger, and possibly more volatile differences are observed for the busiest port, Dublin.

Figure 6.

Port calls by type of Cargo, 2022.

Figure 6 provides a comparison by type of vessel, showing broadly similar breakdowns as would be expected. In 2022 cargo ships accounted for 76% of the total port calls in Ireland according to the administrative data, while both AIS-based port call methods show that cargo ships accounted for nearly 77% of visits. The smallest share of arrivals were dry bulk and specialised ships, and Fig. 6 highlights that AIS data also captures this breakdown correctly. The same was found for 2021. The key advantage of these AIS-based statistics is that they track the overall trend well and can be produced in real-time to provide early maritime estimates.

Figure 7.

Duration of Stopped Ships under 264 hours, 2021.

Figure 8.

Duration of Stopped Ships under 264 hours, 2022.

Figures 7 and 8 show the duration of stopped ships by vessel type. For comparison purposes we split general cargo into Roll-on Roll-off (RoRo) and not RoRo. The duration of port visits is not available in the administrative data sources used in the compilation of official statistics and Figs 6 and 7 have been included to show the potential of AIS data.

The assumption was that 264 hours (11 days) is a reasonable upper limit for a port visit, and as such, any port stops above this limit and those related to stopped ship events located near a dry dock/ship repair yard were excluded from the compilation of the AIS based estimates. Based on investigation of ships with long durations in port, a stay past 11 days indicates a reason other than economic activity, such as detainment. Figures 7 and 8 show the short turnaround time for RoRo vessels, while container and dry bulk vessels stay in port for a longer period.

6.Discussion and concluding remarks

The use of big data in official statistics is still relatively recent and can provide challenges. However, novel data sources provide new opportunities.

The AIS based data source is fairly new. A significant benefit of this data source is that there is one global standard, providing for the opportunity of easily compiled coherent and standardised maritime statistics. International standards are currently being developed based on experiences using this data in the UNGP and other projects.

The analysis in this paper does produce differences between the AIS based estimates and official statistics for the two years investigated. As noted earlier, official port statistics in Ireland are compiled from summary data provided by each port authority on a quarterly basis, and as such, detailed disaggregation beyond what is received is not possible without significant burden on the part of the port authorities. There are two possible reasons (or a combination there of) for the higher estimate in AIS based statistics compared to the administrative data. The first reason is that the difference could be explained by a higher coverage in the AIS based estimates – indicating that the current official statistics are underestimating the number of port calls. The second reason is that there may be an element of double counting included in the AIS based estimation procedure indicating over-coverage errors with the AIS based estimates. In all likelihood the differences are probably explained by some combination of these error types.

The AIS based source, like many novel data sources, will enable greater temporal granularity (daily and hourly statistics are possible if required). The AIS based data source also provides for other statistical opportunities including:

There is significant incentive to exploit the associated benefits for official statistics. However, directly replacing the existing system of official statistics with the AIS based estimates will involve a discontinuity as there is some incoherence between them.

The traditional approach, summary reporting by port authorities, has the advantage that different attribute values can be agreed over time for collection by port authorities. The traditional approach can adapt to changing needs. There is significant value for an NSI to have access to the underlying microdata used to compile reports and reduce the burden on the port authorities. Access to underlying microdata would also provide for more in depth analysis in the differences between the two sets of estimates. This paper recommends that where NSIs do not have access to underlying microdata to compile statistics that they should seek it. This would be in the spirit of the UN Fundamental Principles of Statistics [40] where NSIs need to be open and transparent in how the compile their statistics.

The methods and approach described in this paper have been designed to be applied in any geographical setting – they should easily translate to different country settings. The H3 Index greatly enhances processing efficiency. The SMBM allows for events to be quickly and efficiently identified from the message data. The paper considers that SMBM may overestimate port calls, and as such, the authors recommend further evaluation to investigate how big a problem this might be, and if it is indeed a problem, if the method itself can be adjusted to mitigate it.

Given the experiences in this project, the authors suggest the following course of action as a migration path to using AIS based data for compilation of official statistics.

The CSO should continue to develop and produce experimental statistics for port traffic in parallel with the official statistics. This will allow monitoring of how both sets of statistics behave over a greater time scale. The experimental statistics can be compiled at a greater granularity (say monthly) and exploit the attributes and possibilities available in the AIS based data sources for new statistical indicators of value. It should promote benefits of experimental statistics to users and seek feedback. Investigate how the traditional reporting used by the port authorities might best be deployed to enhance the new AIS based estimates – particularly where there are information gaps. In the meantime, timely publication of these experimental statistics will serve as early indicators for the official statistics.

Adoption of AIS based estimates worldwide holds the promise of higher quality statistics when we consider the UN Fundamental Principles. Such statistics would be more timely with higher granularity on many dimensions and use a standard method on a standard data source with global coverage that can easily be audited. The whole process can also be easily audited by external parties.

The methods developed and used in this paper are designed to be applicable in different geographical settings, and as such this paper should make a contribution in signposting the pathway to using the AIS data source for the compilation of official statistics.