GERBIL – Benchmarking Named Entity Recognition and Linking consistently

Article type: Research Article

Authors: Röder, Michael^{a; *} | Usbeck, Ricardo^b | Ngonga Ngomo, Axel-Cyrille^{a; b}

Affiliations: [a] AKSW, Leipzig University, Germany. E-mails: roeder@informatik.uni-leipzig.de, ngonga@informatik.uni-leipzig.de | [b] Data Science Group, University of Paderborn, Germany. E-mails: ricardo.usbeck@upb.de, ngonga@upb.de

Correspondence: [*] Corresponding author. E-mail: roeder@informatik.uni-leipzig.de.

Abstract: The ability to compare systems from the same domain is of central importance for their introduction into complex applications. In the domains of named entity recognition and entity linking, the large number of systems and their orthogonal evaluation w.r.t. measures and datasets has led to an unclear landscape regarding the abilities and weaknesses of the different approaches. We present Gerbil – an improved platform for repeatable, storable and citable semantic annotation experiments – and its extension since being release. Gerbil has narrowed this evaluation gap by generating concise, archivable, human- and machine-readable experiments, analytics and diagnostics. The rationale behind our framework is to provide developers, end users and researchers with easy-to-use interfaces that allow for the agile, fine-grained and uniform evaluation of annotation tools on multiple datasets. By these means, we aim to ensure that both tool developers and end users can derive meaningful insights into the extension, integration and use of annotation applications. In particular, Gerbil provides comparable results to tool developers, simplifying the discovery of strengths and weaknesses of their implementations with respect to the state-of-the-art. With the permanent experiment URIs provided by our framework, we ensure the reproducibility and archiving of evaluation results. Moreover, the framework generates data in a machine-processable format, allowing for the efficient querying and post-processing of evaluation results. Additionally, the tool diagnostics provided by Gerbil provide insights into the areas where tools need further refinement, thus allowing developers to create an informed agenda for extensions and end users to detect the right tools for their purposes. Finally, we implemented additional types of experiments including entity typing. Gerbil aims to become a focal point for the state-of-the-art, driving the research agenda of the community by presenting comparable objective evaluation results. Furthermore, we tackle the central problem of the evaluation of entity linking, i.e., we answer the question of how an evaluation algorithm can compare two URIs to each other without being bound to a specific knowledge base. Our approach to this problem opens a way to address the deprecation of URIs of existing gold standards for named entity recognition and entity linking, a feature which is currently not supported by the state-of-the-art. We derived the importance of this feature from usage and dataset requirements collected from the Gerbil user community, which has already carried out more than 24,000 single evaluations using our framework. Through the resulting updates, Gerbil now supports 8 tasks, 46 datasets and 20 systems.

Keywords: Semantic entity annotation system, reusability, archivability, benchmarking framework, Named Entity Recognition, Linking, Disambiguation

DOI: 10.3233/SW-170286

Journal: Semantic Web, vol. 9, no. 5, pp. 605-625, 2018