LegalNERo: A linked corpus for named entity recognition in the Romanian legal domain

1.Introduction

Named entity recognition (NER) is the task of identifying named entities (NE) in text [45], such as persons, locations, organizations, and proteins. Starting in 1995, within the MUC-6 conference [14], there have been periodic tasks on various aspects of NER. For example, for the CoNLL-2003 shared task [40], NEs were considered “phrases that contain the names of persons, organizations and locations”. In this context, in the biomedical domain, a number of works have addressed entities such as genes, proteins, diseases [17], cell types [36], and chemicals [13,18].

In the legal domain, the TREC conference had a dedicated track [7] for evaluating the application of Information Retrieval (IR) methods to e-discovery in the context of the U.S. civil litigation from 2006 until 2011 [26]. The Competition on Legal Information Extraction and Entailment (COLIEE) [19] ran over multiple editions allowed further exploration of tools and algorithms for information extraction in the legal domain.

This paper presents a manually annotated corpus comprising a subset of documents from the MARCELL Romanian corpus, with NEs in the legal domain. We considered the classical entity types (organizations, persons, locations) and time expressions as they appear in legal documents and added a new entity type in the form of legal references to documents (such as laws, government decisions).

The paper is structured as follows: in Section 2 we present related work; in Section 3 we introduce the annotation process; Section 4 describes different aspects of the corpus, such as the annotation levels, the representation of the linked data and statistics; Section 5 considers the usage of the RDF version; Section 6 presents use cases; Section 7 discusses quality and stability, and finally, we conclude in Section 8.

2.Related work

In the context of the EU project “Multilingual Resources for CEF.AT in the legal domain” (MARCELL)11 a large comparable corpus of legal documents for seven languages was created [43]. Comparable corpora can be exploited for improving machine translation [24] and parallel sentences can be identified automatically, potentially containing NEs. The Romanian sub-corpus [42], as well as the other MARCELL corpora, was split at sentence and token levels, lemmatized, annotated at token level (part-of-speech, dependency parsing, NEs), and finally the corpus was enriched with IATE terms and EUROVOC descriptors. All annotations in the Romanian MARCELL corpus were realized using automatic processes.

Existing Romanian NE corpora include: RONEC [10], Romanian TimeBank [11] and SiMoNERo [2]. The RONEC corpus contains 26,377 NEs, belonging to 16 different classes. The Romanian TimeBank is an annotated parallel corpus for temporal information. SiMoNERo is a gold standard corpus for the biomedical domain, manually annotated with four types of domain-specific NEs. SiMoNERo has 14,133 NEs distributed in 4,987 sentences. All these corpora contain entities such as organizations, persons, locations. None of these corpora contains legal texts or legal entities.

Dozier et al. [9] explore NER in legal documents such as US case law, depositions, pleadings and other trial documents. The types of entities include judges, attorneys, companies, jurisdictions, and courts.

Cardellino et al. [3] explored using the LKIF ontology [16] further mapped to the YAGO ontology [39] to train a NE recognizer, classifier and linker. The resulting system is applied to a corpus comprising judgements of the European Court of Human Rights. The authors recognize that in the legal domain NEs are also names of laws, typified procedures and even concepts. Furthermore, when dealing with human annotators they observe that the classes and subclasses of Document, Organization and Person were the most consistent across annotators.

Glaser et al. [12] explored the suitability of NER systems in the case of legal contracts. The entity classes are person, organization, location, date, money value, reference, and other.

Leitner et al. [22] introduced a German legal NE corpus comprising seven coarse-grained and 19 fine-grained classes. In this case, a “person” entity can be classified into a regular person, a judge or a lawyer. Similarly, a “legal norm” entity can be expanded into law, ordinance or European legal norm.

In MARCELL, Romanian NEs were identified using a general-purpose tool [27], available at that time for the Romanian language, that was not adapted to the legal domain, allowing only entities such as organization, persons, locations and time expressions. The tool was not trained on any legal texts, but since no legal-domain tool was available it was used on this corpus. Thus, the need for a legal-domain NER corpus and system became apparent, which led us to the development of the LegalNERo corpus.

3.Annotation process

Annotation was performed by five human annotators under the supervision of two senior researchers at the Institute for Artificial Intelligence “Mihai Drăgănescu” of the Romanian Academy (RACAI). Annotators followed specific guidelines,22 inspired by the Linguistic Data Consortium (LDC) guidelines.33

We considered five classes: person (PER), location (LOC), organization (ORG), time expressions (TIME) and legal document references (LEGAL). For person entities, we considered only person names. Titles and honorifics present in text near a person name were not included. Organizations must have some formally established association. Typical examples are businesses, government units and political parties. Locations are defined on a geographical basis. References are introduced similar to [21] and the coarse-grained class of [22], without additional sub-classes. Thus, they are references to legal documents such as laws, ordinances, government decisions, etc. Even though we only annotated the legal reference coarse-grained class, most of these entities can be mapped to fine-grained classes using automated processes, employing other linked data resources (Section 5).

Each annotator was given instructions on how to annotate the documents and then annotated a single document. We discussed issues or questions the annotators had. Subsequently, a collection of 100 documents was attributed to each annotator. Thirty documents (out of the 100) were shared with two other annotators, allowing us to compute inter-annotator agreement (IAA).

Corpus and account management for the annotators was realized through the RELATE platform [29]. Actual annotation was handled using BRAT [37], integrated into RELATE. This allowed the annotators to view one document, select the identified entity and then associate an entity type with the selected span.

After the annotation process, we computed IAA between each pair of annotators, using Cohen’s Kappa. This was accomplished at token level and led to an average Kappa of 0.87. We further investigated the differences and were able to detect some recurring mistakes, such as the inclusion of indicative words in the entities (for example “oraṣul Bucureṣti”/“the city of Bucharest” instead of just “Bucureṣti”/“Bucharest”). A script was created to correct these mistakes.

Finally, we constructed an application to merge the annotations into a single file. For each entity, the application shows all other entities overlapping the same span and allows the user to select the entities that go in the merged file. To aid the user, the application highlights entities found by multiple annotators.

Once all common annotations were merged we recomputed Cohen’s Kappa measure between the merged corpus and each annotator. This produced an average Kappa of 0.89 and we consider this to be the final result. According to Landis and Koch [20], a Kappa value greater than 0.81 is indicative of an “almost perfect” agreement. The remaining disagreements account for mistakes made by individual annotators, such as missing an entity or a sub-entity. This is particularly reflected in potentially ambiguous situations such as the legal reference “Regulamentul CE nr. 765/2008” (en. “Council Regulation EC No 765/2008”). In this case certain annotators identified “CE” as an organization sub-entity, while others did not. In a few cases the end-of-sentence punctuation coincides with the dot indicating an abbreviation. Some annotators included the punctuation in the organization entity abbreviation, while others considered it to be sentence punctuation and did not include it (for example: “S.R.L.” vs “S.R.L” or “AFER.” vs “AFER”).

4.Corpus description

4.1.Annotation levels

Raw text files were extracted from the Romanian part of the MARCELL corpus. They contain national legislation gathered by crawling from the public Romanian legislative portal.44 As described in [43], the texts were extracted from HTML and converted into TXT files. For the LegalNERo corpus, we selected 370 documents of similar size, published between 2020–2021. We performed an initial check to ensure that the files contain correct Romanian characters (with diacritics) and do not contain tables or other structures that may impact the annotations.

Annotation was performed using BRAT integrated into RELATE. Thus, the primary annotation output is represented by BRAT-specific files. Each line contains an entity ID, followed by entity type, text span (start and end) and the text. This annotation format allows for multiple annotations in overlapping spans.

We used UDPipe [38] on the text files for operations such as tokenization, lemmatization, part of speech tagging and dependency parsing. The resulting files were in CoNLL-U format.55 We added a new column “RELATE:NE” (the 11th column) for NE annotations. We mapped the identified annotation text spans to tokens using a BIO notation format [35]. The associated entity annotation is prefixed with one of “B-” (for entity beginning) or “I-” (for a token inside the entity). Tokens that are not part of any entity are annotated with “O” (“outside”).

The use of BIO scheme means that there is no support for overlapping entities. A token is associated with a single entity type. Therefore, we created two separate token-based annotations, stored in two folders: one for storing all entity types, without embedded entities, considering only the largest text spans, and another for storing only person, organization, and location entities and time expressions. Provision of the two folders means the corpus can be used either for legal domain annotations (considering the legal references) or for general annotations (the other entity types).

Initial annotations (BRAT and CoNLL-U Plus) were converted to RDF, specific to applications exploiting linked data. This increases the usability of the corpus and allows analysis using RDF queries and linking to external databases. Location entities can be resolved using geographical databases, such as GeoNames. The annotation is available in both CoNLL-U Plus files (column 12, “RELATE:GEONAMES”) and in the RDF representation (Figs 1 and 2).

Fig. 1.

Span-based annotation in ann format.

Fig. 2.

Token-based annotation in CoNLL-U Plus format.

4.2.Linked data representation

Already having the text span annotations (in BRAT format) and the token-based annotations (in CoNLL-U Plus format) we were faced with the problem of designing a schema useful for linked data applications. We considered multiple types of ontology-based representations:

• – CoNLL-RDF representation [5,6]. It directly translates from tab-separated CoNLL format to RDF by employing the prefix “conll” with the column name. It associates a token representation with the NLP Interchange Format (NIF) [15].

• – POWLA ontology [4], designed to support any kind of text-oriented annotation, allowing for “document layers” that contain the annotations.

• – NERD ontology [34] provides classes such as “nerd:Location”, “nerd:Person”, “nerd:Organization” and “nerd:Time”.

• – European Legislation Identifier (ELI) ontology provides a framework for structuring metadata of legislative resources and publishing them as linked data. It provides the “eli:LegalResource” class which is defined as a work in a legislative corpus, applying to acts that have been legally enacted (whether or not they are still in force).

• – GeoNames integrates geographical data such as the names of places in various languages, elevation, population and others. We linked location entities with GeoNames by using the feature identifiers. The annotation was performed automatically and then manually validated.

Table 1 presents the vocabularies used in the corpus. The key concepts and relationships expressed in the dataset are visualized in Fig. 3, and a description is given in Table 2. Some of the vocabularies from Table 1 were used only as part of metadata specification. Therefore, they do not appear in the diagram. A complete example is included in the Appendix.

Table 1

Used vocabularies

Prefix	Name	URI
nif	NLP Interchange Format (NIF)	http://persistence.uni-leipzig.org/nlp2rdf/ontologies/nif-core#
powla	POWLA Ontology	http://purl.org/powla/powla.owl#
nerd	NERD Ontology	http://nerd.eurecom.fr/ontology#
conllu	CoNLL-U tabular format	https://universaldependencies.org/format.html#
conllup	CoNLL-U Plus format	https://universaldependencies.org/ext-format.html#
eli	European Legislation Identifier (ELI)	http://data.europa.eu/eli/ontology#
gn	GeoNames	http://www.geonames.org/ontology#
rdf	RDF	http://www.w3.org/1999/02/22-rdf-syntax-ns#
rdfs	RDF Schema	http://www.w3.org/2000/01/rdf-schema#
owl	OWL	http://www.w3.org/2002/07/owl#
dcat	DCAT 2 Vocabulary	http://www.w3.org/ns/dcat#
dct	DCMI Metadata Terms	http://purl.org/dc/terms/
skos	SKOS Simple Knowledge Organization System	http://www.w3.org/2004/02/skos/core#
xsd	XSD	http://www.w3.org/2001/XMLSchema#
prov	PROV	http://www.w3.org/ns/prov#
foaf	FOAF	http://xmlns.com/foaf/0.1/
pav	PAV – Provenance, Authoring and Versioning	http://pav-ontology.github.io/pav/

Fig. 3.

Key concepts and relationships.

The corpus comprises multiple documents, represented as “powla:Document” elements. Each document is organized into three layers, corresponding to sentences, tokens and NE text spans. Each object’s layer is indicated by the “powla:hasLayer” attribute and the layer is linked to a document using the “powla:hasDocument” attribute. Tokens are linked to sentences, using the “nif:sentence” and “powla:hasParent”. The sentences document layer contains objects of type “nif:Sentence”. To maintain the order of sentences, the “nif:nextSentence” and “nif:previousSentence” relations are used. Furthermore, the “nif:firstWord” and “nif:lastWord” attributes are employed to allow direct access to the first and last tokens of a sentence.

The named entities document layer contains elements from the NERD and European Legislation Identifier ontologies. The elements also inherit from “nif:Phrase”, thus specifying the beginning (“nif:beginIndex”) and end positions (“nif:endIndex”) for associated strings. Furthermore, the GeoNames feature identifier (“gn:Feature”) is specified when available for corresponding “nerd:Location” entities.

An example encoding in the LegalNERo corpus is given in the Appendix. where a token d1_s1_1 of type Word for the “nif” ontology and of type Node for the “powla” ontology is described. The example given in the Appendix corresponds to the sample ann and CoNLL-U Plus annotations given in Figs 1 and 2.

5.Using the RDF version of LegalNERo

The LegalNERo corpus [32] is available for download from Zenodo66 as a single archive containing all representations described in this paper. In the “rdf” folder there is a file containing all triples in RDF-Turtle. A SPARQL endpoint77 is available from the RELATE platform, offered via Apache Jena Fuseki, with a graphical query interface.88 Figure 4 presents a SPARQL query to list legal references. This type of queries is useful in creating gazetteer resources.

Fig. 4.

SPARQL query to list legal references and result.

Additional examples are provided in Figs 5 and 6. In the first case, the SPARQL query allows listing of location entities with associated GeoNames identifiers. The result will contain only those entities that have a GeoNames identifier. Figure 6 makes use of the token layer and displays organization entities, tokenized, with associated UPOS tags concatenated. In this example, only entities comprised of up to five tokens are considered. This type of query is useful in finding patterns associated with the NEs present in the corpus. Patterns can then be used with simpler pattern-based NER systems, such as Stanford RegexNER, available from the Stanford CoreNLP [23] package.

The advantage of having a linked data resource comes from the ability to interlink it with other resources. Recently, a number of other Romanian resources were converted to linked data format [1] and are available on the same SPARQL server as LegalNERo. This enables complex federated queries to be performed across multiple resources. An example is the refinement of the legal reference class into multiple fine-grained classes. Considering the classes “law”, “decision”, “government decision”, and “government ordinances”, they correspond to the presence in the entity of different forms of the Romanian words “lege”, “decizie”, “hotărâre”, and “ordonanṭă”. Thus, we can exploit the RoLEX lexicon to obtain the word forms associated with the words and then use these to classify the existing entities (see Fig. 7).

Fig. 5.

SPARQL query to list location entities with associated GeoNames identifiers.

Fig. 6.

SPARQL query to list organization entities at token level (comprising up to five tokens) with associated UPOS tags.

Fig. 7.

Classify legal reference entities into fine-grained classes.

6.Corpus usage

In accordance with the LegalNERo corpus, we developed two NER models [31]: one for all the entities and expressions, and another dealing only with persons, locations, organizations and time expressions. These models are based on a recurrent neural network with a final CRF layer, trained using NeuroNER [8]. To improve performance, we used pre-trained word embeddings [33] representations trained on the Representative Corpus of Contemporary Romanian Language (CoRoLa) [41]. The models were integrated in RELATE [28,29] and are available for online usage and download,99 together with the embeddings.1010

In the context of the “Curated Multilingual Language Resources for CEF.AT” (CURLICAT) project [44], we aim to develop an anonymization solution for Romanian. Part of this solution, we need the identification of NEs. Of course the purpose is not to anonymize legislation, but we consider that the NER models developed based on LegalNERo, have the ability to complement other models developed on more general corpora and rule-based approaches. A current prototype of the anonymization solution is available in RELATE.1111

7.Quality and stability

The underlying structures used to construct the linked data representation of LegalNERo are stable in the sense that we do not plan to change any of the classes or available attributes. Additions will be in the form of new data, following the same structure. Furthermore, any future extensions with regard to available NEs will follow the same general structure, with new classes being added, without removing existing information. Finally, the versioning system provided by Zenodo allows retrieval of the corpus at any point in time and ensures its continuous availability.

Metrics for assessing linked data quality have been proposed [25,46], considering both the data content and the metadata. The embedded metadata provides an indication of the dataset provenance, improving also the trustworthiness of the dataset. We offer the corpus under a Creative Commons licence (CC BY-ND 4.0). This information is also part of the metadata (machine-readable) and indicated in human-readable format on the corpus download page.

Before release, the dataset was checked for syntactic validity and no errors have been found. Furthermore, there are no consistency issues with regard to the data structure (no misplaced classes or properties, no inconsistent values). With regard to the actual annotations, the only inconsistencies may arise from the IAA (Section 3). The SPARQL endpoint is offered from a research server, shared with other projects, and thus performance metrics (such as low latency, high throughput or scalability) were not considered.

8.Conclusion and future work

This paper introduced the LegalNERo corpus, a manually annotated corpus for NER considering legal references in the Romanian language, providing span-based annotations, token-based annotations and RDF-Turtle format. The corpus represents a subset of the MARCELL [43] legislative corpus; for certain applications these corpora could be used together. LegalNERo also provides annotations for sub-entities present inside the legal references. This can be exploited to allow usage of the corpus for training more classic NER systems considering only persons, locations, organizations, and time expressions. We further offer a SPARQL endpoint. Finally, the corpus was integrated into the Linked Open Data Cloud.1212

Our aim is to further use this corpus to construct an improved NER system for the Romanian legal domain. Currently available models achieved an average F1 score of 84% (considering all entities) and 84.70% (without the legal reference entity type). This already presents an improved performance compared to the one [27] previously used to automatically annotate the Romanian Legal Corpus [42]. We re-evaluated the old NER system [27] on the LegalNERo corpus and it achieved only a 49.38% average F1 score (with individual F1 scores 84.06% for time expressions, 56.7% for organizations, 26.85% for locations, and 19.3% for persons). This difference comes from the lack of legal-domain text used in training the old system. Even though the size of the corpus is small, it has proven useful in improving the performance of the NER system and can be considered an important first resource for NER for the Romanian language in the legal domain. Considering additional techniques, such as word embedding combinations [30] could prove beneficial in improving the overall performance.