A Shape Expression approach for assessing the quality of Linked Open Data in libraries

2.1.Background

The Semantic Web is a web of data that is machine-readable and includes a collection of technologies to describe and query the data, as well as to define standard vocabularies. Linked Data was introduced by Tim Berners-Lee [6] as a essential component of the Semantic Web to create relationships between datasets. Thus, the Resource Description Framework (RDF) [53] lies at the heart of the Semantic Web as it provides a standard model for data interchange on the Web and extends the Web’s linking structure by means of URIs. In addition, SPARQL provides a standardised query language for data represented as RDF in which a query can include a list of triple patterns, conjunctions, disjunctions, and optional patterns [54].

Libraries have traditionally provided the descriptive metadata of bibliographic records using standards such as MARC.11 While MARC is the most common formats used by libraries to publish bibliographic information, it presents limitations regarding its use as RDF, since MARC was not defined for a Web environment [12].

In this sense, several initiatives provide a more expressive and modern framework for bibliographic information based on Semantic Web technologies. Some examples include: Functional Requirements for Bibliographic Records (FRBR), the family of conceptual models [24], and Resource Description and Access (RDA) specification [40], the IFLA Library Reference Model (LRM) [25], the Bibliographic Ontology (BIBO) [15], the Bibliographic Framework Initiative (BIBFRAME) [31] and FRBRoo [52]. However, translating the old records into the new format is not an easy task [1], since libraries usually host large catalogues, including many types of resources that often require a manual revision to transform the data with accuracy.

Several major libraries (e.g., the OCLC, the British Library, the National Library of France), publishers, and library catalogue vendors have applied LOD to their catalogues in an effort to make these records more useful to users. For instance, the Library of Congress Linked Data Service (id.loc.gov) provides access to authority data. The Bibliothèque nationale de France (BnF) has published data.bnf.fr by aggregating information on works, authors and subjects. The Biblioteca Nacional de España (BNE) has transformed its catalogue to RDF and is available at datos.bne.es [51]. The Biblioteca Virtual Miguel de Cervantes (BVMC) catalogue has been transformed to RDF based on the RDA vocabulary to describe the resources [9]. The British National Bibliography (BNB) LOD platform provides access to the British National Bibliography published as LOD and made available through SPARQL.22

LOD promotes cultural heritage discovery and access by providing a resource context through the linking of bibliographic catalogue records to external repositories such as Wikidata, GeoNames and the Virtual International Authority File (VIAF). GLAM institutions are increasingly embracing the value of contributing information to open knowledge and collaborative projects such as Wikidata. In this sense, many institutions have linked their collections to Wikidata by means of dedicated properties. For instance, the property BNB person ID (P5361) at Wikidata links to the BNB LOD platform. The linking and enrichment of entities enables combining information from datasets stored in different places and with varying SPARQL endpoints [13].

2.2.Validating LOD

For researchers, data quality is a key factor when choosing a dataset for reuse [17,18]. In this way, several methods and tools have recently emerged allowing to assess the quality of datasets built using Semantic Web technologies. In addition, the research community has highlighted the need for reproducible research by providing articles, data and code [4].

The development of tools to support data validation has accelerated over the past few years [35]. SemQuire consists of a quality assessment tool for analysing aspects of quality of particular LOD. It recommends a series of 55 intrinsic, representational, contextual and accessibility quality metrics [30]. Stardog Integrity Constraint Validation (ICV) allows to write constraints that are translated to SPARQL in order to assess RDF triples in a repository [9]. DistQualityAssessment is an open source implementation of quality assessment of large RDF datasets using Apache Spark [42]. Luzzu is a platform that assesses Linked Data quality using a library of generic and user-provided domain specific quality metrics [16].

Shapes Constraint Language (SHACL) is a World Wide Web Consortium (W3C) specification for validating graph-based data against a set of conditions [55]. As a result of the validation process, SHACL provides a validation report described with the SHACL Validation Report Vocabulary that reports the conformance and the set of all validation results. It provides advanced features such as SHACL-SPARQL that can be used to express restrictions based on a SPARQL query.

ShEx enables RDF validation through the declaration of constraints on the RDF model [37]. ShEx schemas are defined using terms from RDF semantics such as node which corresponds to one IRI, a blank node or a literal, and graph as a set of triples described as subject, predicate, object. ShEx enables defining of node constraints to determine the set of a node’s allowed values, including their associated cardinalities and datatypes.

ShEx also enables the definition of constraints on the allowed neighbourhood of a node called Shape, in terms of the allowed triples that contain this node as subject or object. Listing 1 shows an example of ShEx to validate entities of type person described using FOAF.

Listing 1.

A ShEx Shape to validate a person described using the FOAF ontology. Person1 matches PersonShape including the required properties

There are several implementations of ShEx including shex.js for Javascript,33 Shaclex for Scala44 and Java ShEx for Java.55 In particular, shex.js includes a Simple Online Validator66 to provide a configuration file called manifest that can load a schema, load and execute a query against a particular SPARQL endpoint, and validate the nodes selected by the query. The combination of the validation tool, the ShEx definitions and the manifests offers a reproducible environment with which to replicate the research result. Work on a collection of ShEx schemas has also begun for several vocabularies.77

The international research community has become increasingly interested in applying and using ShEx for the validation of RDF data. One example is the description and validation of Fast Healthcare Interoperability Resources (FHIR) for RDF transformations by means of ShEx [45]. Moreover, ShEx is employed in several Wikidata projects to ensure data quality by developing quality-control pipelines [48]. ShEx is also used to facilitate the creation of RDF resources that are validated upon creation [49]. Another approach proposes a set of mappings that can be used to convert from XML Schema to ShEx [20].

While ShEx and SHACL behave similarly with simple examples, ShEx is more grammar-oriented (shapes look like grammar rules) and SHACL is more constraint-oriented. ShEx provides an abstract syntax that can be easily serialized to several formats. SHACL uses inference (e.g. checking rdfs:subClassOf relationships) while ShEx focuses on RDF nodes.

Table 1 contrasts all the tools mentioned above to assess LOD by using the following features: (i) published as open source; (ii) available for use or download; (iii) using a grammar-oriented and friendly syntax; and (iv) installation required to start using it.

2.3.Data quality criteria

The definition of LOD quality criteria has been attracting ever more interest. A LOD quality model specifies a set of quality characteristics and quality measures related to Linked Data, together with formulas to calculate measures [39]. A data quality criteria according to which large-scale cross-domain LOD repositories can be analysed provides 34 data quality dimensions grouped into 4 data quality categories [18].

With regard to libraries, a methodology for assessing the quality of linked data resources based on SPARQL query templates has been presented together with an extensive evaluation of five LOD datasets, including the BNE [29]. Another example is based on Europeana; it describes an approach for capturing multilinguality as part of data quality dimensions, including completeness, consistency and accessibility [11]. A new method and the validation results of several catalogues using MARC as a metadata format identifies the structural features of the records and most frequent issues [28]. Moreover, an extensible quality assessment framework which supports multiple metadata schemas describes the requirements that must be considered during the design of such software [27]. A previous computational analysis is based on art historical linked data to assess the authoritativeness of secondary sources recording artwork attributions [14].

A recent methodology provides the dimensions and data quality criteria to assess the LOD published by libraries (see Table 2) [10]. In particular, the dimension category includes the criterion Consistency of statements with respect to relation constraints. Let g be the dataset to assess and prop the list of properties, this criterion measures the extent to which the instance data is consistent by averaging the scores obtained from the single metrics mconRelat,i:

Let Rp be the set of all property constraints defined in prop,

Then we can define the metrics mconRelatRp(g) as follows:

In the case of an empty set of relation constraints (Rp), the respective metric should evaluate to 1.

However, in these previous works this criterion is used to assess only the properties rdfs:range and owl:FunctionalProperty. A more complete and detailed assessment based on the properties could be beneficial for potential institutions willing to publish and validate their LOD data.

These efforts provide an extensive demonstration of how LOD can be assessed, specifying how each criterion can be evaluated. Nevertheless, to the best of our knowledge, no evaluation has been conducted regarding the consistency of statements with regard to LOD relation constraints published by libraries using ShEx.

3.1.Identification of resources

The identification of resources is a crucial step when analysing the elements and properties to be assessed by means of ShEx.

Publication workflows in libraries are becoming ever more complicated as metadata maintenance is a dynamic and evolving process [5]. In this sense, bibliographic information is stored as metadata using common entities (e.g. author, work, date). Metadata comes in an increasing number of options, including FRBR, BIBFRAME, RDA, Dublin Core (DC), schema.org, Europeana Data Model (EDM) and FRBRoo. In addition, the vocabulary used to describe the contents can be complex, as in the particular case of FRBR based vocabularies in which entities typed as Work follow a hierarchical organisation that includes several layers.

Table 3 shows an overview of the main entities in LOD vocabularies used by libraries to publish their catalogues.88 Each entity may include several properties with different levels of granularity depending on the vocabulary. For instance, DC includes the properties dc:author and dc:contributor, while RDA provides a much more expressive vocabulary to describe the roles with works, expressions and manifestations.

The resources are identified by means of the SPARQL query in Listing 2 that shows an example of how to retrieve the different classes stored in an RDF repository. Several classes can be used to type the same resource. For instance, a book can be typed as dc-terms:BibliographicResource, as well as bibo:Book and schema:Book. Resources may be identified by descriptive URIs providing a readable description of the entity (e.g. person or work) or using opaque URIs based on identifiers that are non human-readable.99

Listing 2.

A SPARQL query to retrieve the different classes stored in an RDF repository

Once we extracted the main resources described in the repository and identified their type, we extracted the properties of each class using SPARQL queries. For instance, Listing 3 shows a SPARQL query to retrieve the different properties used by the class bibo:Book. In some cases, the fact of matching text against a regular expression pattern by means of the SPARQL instruction regex can facilitate the filtering and exclusion of properties that are not necessary for a particular purpose. For example, libraries may host a long list of distinct roles to describe how authors contributed to works (e.g. illustrator, transcriber or writer) though users may not be interested in these properties.

Listing 3.

A SPARQL query to retrieve the different properties used by the class bibo:Book

3.2.Definition of ShEx

A collection of RDF triples can be assessed by means of a ShEx definition to determine whether the collection meets the requirements defined in the schema.

According to the entities and its properties identified in the previous step, ShEx schemas are defined to assess RDF data. ShEx can be represented in JSON structures (ShExJ) – intended for human consumption – or a compact syntax (ShExC) – for machine processing – [36].

ShEx has several serialization formats [21]:

Following other approaches [47], the ShEx-based validation workflow for libraries consists of:

A manifest file includes several properties: (i) a label for the schema; (ii) a ShEx schema; (iii) a data label describing the dataset; (iv) a data property including a SPARQL endpoint; (v) the SPARQL query to retrieve the data; and (vi) a status property with the value conformant. Listing 4 shows an example of manifest file to test entities typed as Person in the LOD repository published by the BNE. ShEx manifests can be hosted on GitHub so they can be used by online services.

When defining the ShEx schemas to assess a dataset, previous examples of ShEx can be reused as a starting point. For instance, if a dataset is based on FOAF, we could use previous examples based on this vocabulary to define the new ShEx schema.

Listing 4.

An example of manifest file to test entities typed as person – that corresponds to the class ns2:C10005 – in the LOD repository published by the BNE

In addition, the definition of ShEx constraints for an existing dataset and its validation can be performed by means of graphical tools aimed at novices and experts; they enable combination and modification functionalities allowing the building of complex ShEx schema [8].

3.3.Validation results

The last step consists of the conformance of the entity data from the library with the ShEx manifest defined in the previous step.

The ShEx2 Simple Online Validator1010 can be used to test and experiment with ShEx. The prototype provides several examples showing how to use ShEx. The validator requires a ShEx expression and a SPARQL endpoint and query to retrieve the entities in order to assess them. The results are shown item by item which allows fixing possible issues in the definition of the ShEx rules.

The ShEx2 Simple Online Validator allows users to select a manifest using a button in the left-hand list. Once a manifest is selected, a query can be chosen using a button in the right-hand list. The validate button then produces a list of results according to the items retrieved by the query selected in the previous step. Users are allowed to edit the schema and query inputs in order to re-execute the query and the validation. The list of results may include errors detailing the resource and the property involved, together with a textual description. Some examples of errors and their interpretation are:

Prototypes and tools as illustrated in the example enable the reproducibility of the results. Researchers may thus replicate, reuse and extend findings, and thereby drive scientific progress. Nevertheless, there are some aspects to consider when using a LOD dataset published by a library for assessment. For instance, in order to use the ShEx2 Simple Online Validator, the DL must provide a SPARQL endpoint via the secure HTTPS protocol.

4.1.Selecting datasets

The selection of a LOD repository is a critical factor as well as a complicated task since many institutions are publishing their metadata as LOD. Choosing the right subject ensures the possibility of replicating existing results as well as presenting new challenges to researchers.

In this sense, benchmarks provide an experimental basis for evaluating and comparing the performance of computer systems [23,43] as well as the possibility of replicating existing results [46]. Previous research has focused on four LOD repositories published by libraries – BVMC, BnF, BNB and BNE – that serve as a benchmark and has discussed the methodology employed to evaluate linked data in libraries [10]. Other approaches provide a list of potential LOD datasets for reuse such as the LOD Cloud1414 and the list of SPARQL endpoints provided by Wikidata.1515

There are many aspects to consider when using a LOD repository. For instance, open licenses and clear terms of use and conditions are key when reusing datasets. Depending on the requirements, a SPARQL endpoint may be necessary in order to assess the information provided by the repository. Table 4 shows an overview of LOD repositories published by libraries and the vocabulary used.

In some cases, organisations provide a dump file instead of having a public SPARQL endpoint available. The Library of Congress, for example, suggests to download the bulk metadata and use a SPARQL engine to create custom queries such as RDF4J.1616

The ShEx Online Validator requires a public SPARQL endpoint that uses HTTPS to test the entities. However, some organisations do not provide this protocol in their services such as the BNE and BVMC. To showcase the re-usability of our methodology to assess LOD, we identified datasets in Wikidata and the current LOD Cloud whose description contains terms such as library, or are included in Sections 2–3, providing a SPARQL endpoint via the secure HTTPS protocol. As a result, we chose the following datasets for evaluation:

In order to show how our methodology can be adapted to other domains, we selected an additional dataset from LOD Cloud, the Linked Open Vocabularies (LOV).1717

The SPARQL endpoints publicly available are used to assess the LOD datasets. The main difference between the repositories is the vocabulary used to describe the information, in particular the entities and properties.

BnF and BNB are linked to Wikidata by means of specific properties. In this way, and in addition to the ShEx definitions created according to the vocabularies used by the libraries, we have created a ShEx schema per library to assess whether the resources linked to Wikidata were typed as human (wd:Q5) via the public Wikidata SPARQL endpoint provided by the Wikidata infrastructure.

4.2.The BNB Linked Data Platform

The BNB Linked Data Platform provides access to the British National Bibliography1818 published as LOD and made available through a SPARQL endpoint. The Linked Open BNB is a subset of the full BNB including published books, serial publications and new and forthcoming books, representing approximately 4.4 million records. It is available under a Creative Commons CC0 licence.1919

The dataset is accessible through different interfaces: (i) a SPARQL online editor; (ii) a SPARQL endpoint for remote access; and (iii) a web interfaces providing a search box to enter a plain text term.

The BNB dataset has been modelled and represented in RDF using a number of standard schemas including the British Library Terms,2020 BIBO, the unconstrained version of the RDA element sets, schema.org and DC, amongst others. In addition, the BNB dataset has been enriched by means of the creation of links to several external datasets such as Wikidata, GeoNames and VIAF. The Book Data model provides an overview of the main classes and properties involved in the data model.2121 Fig. 2 shows a summary of the main classes used in the BNB repository.

Fig. 2.

Main classes retrieved from BNB LOD platform based on BIBO, SKOS and FOAF controlled vocabularies, and how they interact to create meaning.

A ShEx definition was created for each class to perform the assessment. As an example, the definition corresponding to bibo:Book can be found in Listing 5. All the definitions were included in a manifest file that can be consumed by the ShEx validation tool [38].2222

Listing 5.

A ShEx Shape to validate the resources typed as bibo:Book at BNB Linked Data platform. Each line corresponds to a property based on a particular vocabulary used to describe the resources

In addition, we created a further ShEx schema to assess the resources linked to the BNB Linked Data platform of Wikidata by means of the property BNB person ID (P5361) were typed as human (wd:Q5).

4.3.Bibliothèque nationale de France as LOD: data.bnf.fr

The data.bnf.fr project endeavours to make the data produced by Bibliothèque nationale de France more useful on the Web using Semantic Web technologies.

The dataset integrates several databases including the BnF main catalogue, BnF archives and manuscripts, and Gallica. The data model is based on FRBR, FOAF and SKOS as main vocabularies and provides links to external repositories such as GeoNames, Library of Congress and VIAF.2323 The dataset can be used via a public SPARQL endpoint or as a dump file.2424

An overview of the main classes stored in the LOD repository has been extracted and is shown in Fig. 3. A new vocabulary has been defined to describe roles in which resources are linked to the Library of Congress subject headings (LCSH).2525

Fig. 3.

Overview of the main classes based on FRBR, FOAF and SKOS retrieved from data.bnf.fr and how they interact to create meaning.

Once we extracted the main resources described in the repository and identified their type, we extracted the properties for each class using SPARQL queries. For instance, Listing 6 shows a SPARQL query to retrieve the different properties used by the class frbr-rda:Work. In this case, 394 unique properties where identified to define the ShEx schema including a long list of roles.

Listing 6.

A SPARQL query to retrieve the different properties used by the class frbr-rda:Work. The FILTER instructions exclude the roles and relators properties

A ShEx schema was defined for each class to perform the validation as is shown in Listing 7. As in the previous use case, all the ShEx schemas were included in a manifest file that is used by the validation tool as an input.

Listing 7.

A ShEx to validate the resources typed as rda-frbr:Work at data.bnf.fr. Each line corresponds to a property used to describe the resources

Moreover, we created an additional ShEx schema to check whether the resources linked from Wikidata to data.bnf.fr by means of the property Bibliothèque nationale de France ID (P268) were typed as human (wd:Q5).

4.4.National Library of Finland as LOD

The Finnish National Bibliography was published as LOD in 2017. The dataset containing about 40 million of RDF triples and based on schema.org and BIBFRAME, was extracted from MARC bibliographic records. The dataset contains a wide range of materials, including books, maps, journals and digitized documents.

A ShEx schema was defined for each class based on schema.org (e.g. CreativeWork, Periodical, Person and Place) to perform the validation. Properties are mainly based on schema.org as well as additional vocabularies such as the agent and the unconstrained version of the RDA element sets. Groups of related items such as Periodicals and CreativeWorkSeries were assessed by means of the properties schema:isPartOf and schema:has_part (e.g. nlf-periodical.shex). The collection of ShEx schemas were collected in a manifest file that is used by the validation tool as an input.

4.5.Linked Open Vocabularies

The purpose of LOV is to promote and facilitate the use of well documented vocabularies in the Linked Data environment [50]. The vocabulary collection is maintained by the LOV team of curators and is constantly growing (749 as of April 2021).

The data model is based on specifications to describe vocabularies including Vocabulary of a Friend (VOAF) and VANN – a vocabulary for annotating vocabulary descriptions–, and additional vocabularies such as FOAF, schema.org and DC.

A ShEx schema for the most relevant classes is provided such as voaf:Vocabulary, foaf:Person, foaf:Organization and lexvo:Language. Since some of the main classes used in LOV such as foaf:Person and foaf:Organization are used in the rest of use cases of this work, ShEx schemas were reused. In addition, a manifest file was created that includes all the ShEx schemas to be assessed.

4.6.Results and discussion

In order to assess the datasets, the main resources were identified and validated using a random sample of 1000 items retrieved per entity and library from their SPARQL endpoints. A total of 37 ShEx definitions were created to validate the LOD published by libraries. Table 5 shows the description of the classes, ShEx and manifest files used to assess the BNB and the BnF. Fig. 4 shows the ShEx validation interface consuming the manifest file and presenting the evaluation results for the BNB Linked Data platform.

Fig. 4.

The ShEx validator interface that uses the manifest file provided for the BNB Linked Data platform to assess each of the ShEx definitions showing the results. Online access to run the examples is available at https://github.com/hibernator11/ShEx-DLs/.

Table 6 provides an overview of the data quality evaluation. All the assessed repositories obtained a high score, notably the BNB, the NLF and the BnF.

The BNB reached the highest score. We applied constraints to it based on several properties including dct:title, dct:language or schema:author. However, in some cases, the work’s author is missing and we recommend setting them as anonymous.2626 The 414 resources typed as foaf:Person and foaf:Organization fail due to the lack of the property schema:name.

The BNF obtains a high score, even though that some constraints are violated. For instance, resources typed as frbr-rda:Expression should provide a language and there are 12 cases in which the property dcterms:language is missing. The Wikidata property score for the BnF can be attributed to the fact that the property is used to link all types of content such as countries, persons and organisations.

Regarding the NLF dataset, we identified properties such as rdaa:P50025 for variant names of corporate body entities that provide a list of literals typed as rdf:langString and xsd:string, producing a data type mismatch error.2727 The node constraint can be updated (e.g., Literal*) to solve this issue in order to obtain a better performance. There are a number of errors due to the lack of the property schema:inLanguage, since the ShEx schema requires at least one value for the property. In addition, a more expressive vocabulary such as FRBR or RDA could be used to describe the relationships between aggregated works in order to differentiate the items involved (e.g. article, work and periodical publications) instead of using the relationships schema:hasPart and schema:isPartOf.

The lower score for LOV can be attributed to the lack of values for several properties. Among the LOV dataset errors are the following:

In general, the ShEx schemas could be improved by setting the constraint owl:sameAs to a more restricted cardinality (e.g. 1 or more) in order to identify resources that are not linked to external repositories. In some cases, the number of resources to test were below 1000 given the random selection, such as the case of the class voaf:Vocabulary from LOV.

Moreover, some resources may not include sufficient information to be assessed. For instance, the resources typed as skos:Concept only includes a rdfs:label in the BNB Linked Data platform. In some cases, the same schema can be used for different classes such as blterms:PublicationEvent and blterms:ProjectedPublicationEvent because they are based on the same properties and vocabularies. The ShEx definitions defined for the BnF are more detailed since the FRBR model provides additional classes to describe the resources compared to the BIBO vocabulary. Tables 7, 8, 9 and 10 illustrate the total errors per dataset aggregated by class.

The results of the assessment are useful for librarians in several ways since they provide valuable information with which to refine and improve their LOD catalogues. It is thus possible to identify potential properties that are not properly used to describe the bibliographic information. In the same way, they can measure the extent to which entities are described by means of a sufficient number of properties. For example, a librarian could be interested in assessing if the authors contain at least a name, a date of birth and an identifier matching a specific pattern.

The ShEx schemas provided in this study can be used as a starting point for other institutions willing to assess their LOD. In this way, the schemas could be further refined with additional node constraints as well as the incorporation of new vocabularies to assess further datasets. The adoption and use of this methodology in other contexts is also feasible as is shown in the variety of datasets and vocabularies used to assess the methodology.

With regard to the methodology, this approach is limited to one data quality dimension. In order to improve the methodology, additional data quality dimensions and criteria could be used such as license, completeness and trustworthiness (see Table 2). In addition, the ShEx schemas are based on the most relevant classes in each dataset.