The open discovery initiative & promoting transparency in discovery

1.1.History

The Open Discovery Initiative (ODI) project grew out of a meeting of interested parties held at the American Library (ALA) Annual 2011, convened by Marshall Breeding (an independent library consultant) and Jenny Walker (then of Ex Libris). At the time, web-scale discovery services - massive databases that indexed both multiple vendors’ abstracting and indexing records as well as the full text of articles from scholarly and other journals - were first coming to market. Serials Solutions’ Summon product (later a division of ProQuest) arrived first, joining OCLC’s FirstSearch and, soon thereafter, by the EBSCO Discovery Service (EDS). As these products were coming to market, librarians and content providers wanted to better understand what data was available in a web-sale discovery index relative to the independent products that comprised it, what metadata was indexed in them, to what extent a library needed to subscribe to both the vendor product and the discovery product to receive benefit of their investments, how linking decisions were made, and much more.

Out of those first conversations grew a NISO Working Group, the Open Discovery Initiative (ODI), to explore these questions and develop a NISO Recommended Practice around web-scale discovery. ODI’s members, representing libraries, content providers, and discovery providers, met intensively over three years from 2011–2014 to explore mutual concerns and issues among the three parties and develop a Recommended Practice (RP). This original RP was designed to establish basic information sharing so that all three partners in the discovery process could understand each other, what metadata and full text were expected to be made available through the ecosystem, what was indexed, and what was displayed in the discovery interface. This first RP included such items as affirmations of core/required metadata (basic information elements about an item that included titles, authors, abstract, publication dates, etc.) and additional metadata elements that were recommended to be shared by content providers and indexed by discovery providers, but were not required to be in ODI conformance. The Recommended Practice also noted a number of metadata transfer formats, depending on the type of content, to ease information sharing between content providers and discovery providers.

With publication of the original Recommended Practice in 2014, the original Working Group transitioned to a Standing Committee to oversee the RP and determine when the time was appropriate to work on a new version of the document. By 2017, the discovery ecosystem had evolved to the point where there were new content types in the discovery environment that were not previously well described. New questions arose about how choices were made by discovery providers as to what source would be the default link for full text links and the need for library-focused best practices in configuring and maintaining their discovery systems. With these opportunities in mind, the ODI Standing Committee embarked on what turned out to be a three-year process to update the 2014 Recommended Practice, releasing a revised version in August 2020, after internal discussions, research, and a public comment period.

1.2.Goals and benefits

The Open Discovery Initiative’s tagline is “promoting transparency in discovery”. That phrase is at the center of the Recommended Practice. ODI’s goal is to allow each of the three partners in discovery - libraries, content providers, and discovery providers - to understand what is expected of them, what is available to them, and how data flows from its source to the end user.

From the library perspective, web-scale discovery systems are increasingly important to the work of libraries in service of their users. As these discovery systems become more complex, librarians are less able to understand or explain to their users what content is included or how. The Open Discovery Initiative’s recommended practice represents a significant opportunity for libraries to understand what is indexed by their chosen discovery service, the sources from which that content is derived, and how best to search against it. Additionally, the library conformance statement introduced with the 2020 revision provides libraries with a checklist of regular tasks and activities to help ensure that library investments in discovery are managed effectively.

From the content provider perspective, being able to address discovery and library concerns and ensure that their end customers (libraries) are getting the full benefit of the content that they license is important. Especially for smaller content providers, knowing the “normal” paths to providing their content to discovery indexes can save effort and investment. The content provider conformance statement, through its standardization, provides content providers with a straightforward way to communicate with libraries about the metadata and content libraries should expect to see in their discovery environment.

And finally, for discovery providers, being able to affirm to both libraries and content providers how metadata and links to full text are indexed, displayed, and managed in the discovery interface can build trust in the discovery-to-delivery ecosystem.

Ultimately, of course, it is the user of library resources who benefits from the transparency created through the Open Discovery Intitiative’s conformance statements, though the user is likely unaware of this hidden work behind the scenes.

2.1.Content provider role

For the library user to receive these benefits, content providers, discovery providers, and libraries need to work together to achieve maximum transparency and usability in discovery systems. Each party has its own role to play in the successful implementation of a discovery layer. Content providers, as mentioned above, can address library and discovery concerns by providing transparency and access to their content that is indexed in a discovery system. The most crucial service to libraries and discovery systems a content provider can undertake is to ensure that there is core metadata for each of the records supplied to the discovery system. For material to be discoverable in the enormous universe of discovery system content, at a minimum, the following core metadata elements need to be included when applicable: title, author, author identifier, publisher name, volume, issue, page, date(s), item identifier, component of title, component of title identifier, item URL, Open Access Designation, Full text flag, Content type, Content format, Language, Indexing data, and Abstract. For discovery systems to work effectively, the metadata must include enough description to make the sources discoverable amongst millions and sometimes billions of records.

The availability of core metadata has a direct impact on the discovery experience for users. Several studies have demonstrated the influence of good metadata on search results. In a 2021 study examining how discoverable MARC records are within a discovery system, Woolcott, Payant, Skeen, and Daybell found that certain metadata fields, particularly the contents notes and the summary notes, will often contain the search phrase or search terms that the user has entered in their search [1]. In a 2005 study Gross and Taylor found that controlled vocabulary subject headings were important in the return of search results and that without this metadata users would be 35% less successful in their searching [2]. A more recent study done by Irina Trapido in 2016 found that metadata quality had an impact on the success of the discovery system user and that it is important to “… ensure consistency, granularity, and uniformity of metadata, especially as libraries are increasingly faced with the problem of integrating heterogeneous pools of metadata into a single discovery tool” [3]. The ODI Recommended Practice provides a framework that encourages the necessary consistency, granularity, and uniformity that will ensure that the end user is better able to effectively use the discovery system.

Content providers will often include data points that support direct linking to an article or other resource. Even if these data points exist, the recommended practice stresses that direct linking should not replace OpenURLs, as omitting this link could harm the library’s ability to choose the platform to which they direct users or to provide a fallback access method should the direct link not be successful. While there is concern that OpenURLs can be unreliable, it is important to note that with the correct metadata OpenUrl is accurate. In most cases these failures can be traced back to metadata errors [4,5]. To counter these potential issues NISO has produced a recommended practice, Knowledge Bases and Related Tools (KBART) [6]. As citations to content will often appear from multiple content providers it is important to librarians to be able to control both the order of links to the platforms and which platforms through which the library user will link to content. This does not preclude direct linking, but direct linking should not take the place of OpenURLs.

A third area of importance to content providers is the indication of whether an item is viewable without any authentication. If an item can be viewed with no authentication it should have “free-to-read” included in the metadata. It is worth noting that Open Access materials are included in the free-to-read specification and that the recommended practice includes a mechanism to include metadata at the record level stating the availability of that content viewed by the end consumer. When this information is present in an item record, discovery platforms will pass it on to the user interface.

2.2.Discovery provider role

Above it is stated that discovery providers need to provide clarity on what is included in their indexes, both in terms of the specific journal or other resource titles that are added to the index, and the larger market product bundles (for example, databases that index a range of publications). Libraries and their end users need to understand what is available through the discovery system’s central index so that they can use the discovery layer in the best and most effective manner possible. In 2013, Buck and Steffy noted that many librarians consider an important part of discovery tool instruction to be providing an explanation of how the tool works and what it encompasses [7]. In their 2011 presentation, Howard and Wiebrands noted the propensity of librarians to be concerned with the coverage of specific databases within the discovery layer. A lack of coverage information has a negative impact on a librarian’s ability to communicate to students what was being searched [8].

Furthermore, a key pillar of the Framework for Information Literacy for Higher Education, Strategic Exploration, requires that a student be able to evaluate potential information sources. For a user to figure out scope, design and refine search strategies, and identify what parties may produce information about a particular topic, they need to understand not just that the discovery system is an organized system, but that it is not complete and does not contain “everything” [9].

The elements that discovery providers need to be included in a content report include, at the collection level, Provider, Market product (database name), number of titles in central Index, number of titles in knowledge base, number of unique records in central index, percent of records full text searchable in central index, percent of records from provider that are subject searchable in central index, percent of articles that are free to read, date of last market product update, and date of last report update. The ability to drill down to a detailed title list is also important. This means fields such as title, format, and ISSN number should also be available. The Discovery Provider also needs to consider the following issues:

2.3.Librarian’s role

In addition to the responsibilities of the Content and Discovery Providers, there are several areas to which libraries need to pay close attention: configuration, training and conformance.