The role of the information community in ensuring that information is authoritative: Strategies from NISO Plus 2022

1.Introduction

Ensuring that information is authoritative has been a challenge for researchers, publishers, librarians, students, and lay public since the very first scholarly publications. This issue is far from solved. Rapid sharing of results during the research process – from data sets to preprints with multiple versions and copies on the Internet – has not made it easier to ascertain what information is authoritative. Peer review, which has been held up as the gold standard for decades, is increasingly under pressure due to the speed and volume of research. And retracted articles can continue a vibrant zombie existence on the web. But in the midst of a pandemic and a climate crisis we need authoritative information more than ever before. We report on a NISO Plus 2022 session that addressed what can be done to safeguard the integrity of scholarly content being created, disseminated, and used.

Because authoritative research results should be verifiable and reproducible, data has been one focus. The first section, on crowdsourced data on the Zooniverse platform (see: https://www.zooniverse.org), addresses the impact of intentional design on data creation and management. Opening up data sets by encouraging good data curation practices, asking reviewers to check the data [1] and ensuring that data sets are FAIR [2] has long been a focus of the information community. Academic publishers can sometimes feel that data is the responsibility of research institutes. However, because the platforms used to create, structure, manage, share, store, analyze and tag research data are often created by or in cooperation with the information community, we have more say in the data management than we imagine – as well as more responsibility.

In the next section, we explore signals of trust in open preprint platforms. In the current ecosystem, authoritative information is still associated with measures of prestige such as the journal impact factor. However, in an open science context with thousands of preprints posted freely online every day, new measures and flags are required. Systems can be designed to reward best practices and ideally preprint platforms would highlight the most scientifically rigorous research. Both the Center for Open Science and ScienceOpen, represented here, along with other preprint platforms are testing and trialing methods to increase the trustworthiness of the research that they host as unvetted preprints. Identifying the qualities, signals, or flags of the most authoritative research will help in designing more responsible platforms for the future and will increase trust in an open science context.

Transparency and rigor are essential to creating an environment of trust. Journal publishers work on many fronts to ensure that the research they publish is authoritative. Each academic field has its own requirements, with peer review at the core. Peer review has come under intense scrutiny over the past few years as a wide range of research has proven not to be reproducible. In the third section, we argue that publishers can respond with increased transparency about the processes and biases embedded in the system. These signals of trust are essential for researchers building on the published literature towards the next breakthrough. And when the system fails, transparency is useful to understand what went wrong and how we can improve the system in the future.

Acknowledging failure is also a key component in trustworthy systems, as we demonstrate in the final case study on retraction. When bad data has entered the knowledge base – whether by accident or fraud, as a preprint or as an article in a prestigious journal – it is essential that it be clearly flagged in a timely fashion. But while this sounds straightforward, our decentralized network of knowledge infrastructure often does not pick up the information that a paper has been retracted, which itself can erode trust in academic research, as we discuss in the fourth section. With this journey from data set to preprint to publication and, hopefully not, retraction we explore the requirements and limitations of current best practices for ensuring that the information we allow to enter the global knowledgebase is authoritative and trustworthy.

2.Design as a tool for producing authoritative information in online crowdsourcing

Within the field of online crowdsourcing, authority has traditionally functioned as a barrier to trust; specifically, the presumed lack of authority around crowd-produced data as a reason not to trust it for authoritative purposes such as research, scholarly publication, etc. Within the past decade a broad range of successes from across research fields have proven that volunteers can produce results that are equal to data generated by experts [3]. Even with this strong record of accuracy, project teams must consider additional qualities for their data, like fidelity (the digital representation of an object along project guidelines, e.g. a diplomatic transcription, which preserves things like spelling errors in service of accurate preservation of the original text) and auditability (information about how the data was produced) [4]. Taking the time to include this information will bring context and history to the data and give potential users the opportunity to audit its origin as part of their vetting process.

This section will provide a perspective on questions of authority within the world of online crowdsourcing, and the responsibilities that practitioners (e.g. platform maintainers, or others who help to create the digital spaces in which this crowdsourcing work takes place) have to ensure that the data being produced is at a quality level that meets the expectations of the project stakeholders, including the volunteers who create the data and any potential users of the results. The case study for this section is Zooniverse [5], the world’s largest platform for crowdsourced research. Zooniverse supports researchers via its free, open-source Project Builder, a browser-based tool that allows anyone to create and run a crowdsourcing project, engage with volunteers, and export results. Since 2009, the Zooniverse has supported hundreds of teams around the world in creating and running projects, and currently has a volunteer community of more than 2.4 million registered participants.

Authority can also be a barrier to participants in online crowdsourcing projects, often in the form of hesitation or low confidence. On Zooniverse project message boards across disciplines, the most common concern from volunteers is, “What if I get it wrong?” It is the responsibility of project teams to help mitigate these feelings of uncertainty by clearly signposting any requirement of prerequisite knowledge, including when no background knowledge is required to participate in a given project.

Design solutions to crowdsourcing’s trust problems include research and development methods for creating tools that produce quality data, project workflow iteration based on user review and testing, and clear, consistent communication with volunteers. Rather than waiting until data is produced and implementing quality control or validation steps on the results, platform maintainers can design tools based on experience as well as through experimentation and testing [6,7].

Projects should be reviewed before launching publicly, both by platform maintainers and volunteers. Zooniverse projects are required to go through a process of internal review by the Zooniverse team, as well as beta review by the Zooniverse volunteer community before they can launch. Teams are required to take this feedback seriously and demonstrate how they edited their project based on the beta feedback they received. This process means project teams put in more effort up front (i.e., before a public launch), but it helps to ensure the usability and reliability of the data being created.

Finally, design solutions can mitigate volunteer hesitancy by supporting broad participation. Crowdsourcing assumes a large, public audience, therefore projects that are designed for holders of specialist knowledge cannot be true “crowdsourcing” projects, as any prerequisites also serve as barriers to participation. Additionally, teams should take care to avoid design choices that, even unintentionally, prioritize the ability of certain demographic groups to participate over others [6]. These considerations can at times be at odds with the desire to design for optimization and data quality, but it is possible to find a balance between the two [8].

3.Open science and signals of trust

Open science provides an opportunity to open up the research process with transparency and openness, and to catalyze an increased trust in science through verification, review, and evaluation of scholarship [9]. To shift research culture towards open practices, we must address the systems scientific research run on through a systems approach: systems of interdependent activity with infrastructure to make it possible and efficient to practice openness in researcher workflows, with communities to make open practices normative, with the incentives driving behaviors to make it rewarding, and with the policies requiring openness and transparency. The goals and incentives should be reconfigured, to start with the end in mind, and incentivizing not publication, but rather the quality and rigor of the research [10].

Scholarly publication has several models that encourage open science, including new models to advance rigor, transparency, and rapid dissemination. Registered Reports emphasize the importance of the research question and the quality of the methodology by approving a publication with peer review conducted prior to data collection [11]. High quality study designs are provisionally accepted for publication if the authors conduct the research study as specified. This aligns the incentive of publication with the goals of best practice and eliminates selective reporting of results, publication bias, and low statistical power [11]. Registered Reports facilitate publication of results regardless of the findings, increasing the dissemination of null findings and further ensuring shared knowledge of research, reducing the duplication of work, and extending the impact of research efforts.

Open Science Badges can be applied to existing and new models for scholarly publishing to offer incentives for researchers to share data, materials, or to preregister. Badges can be added to scholarly outputs to signal verified and persistent open data, open materials, or preregistration. This increased transparency of methods and data can increase the reproducibility and trust in scientific evidence and has shown to increase the rate of data sharing in publications [12]. Badges can offer another tool for aligning the incentives for doing more open and transparent research practices with the goals researchers have for meeting funder and journal requests for sharing their data and outputs.

A preprint is a manuscript describing research data and methods on a public server without having undergone peer review. An increasing number of preprints are being shared across servers, encompassing research from diverse disciplines and geographic locations [13]. Preprints are free to upload and make discoverable on preprint servers, and also free to discover and consume, making them a low/no cost, high reward action for researchers. On preprint servers, the steps between upload and discovery are also minimal, taking hours or days, compared to weeks to months before public consumption in open access article publishing models. Preprint sharing can align the goals of researchers to collaborate and share their work with the incentives to publish or perish because citations are higher for articles that have preprints [14].

Preprints enable rapid dissemination of scholarship; however, that speed comes at the cost of peer review, verification of claims, and evaluation of the rigor of the research process – a risk made more evident with the COVID-19 pandemic. Many preprints were shared as science happened in real time, while the entire globe watched and tried to make sense of the outcomes. Many were looking for signals of trust to know if the preprint could be used to make policy and societal decisions.

In a 2019 study of three thousand seven hundred and fifty-five researchers, open science information was found to be very important for assessing the credibility of preprints [15]. Respondents were asked questions in a survey related to their feelings towards and use of preprints, effect of icons on preprints on credibility, and the effect of service characteristics on credibility. The survey found that it was perceived to be extremely important to have links to materials, study data, preregistrations, and analysis scripts used in conducting the study. It was also found that information about independent reproductions, robustness check, and linked data and resources supporting the preprint would support more trust in the preprint. The disclosure of any conflicts of interests from the preprint authors would further ensure that risks were mitigated through transparency. Each of these signals of trust refer to open science best practices, which could also increase the rigor and reproducibility of the study results, reduce bias and other questionable research practices, and align incentives with the use of badges for practicing openness and transparency.

4.Providing information and building trust

Journal publishers and learned societies use their communications channels, workflows, best practice documents, tools, and platforms as well as collected and analyzed data to ensure the trustworthiness of the information they share. They do so throughout the publication cycle from journal information to their submission and peer review systems and processes as well as the published article page. In a sense scholarly publishing is all about information provided in a trusted manner. This is important because academics’ choice of publication venue (mostly journals) heavily relies on the journal’s audience, author guidelines and requirements, as well as the peer review process the journal applies.

According to a survey run jointly by Sense About Science and Elsevier in 2019, sixty-two percent of two thousand seven hundred and fifteen surveyed academics across the globe trust the majority of research output they encounter but a third doubt the quality of at least half of it. To compensate for this, they check supplementary material/data carefully, read only information associated with peer-reviewed journals or seek corroboration from other trusted sources [16]. Publishers and societies are working together to improve the trustworthiness of their published content through being more transparent about their editorial and peer review process. One example is the STM/NISO peer review terminology [17,18] that has been adopted in a pilot phase by several members.

Many journal home pages provide detailed information about the journal coverage, aims and scope, author guidelines, journal impact, reviewer instructions, peer review and publication speed, etc. Quite recently, many publishers have started to focus on inclusion and diversity to build on the trust by providing editorial member information that is connected to trusted sources such as ORCID, Scopus, PubMed, and Web of Science. Recently Elsevier journals started to indicate the gender and country diversity among the journal editorial board members, benchmarking it against the portfolio of journals covering the topic and more importantly against the gender diversity in that field through the journal home pages [19]. Since data accuracy plays a major role in the success of such initiatives, fifty-two publishing societies and organizations have come together to create a joint commitment for acting on inclusion and diversity in publishing [20]. This collaborative group has launched standardized questions for self-reported diversity data collection and minimum standards for inclusion and diversity for scholarly publishing [21].

Checklists and guidelines are another item journals employ to enhance the trustworthiness of the information they disseminate. For example, declarations of interest help article readers make their own judgments of potential bias: here, the corresponding author must disclose any potential competing or non-financial interests on behalf of all authors of the manuscript. For example, Elsevier’s Declaration Tool [22] quickly and easily helps authors follow the International Committee of Medical Journal Editors guidelines [23] inside Editorial Manager [24]. Journals look for ways to ensure that materials and methods information is properly provided by authors mostly through requiring them to use reporting guidelines such as STAR Methods [25], RRIDs [26], CONSORT [27], PRISMA [28], and SAGER [29]. This addresses the problem that a major cause of reproducibility problems is presentation of incorrect, incomplete, or even unavailable data and information.

Ensuring trustworthiness of the peer review process is another important focus for journal publishers and scholarly service providers. Tackling bias in peer review sits high on their list. We know that many of our societal structures are biased and often inequitable because humans are biased, and we built these systems. So, any data that attempts to describe these human-built systems or that arises from them will also reflect those biases and scholarly publishing is no different – the publishers’ job is to understand when that bias leads to unintentional harm.

Elsevier uses Scopus data to improve the relevancy of search engine recommendations to help editors to discover the most relevant potential reviewers. But the scholarly system was not designed for women or the minorities who were historically excluded from it. “Publish or perish” is a known pressure for those pursuing an academic career and clearly life events like parental career breaks or working part-time to juggle care commitments do not boost a researcher’s publication output. With the legacy of myths and biases such as that of gendered brains, it is little wonder that the characteristics of published researchers in our scholarly data do not reflect our global demographics.

To mitigate inherited bias, Elsevier worked on a project to embed fairness in our reviewer recommendation pipeline which needed engagement from every function of the business. Whilst our data scientists did a great deal of important work creating fairness metrics and benchmarking model performance across demographic groups, our efforts did not stop there. We made changes to the user interface to give more clarity to editors on the purpose and limitations of the tool. We also engaged with editors to support them in diversifying their reviewer pool.

We know biases exist (and have existed for many years) in the research ecosystem and naturally much of the data built on researcher performance reflects this. So not only do we have a responsibility to create awareness and transparency around what data our algorithms use and how, but there is a limit to how much we can “correct” this with just clever modeling of the biased data itself. We must ensure robust human oversight, governance, and appropriate accountability for these systems.

Another example to ensure research integrity is to provide clear, and harmonized retraction notices not only on the article pages, but also upon submission of a manuscript to authors who cite retracted articles in their reference list. Elsevier recently introduced a reference analysis report, which identifies citations in a submitted manuscript, lists them, and provides links to Scopus, Crossref, and PubMed if applicable. It clearly identifies whether any of the cited papers are retracted. Authors, editors, and reviewers can clearly see whether a retracted paper is cited in a manuscript and inquire whether it was necessary.

5.Safeguarding the integrity of the literature: The role of retraction

Prompt retraction is intended to minimize citation [30]. However, inappropriate, continued citation of retracted papers is common. For example, two COVID-19 articles that were retracted in June 2020 within a month after they were published have over twelve hundred citations each as of June 2022. Science magazine examined two hundred of the post-retraction citations to these papers and concluded that over half inappropriately cited the retracted articles [31]. Although the publisher websites show these articles as retracted, in some online systems, such as Google Scholar, information about the retraction status is hidden by default; to be more likely to see retracted items, users must click “Showing the best result for this search. See all results”.

Scientometrics has analyzed citations to retracted papers since at least 1990 [32–34]. Ongoing citation remains a problem, even when the retraction fundamentally invalidates the paper. After a retraction for falsified data, a human trial article continued to be positively cited eleven years after its October 2008 retraction. Of its post-retraction citations, in the years 2009–2019, ninety-six percent inappropriately cited the retracted article [35]. Finding the retraction notice was difficult. Among the eight databases tested, only one, EMBASE, had a working link to the retraction notice [35]. Such continued citation is not rare, according to a study in biomedicine, which concluded that ninety-four percent of post-retraction citations were inappropriate, considering a database-wide corpus of thirteen thousand post-retraction citations from PubMed Central articles citing retracted papers in PubMed [36].

The Alfred P. Sloan Foundation Reducing the Inadvertent Spread of Retracted Science project [37] conducted a stakeholder consultation and related activities, resulting in four recommendations [39,40]:

As a result of this project, addressing the first recommendation, NISO has chartered a working group to develop the Communication of Retractions, Removals, and Expressions of Concern Recommended Practice [41,42]. Its goal is to address the dissemination of retraction information (metadata & display) to support a consistent, timely transmission of that information to the reader (machine or human), directly or through citing publications, addressing requirements both of the retracted publication and of the retraction notice or expression of concern. The working group will not address the questions of what a retraction is, why an object is retracted, nor define terminology for retractions. The working group's expected outcomes include the following: additions to existing metadata to support awareness of retractions; best practices on population of retraction notices; outlining responsibilities of creators and consumers of retraction information; consistent of display labels, user experience, signaling; and a workflow definition for circulating information.

6.Conclusions

In recent years, many organizations, publishers, and learned societies have critically improved their workflows, products, services, and communication channels to ensure the trustworthiness of the content they provide. Yet these numerous activities are still scattered and siloed, resulting in opportunities for distrust across all phases of research. Lack of shared information around design choices leads to perpetuation of missteps, paywalled research continues to limit access to information, opaqueness around peer review can lead to bias in publication, and retracted articles still get cited for wrong reasons. In this paper, we have outlined practices the information community can take to mitigate these ongoing issues. It is our hope that by sharing these practices across the various sectors of our community, we can learn from one another and continue to raise both the standard of information, and, consequently, its trustworthiness.