BITNIS — Bridging the information gap before the web

1.Introduction

Donald A.B. Lindberg M.D., the U.S. National Library of Medicine (NLM) Director and health informatics pioneer, understood that the success of NLM’s efforts depended not only on their value to the nation, but also to the rest of the world. Before the Internet arrived at the NLM, remote access to its databases from outside the U.S. was mostly reserved for a privileged elite. Dr. Lindberg’s vision and unrelenting pioneering spirit changed that forever.

Through its long history, the NLM has evolved constantly to offer more and better services and products to a growing audience. To that end, Dr. Lindberg led the development of many innovations that revolutionized health informatics. This is the story of one of those innovations, which, although not well known, had a significant impact internationally.

2.Information access, a coveted privilege

In the mid-1980s, Andrés Stutzin invested a few years as a visiting fellow at the National Institutes of Health (NIH). During his temporary research appointment as a visiting fellow with the NIH National Institute of Diabetes, Digestive and Kidney Diseases (NIDDK), he and his colleagues benefited from the NLM’s authoritative bibliographical information almost daily. At his lab, in the heart of the NIH Bethesda campus, his access to the NLM databases laid just steps away at his personal computer. He could also walk to the NIH Library or the NLM itself when he wanted assistance formulating a literature search strategy or submerge himself in the scientific literature. Usually, full-text articles arrived at his lab in government-issue manila envelopes not long after requesting them from one of the libraries. As a research biologist, he marveled at the enabling power of NLM resources for his scientific work. As a medical doctor, he understood the immense value of NLM information to support the core goals of medicine.

In 1987, after resuming his academic and research work in Chile, Stutzin had to face the stark reality of the “information gap” that plagued developing countries and other disadvantaged world regions. The Medical Library at his workplace in the Faculty of Medicine of the University of Chile had a subscription to MEDLINE on CD-ROM, but the information was relatively stale and did not cover the full MEDLINE content. Remote access to MEDLINE from the NLM and other commercial venues was available, but budgetary and other practical constraints made that access hard to use. Journal subscriptions were limited and accessing full-text literature often took significantly more time and effort. At the time, this scenario was not uncommon for international fellows returning home after holding temporary positions at the NIH.

At his institution, Stutzin was a member of the Medical Informatics Commission (CIM, for its name in Spanish), a faculty-level advisory group focused on enhancing information technology-based resources. Among his contributions in this role, Stutzin helped modernize and grow the campus-wide computer network infrastructure and helped coordinate its interconnection with a larger academic network that spanned several local universities. Seeing the power of networks to boost collaboration and host remote information services, he teamed up with Víctor Cid, a CIM advisor, to investigate options for enabling access to NLM databases through this infrastructure. At the time, Cid was with the Faculty of Physical and Mathematical Sciences of the University of Chile and led the engineering team responsible for the national academic network. Their research turned into a quest to make remote access to NLM information resources more available and affordable to support research, education and medical practice in Chile, and beyond.

3.NLM options and realities

In 1987, the world was experiencing a rapid growth of “personal computing”. The trend originated in the late 70s and contributed to the development of many isolated, independent networks of interconnected computers around the globe. Many of them grew in a “collaborative” fashion. The Internet was young, and, although famous, it was still just one of the many computer networks in existence. The World Wide Web was still a few years in the future, but services like electronic mail had already become popular within and between many computer networks, and via independent commercial venues [1]. It was also a time of “big iron”, room-size mainframe computers that large organizations used to conduct business and provide services to multiple users in a centralized manner [2].

At the time, the mainframe computers that hosted the database services of the NLM could be accessed by thousands of remote users through dozens of telephone lines and several commercial packet network service providers. However, these host computers were not connected to computer networks. Dr. Lindberg was never shy about trying new technologies, but to be adopted, new technologies had to demonstrate their usefulness to fulfill a need in the context of the NLM’s mission. The NLM operated on the principle that “technological innovation must be an aid to improved medical research, education and practice” [3].

The Medical Literature Analysis and Retrieval System (MEDLARS) of the NLM had for long been considered a leading bibliographical information resource for biomedical researchers and practitioners worldwide. MEDLARS data was accessible in different ways, but interactive access to MEDLINE (MEDLARS online) was the preferred access method for users needing timely access to the most up-to-date and complete data. International users who wanted to conduct their own database searches also had a few options: access directly from computer terminals connected to mainframe computers that held copies of the NLM information resources at a few international locations; remotely through a computer program called Grateful Med (GM); or via CD-ROM subscriptions. Remote access without GM was also possible, but GM was by far the preferred method (Fig. 1) [4,5].

Fig. 1.

MEDLINE search design screens of Grateful Med for DOS version 4.0, circa 1988 (picture courtesy of the Library of Congress), and GM for Windows version 1.0, circa 1997 (picture courtesy of the NLM).

Remote access to MEDLINE with GM could be achieved a few ways, and generally involved a personal computer, modem and telephone line.¹ Users could place an international phone call with their computers to the NLM, a local or long-distance call to the closest International MEDLARS Center (IMC), or a call to a local commercial packet network service (e.g., TELENET or TYMNET) and use that service in turn to connect to the NLM [6]. In practice, these options were feasible only to international users who were fortunate enough to be close to a MEDLARS Center that hosted local copies of the NLM’s databases or could afford the expense of long-distance or international phone calls and/or the packet network service. Telephone services in most locations were metered, and packet networks typically billed by time and data volume transferred. Also, in many international locales telephone service was not capable or reliable enough to support data communications.

But those were not the only barriers. Accessing the NLM databases was not free, therefore users needed to obtain a MEDLARS user account from the NLM or a ICM and pay a metered on-line access cost. The NLM used the proceeds mainly to offset the telecommunication costs required to provide the service and pay data suppliers. For instance, a remote MEDLINE search session from outside the U.S. could cost a user US$25 or more in 1990 dollars, mostly due to telecommunications costs [7]. As a result of these costs and technological barriers, remote access was often prohibitive for many remote users, especially, although not only, in the developing world.

CD-ROM subscriptions were popular at the time, although they only provided access to a subset of the NLM content, and the currency of the data was necessarily a few months behind. Unfortunately, CD-ROM subscriptions, which were available from a few commercial vendors, sometimes were also unaffordable for individuals and small organizations in developing countries. Subscription costs were in the order of $2500 per year, plus shipping costs, and required equipment that at the time was relatively expensive to obtain and maintain [8]. Also, shipping CD-ROMS to some remote locales at that time was harder than it is today.

4.Bringing NLM closer to international users

In 1966, the NLM started establishing collaborative arrangements with foreign organizations to respond to requests for using the NLM databases from users outside the United States. Through quid-pro-quo/Memorandum of Understanding (MoU) agreements with national entities, the NLM supported the creation of several IMCs, which numbered 17 in 1987 and reached 20 in 1997, in as many countries. The NLM provided the IMCs copies of MEDLINE and other resources and required software, usually on magnetic tapes, technical documentation, and training. In turn, the IMCs provided bibliographical services regionally, service evaluation and technical feedback, and sometimes other services for the NLM, such as indexing international journal articles. The MEDLARS Centers greatly enhanced the reach of the NLM products and services to the international community and helped the NLM enhance its products and services [3].

Since the early 80s, an IMC was available at the Pan American Health Organization’s (PAHO’s) Regional Medical Library for Latin America and the Caribbean in Sao Paulo, Brazil (BIREME, for its Spanish acronym).² BIREME is located at the heart of one of the main Brazilian universities and plays an important role satisfying the biomedical information needs of that university and other health organizations and professionals throughout Latin America and the Caribbean. This library also contributes valuable technical and other innovations for the development, management, and provision of biomedical information services in a manner that is adequate for the cultural, social, and economic realities in the served countries. BIREME also provides inter-library loan services in the region [9].

However, in the 80’s, accessing MEDLINE at BIREME remotely from many places in Latin America was often as difficult as accessing it directly from the NLM. Metered and low-quality telephone services in many locales, and the scarce availability of packet data networks and dedicated telecommunications links to BIREME, made accessing this resource challenging and often unaffordable. There were also language barriers and service support challenges. BIREME’s region of service includes 51 countries and territories and five different official languages. BIREME also had to balance its priorities of serving the needs of its parent organization and others in Brazil, and users throughout the region. BIREME is an outstanding organization, but some of its users felt that its broad responsibilities sometimes seemed to require more resources than BIREME had available.

MEDLINE on CD-ROM made great strides towards satisfying information needs for many international users, but it was necessarily an imperfect substitute for direct access to the NLM databases.

5.The gift of international collaboration

PAHO and the NLM had a MoU that enabled them to collaborate in a variety of information programs since the late 70s [10]. In 1997, Dr. Lindberg learned through PAHO about the effort initiated by researchers of the University of Chile to explore the feasibility of accessing the NLM resources through international computer networks.

The NLM Director was very familiar with the enduring information gap issue, which seemed to have worsened over the years due to the technological disparity between developed and developing nations. For a long time, he worked towards positioning the NLM as a strategic resource for addressing issues such as health inequality, global health and maximizing the productivity of biomedical sciences, all of which required access to and sharing of reliable and updated scientific information at a global scale. However, reducing the information access barriers internationally and extending the NLM’s reach beyond what was possible through IMCs and CD-ROMs was a monumental challenge.

Over time, the Chilean researchers engaged PAHO and proposed to the NLM a solution that involved NASA, IBM Corporation, the University of Maryland at College Park, the NIDDK, the Division of Computer Research and Technology (DCRT) of NIH, and the University of Chile. Dr. Lindberg appreciated the merits of the proposal, but the Chilean experiment came with non-trivial risks. In addition to the multi-institutional coordination challenges, the plan involved interfacing the NLM computers with technology that was foreign to that in use at the NLM and possibly experimenting with unproven ways to recover service costs from database users, among others.

However, Dr. Lindberg could see through and beyond those risks and focused on the opportunities that the proposed remote access mechanism offered to further extend the reach of the NLM services globally. He instructed the NLM Director of International Programs to support the Chilean experiment, in the hope that it could inform future efforts.

6.The power of email

Many computer networks spotting the globe in the 80s were mutually incompatible. Within each network there was great creativity and constant innovation, but their heterogeneity made interconnecting them challenging due to competing and incompatible network protocols, architectures, services, and equipment, among others. However, all computer networks offered some kind of asynchronous messaging capability that enabled users to exchange electronic correspondence within a network and often between them.

The features and capabilities of the email services on those networks varied widely, but at minimum it allowed its users to send each other text communications in a reliable way. In most cases, email was resilient to poor communications infrastructure due to its “store-and-forward” design. Email messages usually hopped from one network node to the next on the path to their destination, where they were stored and then forwarded to the following node along the way. If an intermediary network connection was interrupted or performing poorly, the messages were automatically retransmitted by the preceding node until a successful transmission was completed. Due to varied quality and features of data links, which often involved regular commuted telephone lines, an email could take seconds to days to arrive to its destination, but the network protocols guaranteed the integrity of the message on delivery, unless the transmission was impossible after some retransmission limit. Email has evolved over the years, but the essentials remain the same [11].

In the late 80s, the NLM and PAHO were already very familiar with the advantages of electronic mail, which was available to them from government and commercial providers. Naturally, email was already being used in some cases to provide limited human-assisted database searching services to distant users. However, the University of Chile researchers envisioned a service that was automatic, could handle a large number of service requests, and that was easy to use and affordable to its users. Regardless, the hardest problem was the huge telecommunications gap to Chile.

7.Bridging the gap

The solution came from a fortunate convergence of resources and relationships. At the time, the University of Chile managed several national computer networks, and had developed an expertise on interconnecting dissimilar computer and information systems. The main network was BITNET (Because It’s Time Network), which linked IBM mainframe computers from several universities [12]. However, the BITNET network in Chile was not interconnected with its equivalent in the United States, as the cost of a high-speed, dedicated telecommunications link between Chile and BITNET was at the time economically unfeasible.

The University of Chile had a good collaborative relationship with NASA, which had an installation close to the capital city of Santiago. They learned about the interest in interconnecting University of Chile networks with others in the US and offered access to a dedicated satellite link from Santiago to Goddard Space Flight Center in Greenbelt, MD, a few miles away from the NIH campus. The link was used for NASA operations part of the day and was sufficiently fast for email communications and more. Through a collaborative agreement, NASA enabled the University of Chile to use the link a few hours per day for this experimental project. This solved the problem of reaching the United States, but an additional effort was required to reach the NLM.

Meanwhile, researchers from NIDDK on the NIH campus maintained a close relationship with their counterparts at the University of Chile and viewed this link as an opportunity to enhance their research collaborations. The NIDDK researchers sponsored the link and engaged engineers from the Computer and Space Sciences Department of the University of Maryland (UMD) at College Park to coordinate the implementation of the data link to the Department of Computer Research and Technology (DCRT) at the heart of the main NIH campus in Bethesda. At the time, DCRT was already connected to BITNET and therefore the link enabled interconnecting Chile to BITNET in the U.S. A modem connection via a regular telephone line to DCRT completed the electronic bridge all the way from the NLM mainframe computer room all the way to the University of Chile.

To complete the solution, the Chilean researchers had to develop customized technology to enable the remote user access to the NLM databases. Given that the telecommunications solution only provided intermittent and relatively slow connectivity, the researchers designed a system that enabled processing MEDLARS database queries in “batch” form or off-line. Electronic mail was the natural choice as the transport mechanism to receive database requests from users and send results back, as email did not require permanent connectivity or high bandwidth and could handle the data volume of individual transactions.

8.BITNIS is born

The Stutzin–Cid team implemented a gateway system that interconnected BITNET and the main NLM database server for the first time in 1988. The system emulated a librarian automaton that received emails with database search instructions from BITNET users, logged in to the server and conducted the requested search queries on the required database, captured the search results, optimized them, and sent them back to the requester via BITNET email (Fig. 2).

Fig. 2.

BITNIS functionality: (1) a remote user prepared a search with GM and sent it to BITNET in route to the NLM; (2) the email arrived to BITNET at NIH via the NASA link and the BITNIS gateway downloaded it from the NIH BITNET node; (3) the gateway conducted the requested database search, captured the results and set them back to the requester via BITNET email; (4) the remote user received the response email and could view the search results with GM. The gateway evolved to allow any users using computer networks interconnected with BITNET to conduct database queries, provided they had a MEDLARS user account registered on BITNIS.

This automatic gateway, called BITNIS (the acronym’s meaning evolved over time, but it originally meant “BITNET to NLM Intercommunication System”), also had access control and accounting features that enabled billing users to pay the NLM for database online time. The gateway interacted with the NLM database server faster than a human could over a dial-up line, minimizing the on-line time expense for its users. There was no fee for using BITNIS itself.

Cid also developed “middleware” software to enable users creating a MEDLINE query with Grateful Med in their personal computers, and then embed the query in an email that was then sent to BITNIS for processing. The query language used by the NLM database system was complex and GM greatly simplified formulating searches via its interactive user interface, hence reducing the chances of errors. After BITNIS had replied with search results, the “middleware” also enabled users to view the results with Grateful Med and refine the search if necessary [13–15].

Soon after inception, BITNIS started to process dozens of database queries per day from University of Chile’s faculty, students, researchers, and librarians, in addition to clinicians from its university hospital.

9.BITNIS expansion in Latin America

The experimental gateway proved to be effective. Users from the University of Chile could use MEDLINE without incurring telecommunications costs, enabling many more of them to access the NLM information. The University maintained a deposit account with PAHO, which in turn paid NTIS for the use of the NLM databases in the United States. Very soon, the University started to receive inquiries from other universities and research institutions along the country, and pressure started to build up to expand the service.

Given the success of the Chilean experiment, Dr. Lindberg saw the opportunity to use BITNIS for extending the reach of the NLM services to other underserved communities around the world. With support from PAHO and the University of Chile, the NLM extended the service to any Latin American country that could access international academic networks and the BITNIS gateway via email. Initially, the service operated with a single MEDLARS user account, but it was expanded to enable multiple accounts. Also, thanks to hardware donated by IBM corporation, the gateway was upgraded to support higher transaction rates and shorter service turn-around times. PAHO made special arrangements with the NLM and NTIS³ so PAHO could administer MEDLARS user accounts for BITNIS access, and sponsor or charge users and pay NTIS for their use of the NLM databases.

First the service was expanded to other Chilean universities that had networks connected to the University of Chile. Meanwhile, the academic network continued to grow, which enabled adding more users from other universities and health organizations. Soon, networks in Argentina started to join in, some via their own connections to BITNET or other networks interconnected with BITNET in the U.S., others through connections to Chilean networks [16].

The value of the information access pathway that BITNIS opened had unexpected benefits: it provided an incentive for some national networks in South America to interconnect with each other and join international computer networks. In Chile, the National Commission for Scientific and Technological Research (CONICYT, analogous to the National Science Foundation in the U.S.) financed additional network infrastructure and soon the project did not have to rely on the NASA link anymore. The project results also encouraged PAHO to join BITNET and allocate resources for enhancing computer-based communications and collaborations for medical education in the region [13].

Additionally, the BITNIS expansion in Latin America provided an incentive to develop a collaborative user network. PAHO, the NLM, and CONICYT supported a BITNIS Regional Workshop in Santiago, Chile in May 1993. The event was attended by more than 60 BITNIS users, information technologists and other specialists from universities and other national entities from 10 countries. The meeting enabled the participants to share experiences, get training on searching MEDLINE and other NLM databases, and learn about other information services, technologies, and project opportunities. The meeting also provided the participants important networking opportunities that opened the door to other collaborations.

10.Global expansion

In 1991, Cid joined the NLM and led the transition of the BITNIS gateway from experimental to a production service [17]. The gateway was upgraded from an IBM PC to a faster Sun Spark workstation with a faster network connection. Meanwhile, a collaboration with the National Cancer Institute (NCI) enabled remote access via BITNIS to PDQ (Physician Data Query), a cancer information resource also hosted at the NLM. The NCI maintained a network of collaborating health researchers and practitioners internationally, many of them in developing countries.

Interestingly, some health professionals in developed countries also saw advantages to accessing the NLM databases directly through computer networks and BITNIS. Soon, BITNIS started to process database queries for users in Germany, Switzerland, and Portugal.

After the fall of the Iron Curtain in 1990, countries that were part of the former Soviet Union faced hard challenges in their journey towards joining the rest of the world as independent states. In 1992, the Department of State (DoS) sought the NLM’s help bringing access to authoritative health information to those countries as part of a strategic effort to establish friendly relationships with the new nations and provide humanitarian support. Dr. Lindberg saw BITNIS as the perfect tool to support this effort. The NLM collaborated with the DoS by devoting NLM resources to enable the access to MEDLARS from Confederation of Independent States (CIS) countries through BITNIS [18]. Joint teams from the NLM and DoS were deployed to CIS countries to provide connectivity, equipment and training to health and other national libraries in countries in Eastern Europe and Central Asia. Most libraries had remarkable library assets and experienced librarians, but poor or not functional telephone lines and information technology. Through this initiative, many national libraries in CIS countries were equipped with personal computers and other IT equipment and gained access to international computer networks and NLM information resources via BITNIS.

Around the same time, a non-for-profit organization called SatelLife had developed a communications service based on low earth orbit satellites to improve health communications in developing countries [19]. The initiative, called HealthNet, started to be used to provide access to the NLM information through BITNIS from Africa, parts of Asia, and the Middle East. With support from the World Health Organization, users in Uganda, Zimbabwe, Malawi, Turkey, and other locations gained access to the service using small ground antennas pointed to the sky.

By February 1997, BITNIS served users in 56 countries (Table 1). That year, the number of user accounts in the system had grown to nearly 400. It is hard to estimate the actual number of end users benefitted by the service, as often accounts were shared among groups of users in their localities. For instance, a single account could be used by 20 or more users in a library setting. In 1996, at the peak of the service, the BITNIS gateway processed about 400 search requests per day.

11.The BITNIS legacy

Starting in 1996, users around the world could finally access MEDLINE through the young World Wide Web. The MEDLINE content was now also available in PubMed, an internet-accessible service developed and maintained by the NLM National Center for Biotechnology Information (NCBI). However, the internet remained elusive to underserved communities in many countries, and therefore BITNIS continued to play a role for some time. NLM gradually started to lift the access cost for some of its MEDLARS databases. By 1994, access to AIDSLINE, AIDSDRUGS, AIDSTRIALS, DIRLINE became free in support of the response to the HIV/AIDS pandemic. On June 26, 1997, in a Capitol Hill ceremony featuring Vice-President Al Gore, the NLM announced that MEDLINE became available to the world free of charge on the Internet through PubMed [20]. By then, the Internet was more widespread globally and the BITNIS gateway started to see a marked traffic decline. At that point BITNIS had completed its mission and the service was discontinued. Shortly after, the legacy MEDLINE system was also discontinued. It was the end of an age in the history of the NLM, and the beginning of a bright new one.

The benefits of BITNIS extended beyond connecting underserved communities to quality biomedical information. The implementation of the service in multiple countries not only allowed interconnecting people with the NLM, it promoted technological innovation by providing incentives for network development and inter-networking, hence helping speedup the inclusion of many in the Internet revolution. The project also created new collaborations between the participants which had many other benefits. In Latin America, for example, BITNIS offered a way to create a collaborative network of libraries that supported their institutions through a variety of projects, including, for example, activities that led to the creation of a Latin American Network of Disaster Information, another accomplishment of Dr. Lindberg in the years that followed BITNIS [21]. Within NLM, BITNIS strengthened the interest in international computer networks and the Internet to reach a larger audience, create innovative services and enhance collaboration.

Dr. Lindberg believed that the best way to propel the NLM into the future was to reach to users and look at their needs [22]. BITNIS was a great example of his leadership philosophy. Developing BITNIS required a combination of ingenuity, technical skills, the right players, perseverance, and a leadership with vision and an open mind. BITNIS helped the NLM Director bridge the information gap for many people in developing countries and beyond before the Internet revolution. Through BITNIS, Dr. Lindberg supported the nation’s efforts to spread good will around the world and brought life-saving biomedical information to many underserved communities at a time when it was most needed.