Are we making a difference? Outreach evaluation in practice

1.1.The critical incident technique study of MEDLINE users

Dr. Lindberg suggested using the Critical Incident Technique (CIT) evaluation methodology to identify answers to some of these questions [2]. First used in World War II to identify characteristics and behaviors of successful bomber pilots, the CIT became widely used after the War by psychologists in personnel selection and in a variety of other settings; several in the medical community but none in the library community. Dr. Lindberg tasked Elliot R. Siegel Ph.D., then Assistant Director for Planning and Evaluation, to adapt the methodology for NLM’s use.

Siegel’s CIT study was designed to gather detailed reports of MEDLINE searches that were especially helpful (or not helpful) in carrying out the full range of professional activities of more than 550 physicians, scientists, and other professionals working in a variety of clinical care and other settings. Of these, two-thirds were direct users of MEDLINE who were purposefully recruited for the study; the remaining were academic health sciences and hospital librarians who recounted searches performed by them for others. More than eleven hundred CIT reports were systematically content analyzed from three different perspectives: why the information was sought, the effect of having (or not having) the needed information on professional decisions and actions, and the outcome of the search.

In clinical settings, the study documented that MEDLINE searches were being carried out by and for physicians to meet a wide diversity of clinical information needs. Rapid access to the biomedical literature via MEDLINE, for example, was critical to sound patient care and favorably influenced patient outcomes [3]. All interviews were conducted by telephone and typically lasted 30 minutes. The protocol benefitted greatly from Dr. Lindberg’s insistence that the first 50 calls be pilot tested and conducted personally by Siegel, rather than by the interview-skilled but non-medically experienced contractor’s staff, to sharpen and refine the interview questions. A valuable hands-on lesson that benefitted Siegel’s evaluation work in the years ahead.

1.2.Evaluation at NLM pre-Lindberg–a promising beginning

In the early 1980s, the traditional “wood” book catalog was replaced, and a microform system was installed in its place in libraries without the benefit of evaluation. It proved to be unpopular with both patrons and staff. Then NLM Director, Martin M. Cummings M.D., tasked Siegel with performing a comparative evaluation of two home-grown prototype online catalog systems in the NLM Reading Room, much to the consternation of some staff who were uncomfortable with the perceived intrusion by the front office and the prospect of discomforting patrons. The patrons never complained and seemed intrigued by the new computer monitors and the opportunity to help test a leading-edge technology that could make searching the collection far easier and more effective. A clear favorite amongst the two electronic search systems was identified. Its demonstrated reliability and superior functionalities proved superior to the other system, and of course to the static card catalog it permanently replaced [4].

Dr. Cummings was the inspiration for another notable early evaluation when he returned from an annual Congressional Appropriations hearing greatly frustrated that once again, he was unable to answer a simple question with any precision — Who uses MEDLINE? Cummings approached Siegel who created what turned out to be NLM’s first MEDLINE user study, an early precursor of the CIT study.

It was implemented in a far less rigorous manner; self-selected anonymous MEDLINE users responded to a brief two question online survey that identified their professional (or public) role and the intended purpose of their search. Staff resistance to the possibility of burdening searchers was overcome with the realization they would now have greater insight into the actual identity and needs of their users, especially those for whom the librarian-conducted searches were being performed. The study findings provided Dr. Cummings with a credible answer for the next hearing, much to his delight.

Investigators at NLM and at other institutions later undertook descriptive MEDLINE user studies of their own [5]. Many more would follow the pathfinding CIT study, benefitting from new analytic tools, technologies, and policies that evolved, and are recounted later in this chapter. An early example was a nationwide field evaluation of MEDLINE products on CD-ROM that was organized by NLM under Dr. Lindberg’s careful watch. He viewed the CD-ROM as a transitional technology soon to be replaced by Internet access. Nevertheless, the commercial products existed, and the Library needed to assure itself and the community of users that their technical performance and user acceptance met acceptable standards [6].

1.3.Dr. Lindberg’s influence – A mandate and a way forward

It was Don Lindberg and the CIT study he initiated and personally guided that would lead to the establishment of a robust tradition of evaluation research at NLM. Dr. Lindberg made evaluation an accepted and a necessary requirement to study our users and ask the hard questions of them and of ourselves as program managers — Are we making a difference? He expanded Siegel’s role to include responsibility for coordinating and guiding NLM’s outreach activities and evaluation research throughout NLM and, indeed, introducing new evaluation tools and methodologies to the entire National Institutes of Health (NIH) community as detailed in this chapter. Siegel was joined in 1995 by Frederick B. Wood, MBA, DBA who collaborated closely as lead evaluator in the newly established Office of Health Information Programs Development (OHIPD), within the Office of the NLM Director. Together, Siegel and Wood developed and coordinated new outreach initiatives that sought to include a strong evaluation component.

2.1.Outreach evaluation: The first five years

With the availability of newly appropriated outreach funding in the five years between 1989 and 1994, NLM undertook and supported close to 300 outreach projects involving more than 500 institutions across the country. The key to these efforts was the close collaboration of the National Network of Libraries of Medicine (NN/LM).

In 1995, the OHIPD coordinated a comprehensive evaluation of these outreach activities [11]. The review found very substantial progress in engaging and training individual health professionals to use MEDLINE via the Library’s end-user-friendly interface, GRATEFUL MED. The number of user codes issued increased from approximately 30,000 to almost 100,000 during that time period. There were four million searches of NLM databases in 1989 vs. seven million in 1994. GRATEFUL MED users in 1989 accounted for less than one-third of the searches, in 1994 they conducted more than two-thirds of the searches.

These metrics were certainly very useful indices of impact, but with few exceptions these outreach projects did not get at the fundamental underlying question posed by Lindberg — Are We Making a Difference? — nor did they address the need for intensified outreach to minority populations, and the institutions serving their health information needs at the community level.

A significant part of the identified outreach evaluation challenge was that the library staff undertaking the outreach had little or no formal training in evaluation. They had limited ability to plan an evaluation project, and to articulate answerable questions that surpassed “counting heads”. Nor was there available an appropriate training resource that could fit their current skill set and bring them to the next level.

2.2.Measuring the difference: Guide to planning and evaluation health information outreach

We published an Outreach Evaluation Guide in 2000 as a capacity-building effort. It was a partnership of the NLM and the Pacific Northwest Regional Medical Library (PNRML) at the University of Washington. The lead author was Catherine Burroughs, MLS, at the PNRML, and the project officer was Dr. Wood. The project included an advisory group of community outreach and evaluation experts, several of whom prepared background papers. For an extensive list of advisors and staff contributing to this landmark effort, see the guidebook acknowledgements [12].

The underlying themes of the Guide were: planning and evaluating an outreach initiative is one and the same process; and asking the right questions at the beginning was essential to obtain useful results. Moreover, the Guide was practical in purpose, theory-based, and offered a range of methodological possibilities and strategies that could be adapted to the simplest or most complex of outreach projects. The NN/LM also established an Outreach Evaluation Resource Center (OERC) at the PNRML that could serve in a consulting capacity to member libraries using the Guide and needing hands-on outreach evaluation and related assistance.

As a follow-on to the original manual, the OERC issued a set of three shorter guidebooks (in booklet form) for key stages of outreach evaluation, initially released in 2006 and updated in 2013 [13]: The principal authors of the series were Cynthia Olney, Ph.D. and Susan Barnes, MLS, MS, both at the OERC. The booklets were intended to help address the need of NNLM outreach staff for stronger evaluation skills, with specific emphasis on the Logic Model methods of evaluation that provides a straightforward relationship between project goals, process, and outcome measures. For further discussion of Logic Model applications, see the companion chapters on the NLM’s Library Network: A Face of Outreach [10] and HIV/AIDS Community Information Outreach Program [14].

For further background on evaluation of community-based outreach projects, see the proceedings of the 2004 symposium on this topic co-sponsored by the NLM and NNLM, and especially the overview article, and an article by Dr. Charles Freidman on right sizing outreach evaluation and the use of smaller and incremental evaluations where appropriate [15–17].

2.3.Transitioning to the internet

The overall framework for outreach evaluation from the mid-to late 1990s onward increasingly reflected the emerging Internet platform for biomedical and health information dissemination. Dr. Lindberg was an early leader in the arena of health on the Internet. He developed a long-term interest in medical informatics and served as the first director of the Federal Government’s Office of High-Performance Computing and Communications (HPCC). For several years, Dr. Lindberg wore two hats, NLM Director, and HPCC Director.

Dr. Lindberg was well positioned to discern and foresee the coming Internet revolution and its profound implications for NLM. The Internet would transform how NLM conducted its core information dissemination activities and its outreach around those activities. And likewise, NLM’s evaluation activities needed to evolve and adapt to the Internet.

Under Dr. Lindberg’s leadership, NLM needed to address such topics as:

Thus, the need for new or revised evaluation approaches that would work in the increasingly electronic customer service and outreach environment.

3.1.Tribal connections

The five-year review of NLM outreach concluded that NLM outreach to Native Americans was too limited. In response, NLM reached out to Sherrilynne Fuller, Ph.D., director of PNRML and a committed advocate for Native American outreach. Tribal Connections became a long-running partnership between NLM and the PNRML and was one of NLM’s first major outreach projects with a Native American focus.

Tribal Connections was intended to use a community-based outreach approach to improve the Information Technology (IT) infrastructure and Internet access at select Indian tribes and Native villages and provide training in online access to NLM health information. Tribal Connections was designed to run in parallel with the aforementioned project to develop an outreach evaluation Guide. The intent was that Tribal Connections could benefit from and contribute to the evaluation methods project.

Tribal Connections Phase 1 reached 16 Indian tribes and Native villages in the Pacific Northwest (Washington, Alaska, Montana, Idaho, and Oregon). Each site was intended to have community outreach with one or more site visits, collaborative needs assessment, customized technical support, purchase of necessary IT equipment and services, Internet connectivity upgrades or partnerships as needed, and associated technical and other training. At the time of this project, Internet connectivity at the participating sites ranged from 56 Kbps dial-up, 128 Kbps ISDN, to full T-1. Each site plan was optimized for the baseline Internet connectivity and realistic upgrade options. For detailed discussion of Phase 1, see [18,19].

Later phases of Tribal Connections included the addition of four sites in the Southwest, more intensive training and outreach at selected sites, and a collaboration with medical libraries serving American Indians in the Four Corners Area (parts of Arizona, New Mexico, Utah, and Colorado) [20,21]. The latter project was known as Tribal Connections Four Corners (TC4C).

From an evaluation perspective, the projects early on relied heavily on performing needs assessments with local tribal leaders and key staff, whose engagement and understanding of the technical infrastructure improvements proved to be the single most important factor in determining a successful outcome. Internet connectivity was successfully established at nearly all sites, and training in its use was assessed which identified areas needing improvement. Implementations in the later project phases added measures to better understand the impact on information seeking behavior and actual health-related decision-making.

Dr. Lindberg requested periodic Tribal Connections briefings, several of which occurred at meetings of the NLM Board of Regents. Although not foreseen at the time, Tribal Connections helped lay the groundwork for the subsequent Native Listening Circles and Tribal Consultations that led to NLM’s Native Voices exhibition which is discussed in a companion chapter [22].

3.2.Health information hispanic outreach

The 1995 NLM outreach review also concluded that NLM outreach to Hispanics was limited and needed a boost. A key element of the NLM initial response was the “Health Information Hispanic Outreach” project in the South Texas Lower Rio Grande Valley. The NLM “HIHO” project was built in part on earlier NLM circuit rider outreach pioneered by the iconic Mary Jo Dwyer who made the rounds from San Antonio to visit small, rural libraires and hospitals in South Texas. She helped facilitate access to NLM health information, at first in paper and CD formats, and later via the nascent Internet. Under the direction of Virginia Bowden Ph.D., at the University of Texas Health Sciences Center at San Antonio (UTHSCSA), HIHO included a needs assessment, focus groups, and surveys. It identified additional outreach needs and two specific initiatives for further study: a high school peer tutoring project, and a Colonia Promotoras project [23].

The Peer Tutoring project emphasized the training of high school students at health-focused high schools to serve as “peer tutors” to other students. The peer tutors helped students understand NLM’s online health information resources, MedlinePlus, and MedlinePlus en espanol, and how to use the Internet and World Wide Web to access such information. Peer tutors also were coached on how to provide peer leadership and outreach to local communities, at events such as health fairs, to further enhance health information access. The school librarians and UTHSCSA librarians teamed with high school teachers and staff to provide guidance and training to the peer tutors. Most of the students in these schools were of Hispanic origin, and many would be among the first in their families to graduate from high school and go on to some form of higher education. The Peer Tutor project had an evaluation specialist from the outset, Dr. Olney, who assured that the “Measuring the Difference” manual was properly applied. Each phase of the peer tutor project was evaluated on a variety of process and outcomes measures [23–25]. For many peer tutors, the experience was life changing and helped motivate them on to community or four-year colleges. They continued to use MedlinePlus beyond their high school graduation, and to advocate for family and friend’s use of MedlinePlus. Dr. Lindberg very much enjoyed meeting with the peer tutors and school librarians when they visited NLM and presented at the NLM Board of Regents. Reaching out to and enabling the next generation of student and library leaders was always a priority for Dr. Lindberg.

The Colonia Promotoras project deployed Hispanic community outreach workers and health advocates, many of whom were bilingual [26,27]. The Promotoras served as intermediaries between NLM health information accessible at local computer labs, and low-income community residents of the Colonias which are unincorporated towns near the Texas border with Mexico. The Promotoras submitted written reports of the health topics they helped residents explore on MedlinePlus and the ways in which the residents used the information. These reports along with verbal interviews constituted a database of “stories” that were like those collected in the original NLM Critical Incident Technique study. Dr. Olney was the project evaluator.

4.1.Internet technical performance

The first evaluation component that NLM focused on was the technical performance of the Internet. New NLM policy encouraged use of the Internet to access the Library’s information resources from international locations. Soon enough, Dr. Lindberg was receiving complaints from colleagues in other countries, specifically the United Kingdom and Western Europe, about the slowness of the Internet traffic coming from the United States. Dr. Lindberg asked Dr. Siegel to investigate these concerns. Dr. Lindberg assembled a team including himself, Dr. Wood, and Victor Cid from NLM’s Office of Computer and Communication Systems (OCCS).

Dr. Wood was an electrical engineer with experience in technology studies and extensive prior work at the Congressional Office of Technology Assessment in the information technology arena. Mr. Cid, a computer scientist, had just completed a seven-year project with the BITNIS, a pre-Internet system for responding to user requests for information from NLM’s MEDLINE. BITNIS users originally were clustered in South America and later extended to other geographic areas. BITNIS ran on BITNET, which was a technical network using telecommunications and packet switching among then-dominant mainframe computers [30]. Mr. Cid was familiar with the Internet network architecture and operations at this very early stage. In 1995, BITNET was being phased out in favor of the then nascent Internet.

The OHIPD-OCCS team developed one of the first frameworks for monitoring and measuring Internet quality of service. At the time, there was no widespread commercial Internet performance monitoring system, but there were a few start up companies and engineering experiments.

NLM met with Internet engineering researchers and experts, and developed a system to monitor the bandwidth, speed, and routing of packets of information moving over the Internet. NLM’s system tracked packets between NLM’s computer center and host computers at partnering institutions in the U.S. and abroad. These were from the NNLM, and overseas from the International MEDLARS Centers which were distribution points for access to MEDLINE data stored in the participating home country’s library or information center. The MEDLARS Centers proved to be important for several years prior to and during the transition to the Internet.

The technical testing metrics and tools used by NLM included:

With a test network at home and abroad, and using the metrics and test tools mentioned, NLM was able to answer Dr. Lindberg’s questions about Internet connectivity and performance.

The delays in Internet traffic between the U.S. and the UK and Western Europe were occurring not in the U.S. when Americans were sleeping, but from the UK/European side due to inadequate local infrastructure and the increase in their own Internet traffic associated with the beginning of their business day. This pattern held for several other countries as well.

Further testing showed that the speed of the Internet traffic depended on the bandwidth of the pipe, the time of business day at the sending and receiving locations, and the types of content being transmitted. This Internet performance evaluation approach was of great interest in the U.S., to the Federal Government, private companies, and academic institutions alike, and to international groups such as the G-7 [31].

Eventually, NLM migrated to commercial Internet performance companies, such as Keynote, that assembled their own monitoring networks and provide Internet performance data on a wide-scale basis. The data could be used for benchmarking, Internet capacity planning, and trouble shooting.

In 1998, the NLM team published a landmark paper on this topic in the Journal of the American Medical Informatics Association [32]. Also, the NLM team taught a continuing education class on this topic at the annual AMIA meeting. And Internet performance was included in other NLM and NNLM projects, see e.g. [33].

Internet performance and quality of service continues to be an integral part of Internet network management and engineering in the U.S. and globally, by the IT departments and/or commercial providers, for most organizations of all types and sectors, including the NLM OCCS.

4.2.Website usage data

In the late 1990s, NLM transitioned to website-based platforms for information dissemination and database access, using the rapidly expanding Internet. Dr. Lindberg was vitally interested in understanding the usage of NLM websites, as they became the primary information platforms for NLM.

Dr. Lindberg’s driving interest helped prompt the use of successive generations of web log software by most NLM websites. In order to meet Dr. Lindberg’s desire for usage data across NLM websites, the OHIPD coordinated a trans-NLM project to collect and make available key usage data for the larger NLM websites. The primary metrics selected were:

There was considerable debate about how many and which metrics to include, and how often to collect data. The consensus was to include a small number of required core metrics but allow individual websites to use additional metrics if needed. Most web log software then and now provide additional options, such as first time versus repeat visitors, time per visit, visits per use, and pages most frequently visited. The consensus was to collect data monthly but report on a quarterly basis. The NLM operating divisions assigned staff to manage the data collection and reporting. The data were accessible to all interested NLM staff via the NLM Intranet. This usage data collection and reporting framework worked well for many years.

At the same time, web log software and metrics continue to evolve, and NLM has periodically upgraded and transitioned to newer software packages and systems. NLM is well on its way to implement the type of website data management that Dr. Lindberg dreamed of years earlier but was not yet fully possible in his working lifetime. He would have been very pleased about the progress being made.

4.3.Web user satisfaction data

In the pre-Internet era, Dr. Lindberg was always interested in user opinions of and satisfaction with NLM’s health information resources and databases. Pre-Internet, NLM was able to conduct or sponsor surveys of library patrons, since most access to NLM information physically occurred in a library setting. In a transitional electronic environment, such as Internet Grateful Med, users were required to sign up and pay a connection fee for service. Thus, their names and contact information were known, and they could be contacted with in person, mail, or email customer satisfaction surveys.

NLM’s transition to the free Internet, with no sign up requested or required, fundamentally changed the customer feedback process. NLM no longer had user contact information, and thus could not use standard survey methodology. New, online survey methods were needed and developed.

NLM quickly found that customer response rates to online queries were much lower than in person surveys or surveys mailed to known customers. It was common to obtain 50 percent or even 75 percent response rate from paper surveys in library settings. In contrast, a typical response rage in online surveys was and still is typically five-ten percent at best, and frequently three to five percent. This raises serious questions about non-response bias.

NLM studied the options for online surveys and ended up heavily using the American Customer Satisfaction Index (ACSI) online survey services, for many years from about 2000 through 2018 [34].

4.4.External benchmarking

Dr. Lindberg also was interested in how NLM website performance compared with other websites, in other Federal agencies and in the private sector. In response, OHIPD pursued two avenues to obtain external data on online health consumer and physician use of and satisfaction with online health websites; and to obtain comparative data on health website usage [35].