The state of the science of health literacy measurement

1.Overview and history of health literacy measurement

The 2004 seminal Institute of Medicine report Health Literacy: A Prescription to End Confusion identified the development of new measures of health literacy (HL) as a key priority for the field [35]. Numerous scientific calls and proposals followed to develop and test HL measures in support of that recommendation [30]. Now, more than a decade later, over 150 HL measures exist, demonstrating a sweeping response to the scientific calls and reflecting tremendous productivity in this area [17,34,37]. This growth presents both opportunities and challenges for the field [26,42]. In one respect, each of these measures provides a degree of utility and valuable lessons as the field moves forward. Despite these efforts, however, no “gold standard” measure for HL has emerged, and the variety of measures has made comparing results across studies and populations a serious challenge.

This article distills the state of the science of HL measurement at the level of the individual by addressing several key areas. In Section 2, we examine the relationship, or lack thereof, between definitions and measurement. In Section 3, we summarize the conceptual domains most and least frequently measured, using the most current and well-accepted definitions and theoretical frameworks guiding the field. In Section 4, we synthesize current methods used to validate HL measures and discuss their relative strengths and limitations. In Section 5, we evaluate characteristics of the participants in the validation studies used to establish HL measures. In Section 6, we discuss the practical considerations related to the various modes of administering HL measures. In Section 7, we identify measures that assess HL subjectively versus objectively and discuss the relative advantages and disadvantages. Finally, in Section 8, we provide a road map for the future of HL measurement.

The overarching aims for this article are to help readers identify factors that contribute to stronger HL measures, provide tools to help readers select and use appropriate HL measures, and put forth a rational strategy for advancing HL measurement. An inventory of all the existing HL measures available has been omitted intentionally as an easy-to-use, publicly available Health Literacy Toolshed (www.healthliteracy.bu.edu) serves this purpose.

2.Relationship between definitions and measurement

In general, the HL field has promulgated separate discussions regarding the definition of HL and the measurement of HL [42]. Despite promising recent work developing tools in tandem with definitions, the disconnect between definitions and what the tools measure has been a persistent conceptual stumbling block, which has led to several conundrums that will need to be solved for the field to progress in a coherent manner.

Multiple definitions of HL exist; however, some are vague or inadequately specified to allow measurement. There are even more tools to measure HL, but many are loosely related to a definition [17,34,37,47]. This section examines the interplay between HL definitions and measurement from a legacy perspective verses more recent HL tools. It also looks at the paradox and barriers that exist to more fully integrating HL definitions with measurement.

3.Conceptual domains measured and those rarely measured

Conceptual frameworks can help formulate research questions and examine relationships among predisposing variables, mediators, moderators, and other relevant outcomes [11,36]. Frameworks can also inform an understanding of the domains that comprise a complicated construct. Similar to the fact that the term “health literacy” is widely used but not always well understood or applied consistently – as suggested by the sizeable number of different definitions that exist [3,47] – multiple HL conceptual frameworks have been put forth [25,39,47,49]. Yet, no single framework has gained significant traction or is viewed as the gold standard.

Examining the conceptual domains included in existing HL measures can offer insight into the construct as a whole. The National Assessment of Adult Literacy (NAAL) [20], for example, included three broad domains when examining HL using a more skills-based approach than had been used previously. Specifically, the NAAL used health-related stimulus materials that reflect the type of materials adults encounter in real life to examine three domains: (1) prose literacy, measured as the knowledge or skills needed to search, comprehend, and use information from text organized into sentences or paragraphs; (2) document literacy, defined as the knowledge and skills needed to search, comprehend, and use information from noncontinuous text in various formats (such as job applications, payroll forms, transportation schedules, and maps); and (3) quantitative literacy, measured as the knowledge and skills needed to identify and perform computations using numbers embedded in print materials (such as balancing a checkbook, calculating a tip, or figuring the amount on an order form).

The Health Literacy Toolshed website launched in 2015 and houses over 120 instruments to measure HL, with a goal of increasing the number of instruments available over time. The Toolshed developers organized the initial set of instruments into the following domains: prose (both pronunciation and comprehension); numeracy; communication (both speaking and listening); information seeking in documents, in addition to interactive media navigation; and skills related to the application and function of health information.

Most of the instruments include more than one domain when measuring HL, illustrating the overall complexity of the construct (Table 1). About half of the instruments included the numeracy domain, suggesting a general consensus that it is a core HL component, although instruments that measure only numeracy also exist. A lower number of instruments include a pronunciation domain, reflecting some of the more historical approaches to measuring literacy and perhaps indicating a movement away from this measurement approach. The communication components of HL – including both listening and speaking – are rarely included.

A small but growing number of instruments (n=19) include an application or functional component. About one quarter (n=26) examine information seeking in documents, and fewer (n=13) examine information seeking via interactive media navigation, including websites. Taken together, the relatively large number of domains used across the array of existing instruments to measure HL confirm that it is a multidimensional construct that must be measured carefully and completely using multiple items and stimuli in which a user can demonstrate their ability to interact effectively.

4.Limitations of validation methods

The limited utility and predictive validity of some HL measures may reflect the methods used to develop and validate them. Consequently, exploring how existing measures were validated will provide insight into potential limitations and directions for future work.

At its core, measurement consists of rules for assigning numbers to objects, or concepts, in such a way as to represent quantities of an attribute [36]. The term “rules” indicates that the method of assigning numbers to attributes must be stated explicitly. The construction, scoring, refinement, and validation of latent scales are most commonly guided by psychometric methods associated with Classical Test Theory. However, Modern Measurement Theories offer practical solutions for measurement problems found in health-related research that have been difficult to solve using classical approaches [8,12,14]. This section first examines the advantages of Modern Measurement Theory, then reviews methodological approaches (i.e., Classical Test Theory vs. Modern Measurement Theory) used to develop and validate current HL measures, and finally suggests approaches for moving HL measurement forward. For brevity, it is assumed that most readers have a basic understanding of Classical Test Theory. A short description of Modern Measurement Theory is provided. Resources are available for those who wish to learn more about both approaches [16,36].

5.Limitations of validation samples

Given that the vast majority of HL measures were validated using Classic Test Theory, examining the samples from which they were validated is necessary to understand how these findings may or may not be reasonably generalized. It is easy to forget about such limitations and then make inaccurate conclusions. For example, it would not make sense to develop and validate a tool exclusively with female participants and then to assume that it will perform the same way with males.

A systematic review of HL measures examined the racial and ethnic composition of participants in validation studies for 109 tools to measure HL [34]. Of the 72 English-language measures examined in this review, 17 did not specify the racial/ethnic characteristic of their sample. Of the remaining 55 measures, 10 (18%) did not include blacks, 30 (55%) did not include Hispanics, and 35 (64%) did not include Asians in their validation sample. When Asian Americans and Hispanic Americans were included, they accounted for small percentages and numbers in the overall sample; interquartile range = 10%–34% (n=13–154) and interquartile range = 3.5%–16% (n=5–36), respectively.

Consequently, it is likely that inappropriate assumptions have been made when using these tools in other contexts. Additionally, the nature of the bias introduced by such assumptions cannot be estimated. For example, if a tool misclassifies Hispanics because it was developed with a sample that did not have enough Hispanic participants to ensure validity, subsequent analyses for Hispanic participants in studies using this tool could be misinterpreted. Therefore, it is important to interpret much of the HL literature with caution. If a classical approach is used to validate future HL measures, it is imperative that sampling strategies reflect the needs of high-risk groups. Correspondingly, the characteristics of the sample should be described in sufficient detail because this information has implications for the generalizability of a given measure.

Among the 37 non-English-language measures, only two specified the racial/ethnic characteristics of their sample beyond simply describing the general population in which the measure was being validated. For instance, Ko et al. [19] specified that the sample used to validate their “Health Literacy Test for Singapore” was 52% Chinese, 22% Malay, 24% Indian, and 10% Other. In comparison, most other non-English-language measures were similar to the “Hebrew Health Literacy Test,” which reported simply that 119 Israeli participants were sampled to validate their scale; it is unclear which ethnolinguistic groups were represented from this highly multicultural society. The ethnolinguistic and cultural diversity of a specified population will influence the extent to which this may be problematic. For example, this issue is less relevant for Korean-language HL measures because the population is relatively linguistically homogenous, whereas it may be a greater concern for the use of Hebrew-language or Hindi-language HL measures, as these populations are more linguistically and culturally diverse. Future validation efforts should take into account the cultural diversity of the target population and include such details in reports.

Using Modern Measurement approaches to validate HL measures will reduce some of the challenges associated with the need to revalidate measures when used in different populations. While this property of population invariance found in modern approaches is robust, it does not always hold across all items and populations. Consequently, it will remain important to characterize validation samples to allow for DIF testing. When using Modern Measurement approaches, however, the sampling goal should be to obtain an equal distribution of participants with varying abilities across a latent trait; for example, the sample should include a sufficient number of participants with low, medium, and high levels of HL. This will lead to more stable parameter estimates for each test item. Once the parameters are estimated using an ideal “reference” group, differential item functioning can be tested against any number of different “focal” groups. Additionally, when DIF is identified (e.g., by race), a particular item may be excluded or a correction factor can be introduced. For tools developed with classical methods, if an item does not work the same across groups of participants, investigators do not have recourse. Such results cannot be corrected post facto. Consequently, it would be better to extend the validation cohorts for these tools or to abandon these tools for new data collection ex ante. Racial and ethnic data for the validation samples for each tool listed in the Health Literacy Tool Shed can be reviewed by choosing the “Read all details” option for any specific tool.

6.Practical considerations when using health literacy measures

Researchers and practitioners use HL measures for various reasons, including patient-level assessment, intervention activities, and surveillance. Each situation may invoke the need for a different type of HL tool. For example, clinicians and healthcare professionals may want to assess the HL level of a sample of their patients to understand the general needs of the population they serve, or they may want to assess all new patients and need a tool that is easy to implement in a clinical setting (though clinical screening has not been shown to benefit patients) [40]. Also, researchers may be implementing a public health or community-based intervention and need to measure HL before and after implementation, or use HL as a control variable in an analysis examining a specific health outcome. In both clinical and research settings, HL measurement may be used to trigger specific interventions or to ascertain the possible differential impact of various interventions across the HL strata. In some instances, large-scale periodic HL assessments are conducted at the health system level or even at the national level.

Selecting the right HL tool is critical because different HL measures and different data collection strategies may be needed in a given situation. For example, what is the age and racial/ethnic diversity of the target population, what languages are spoken in the target population, what resources are available, and what are the measurement goals? These factors may influence the data collection method used and should be considered carefully at the outset of any program and in conjunction with the decision about which HL tool is used. Data collection methods range from mail, telephone, web-based or computer-based, mobile device-based, to in-person, or some combination thereof. In-person data collection could use paper-and-pencil or computer self-report, interviewer-facilitated approaches, or face-to-face verbal communication. Based on the 128 measures currently in the Health Literacy Tool Shed, more measures were developed and validated using paper-and-pencil and in-person strategies. A limited number of measures have been validated for web-based data collection, and very few have been validated for telephone administration (Table 2).

Each mode of administration offers strengths and limitations that are also related to the design of the project or study. For example, with web-based data collection, tools can include visual and/or interactive stimuli as part of the HL assessment process, such as food labels, health insurance forms, or health-related websites. With computer-based data collection and computer-assisted telephone interviewing (CATI), responses are recorded automatically into a dataset for analysis and interpretation, making data entry unnecessary. Development and programming expenses reflect certain fixed costs for computer-based and telephone strategies, and items can be modified relatively easily. Print-based HL tools typically require fewer data-collection fixed costs but involve costs associated with mailing surveys to study participants and follow-up and data entry. Mail-only surveys generally have lower response rates, but they can be improved with telephone or other types of follow-up [6]. Neither mail nor telephone surveys allow for the use of interactive stimuli to assess HL. With mail surveys, participants can be asked to read and interpret text and visuals, which is not the case with telephone administration. Web-based and in-person data-collection modes can use aural approaches. Costs, including staff training, can vary greatly depending on the data-collection mode and instrument used. There can also be an impact on response rates and data quality depending on the data-collection method used [6].

Researchers and practitioners carefully consider the needs of the study population when measuring HL. Reading ability, visual and hearing abilities, computer skills, and access to computers should all be factored into the choice of an HL tool and addressed in data-collection planning. Staff training is required not only to ensure data quality, but also to reduce potential harms. For example, individuals collecting data should be sensitive not to give the impression of testing subjects, as this could promote shame or stigma, especially in lower HL populations [7,10,52,53].

In sum, researchers and practitioners may be interested in measuring HL for different reasons. Additionally, researchers need to be mindful of the increasing diversity within populations when measuring HL. Also, having a clear understanding of the measurement goal and the target audience will help identify the best data-collection mode and type of HL tool to use.

7.Subjective versus objective health literacy measures

A complex phenomenon that has developed in HL measurement is the elaboration of objective versus subjective measures. In objective measurement, people are challenged by standardized test stimuli to measure an underlying trait; in subjective measurement, people self-report their responses to questions about their experience, typically on Likert scales. There are distinct benefits and limitations to each of these approaches.

One benefit of subjective measures is the ease of testing because these measures do not require in-person testing and typically involve less cognitive effort than objective measures. This may mean that the risk for stigma is lower for subjective measures than for objective measures. Similarly, most HL measurement tools have been developed for research purposes, but some institutions have implemented subjective HL testing in clinical care. Subjective measures are typically easier to work into the flow of clinical care because they survey peoples’ opinions. Also, subjective HL measures have the potential for rapid application. Indeed, some of the most commonly used subjective measures comprise three questions; some use just a single question [4,28]. At the same time, more elaborate versions of subjective measurement have been developed. For example, the European Health Literacy Questionnaire was developed with 47 subjective items evaluating the three domains of healthcare, disease prevention and health promotion, and a four-component structure reflecting the four dimensions of accessing, understanding, appraising, and applying health information [46,48]. It is possible that with repeated measurement over time, subjective measurements such as this could provide a different judgment by showing at a broad societal level to what extent the healthcare system is meeting the needs of the population.

The main challenge with subjective measurement is that there is no ground truth; meaning there is no way to know how a person’s responses relate to their actual skill level. This is most relevant for certain groups of people who are likely to systematically rate their experiences at a higher level than other people in a manner that does not relate to their actual HL skills. For example, people who have not had much exposure to the health system may not appreciate the high degree of complexity they may encounter, so they may have inflated responses. Alternatively, for example, if male respondents have better scores on a subjective measure in a given project, it would not be clear if this is because men truly have an easier time with the activities being reported or if this difference reflects a subjective phenomenon in the cohort whereby the men in that cultural setting express a higher degree of self-confidence than the women [23]. In some cases, however, subjective measurement may provide the information that is needed. For example, subjective measurement is likely to be more successful in predicting outcomes for populations that have enough experience and enough insight in their HL ability. However, empirical testing is needed to support this perspective, and without additional data it is difficult to interpret results within a given cohort.

The main benefit of objective testing is that it results in a direct measure of the person’s skill. There is an inherent value to having empirically grounded data. While this is often useful, there are multiple complexities with this approach. First, objective testing can feel like a test. People know that their skills are being evaluated; this can cause stigma, especially for people who struggle with the test items. Second, these tests typically require in-person testing. Third, the test items may not directly relate to the HL skills needed for a given scenario; a person’s test score in one domain or content area may not reflect their skill in another aspect of HL. For example, it would be a mistake to assume that getting a perfect score on the TOFHLA means that a person knows how to use an inhaler. Lastly, given the limitations in methodological approaches used to develop and validate most HL measures (i.e., Classical Test Theory vs. IRT), there are concerns around meaningful interpretation of scale scores. For example, under Classical Test Theory it is assumed that score intervals across the scale are equal (i.e., on a measure with scores that can range from 0–10, an individual who scores a 5 has half the ability of an individual who scores a 10). However, when applying IRT to test data, what is often revealed is that items cluster around certain areas of the latent trait; most often the middle region (i.e., items with moderate difficulty). Without modeling the density of items across a latent trait, the value of objective measurement is reduced; for example, if an individual’s HL score improves from 5 to 8, and the 3-point gain comes from items of the same difficulty level, it would be important to ask how much did that individual’s HL ability really improve.

The empirical relationship between objective and subjective HL testing has received limited attention [18]. Kiechle et al. reviewed papers that concurrently used both types of measures and related them to various outcomes. They identified four studies they rated to be fair-quality studies with pertinent data. Among these studies, one reported no difference between objective and subjective HL measures for a rheumatoid arthritis disease severity score; one showed no difference between objective and subjective HL measures for a range of self-reported disease states; one exhibited a difference between objective and subjective HL measures for a patient’s ability to interpret their prescription medication name and dose from a medication bottle; and one provided mixed evidence about the consistency between objective and subjective measurements of numeracy for predicting colorectal cancer screening utilization. While insightful, these studies do not provide adequate reassurance to support the assumption that conclusions from objective and subjective measures could be interchanged. At a conceptual level, these tools measure different constructs. Though it would add a layer of complexity, ideally the HL literature that derives from objective measures should be interpreted separately from reports from studies that used subjective measures.

Overall, the choice to use an objective or subjective HL measure depends on the goals and structural parameters of the work. Currently, most phone-based survey research will need to use subjective measurement, as it is difficult to facilitate current objective tests over the phone. When the goals of testing relate to phenomena that are better served with objective testing, this should be done if feasible. Finally, objective tests (e.g., the NAAL) may be better suited for estimating an individual’s skills, whereas subjective measures (e.g., the European Health Literacy Questionnaire) may be better suited to assess if the healthcare system is serving the population well.

8.Future directions

Much has been learned from the progress made in HL tool development since the 2004 seminal Institute of Medicine Report. This article has provided a critical review of the state of the science of those measures across several dimensions, including (1) the relationship between HL definition and measurement, (2) the conceptual domains of HL most and least frequently measured, (3) the methodological approaches used to develop and validate HL measures, (4) the characteristics of the participants in the validation studies, (5) the practical considerations when using HL measures, and (6) the use of subjective versus objective HL measures. Important patterns emerged from this critical review that can be used to help set future directions for HL measurement.

First and foremost, we need to better align HL measurement with definitions of HL. Given the number of different HL definitions that exists, it is imperative that tool developers are clear about what definition they are using and how that definition influenced the operationalization of HL in the tool. This will not only help guide the purpose and scope of the tool, but also help end-users interpret scores. Additionally, as more people use tools guided by a clear HL definition, it will help the field to further identify which definitions and theoretical frameworks are useful for understanding the mechanisms through which HL operates and how they impact outcomes.

An evaluation of existing HL tools demonstrates that some HL domains are measured more often than others. Specifically, prose/pronunciation and numeracy are the most commonly measured domains, whereas listening and speaking are the least measured domains. Consequently, it will be important to think about novel out-of-the-box mechanisms to measure these rarely measured HL domains. Novel approaches could also be used to address common issues of shame and the lack of time that is often associated with HL measurement. An example of an innovative approach is the use of gaze tracking technology while participants read a standard document [27]. It is hypothesized that the gaze patterns of individuals with high HL differ from individuals with moderate and low HL. Ongoing research is testing this hypothesis.

When reviewing the methodological approaches used to develop, validate, and refine HL measures, classical measurement approaches dominate the literature. This is consistent with patterns seen in measurement development and validation for most other patient reported outcomes [14]. However, there are strong empirical and practical rationales for making an assertive shift toward using modern measurement approaches to develop, validate, and refine HL tools. It is critical that the refinement and alignment between the definition of HL and measurement comes first before the full benefits of Modern Measurement approaches can be realized. Once refining and aligning the definition of HL is achieved, there is a valuable opportunity to build a robust HL item bank for CAT applications given the number of HL measures that exists. Creating a robust item bank has tremendous potential for addressing the lack of standardization in HL measurement, reducing participant burden, and addressing the challenge of making comparisons across studies. Achieving this will require skillful coordination, cooperation, and political will among the developers of HL tools and key stakeholders.

As HL tools continue to be developed, validated, and refined, it is critical to be mindful of validation samples. In situations where classical approaches are used, it is imperative that populations at highest risk for low HL are included in the validation sample because this has implications for the generalizability of the tool. A review of most existing HL measures demonstrates that Hispanic Americans and Asian Americans are rarely included in the validation samples of English-language HL measures, despite the fact that these groups have among the highest rates of low HL. In situations where modern approaches are used, the validation sample should aim to have an equal distribution of participants with varying abilities across a latent trait. This will ensure more stable item parameter estimates. Once a strong focal validation sample is obtained, testing for DIF across a number of different reference groups can be done.

Given the stock of currently available HL tools, a question commonly posed is, what tool should be used? Ultimately, the answer depends on the context of why HL is being measured. This article highlighted the differences between subjective and objective measures. Objective measures are generally better for situations when it is important to have a reliable estimate for an individual or set of individuals that are “grounded” in some verifiable way. Subjective measures are more strategically feasible for large-scale measurement of populations or systems over time. Various modes of measurement were also highlighted, including mail, telephone, web-based or computer-based, mobile device-based, in-person, or some combination thereof. Each mode of administration offers strengths and limitations. Identifying which administration mode is most suitable requires reflecting on the needs of the study population, including reading, visual and hearing abilities, and computer skills and access to computers. Considerations should also be made based on cost, desired response rate, and data quality. The Health Literacy Tool Shed is a valuable resource that can be used to filter existing tools based on their mode of measurement and whether they are subjective or objective in nature.

Finally, two parting reflections warrant brief mention. First, it is important to recognize that HL is a dynamic concept, and the rate at which this concept evolves is affected by language, culture, an increasingly global and mobile world, and sweeping health system changes taking place in many countries. Consequently, it will be necessary to have a more informed and sophisticated understanding of ethnolinguistic nuances and changes that occur naturally in most languages; particularly among languages spoken by people who are highly mobile and global. Ignoring these ethnolinguistic changes may decrease the content validity of HL measures over time. Likewise, to expand the focus of HL measurement into the healthcare context, a more informed and sophisticated understanding is needed of how health systems are evolving, as well as the complexities within and across different health systems that may influence HL and patient outcomes.

Second, many of the measurement-related issues identified in this article are not unique to HL. It is quite common for conceptual and operational definitions to evolve for emerging concepts of high scientific and social value. The increased recognition and inquiry often leads to a proliferation of new measures. However, even after several decades of study, this has not resulted in a gold standard.

For example, coping as a construct has a long history. Initial studies focused on psychopathology. Later, researchers moved toward positive behavior and the role of emotions. Significant concerns were identified in clarifying the concept and matching it with measurement [45]. After providing an updated review of the swiftly widening literature on stress and coping [1], it was noted that measurement remained the most controversial issue in the field. One way researchers addressed this problem was to develop coping measures specialized by situations; for example, coping with sexual trauma. While measures multiplied in number, clarity of the concept of coping in health did not advance.

Lengthy theses can be written to discuss solutions for overcome these challenges. Two broad suggestions are offered. First, it is vital to overcome disunity by moving toward unified, integrative work; both in conceptualizing the definition of HL and operationalizing it into measurement tools. Second, it is imperative that we continue to build from where we are toward higher quality psychometric studies. Continuing to engage in the status quo will do more harm than good for the field.

In closing, much progress has been made in HL tool development, which has led to a number of useful HL tools and to measurement challenges for the field. It is important to contextualize the commonality of the measurement problems unearthed in this article rather than be discouraged by them. Learning from past lessons provides a hopeful path forward.