Enhancing transition outcomes: A toolkit to facilitate data-driven pre-employment transition services

1.1WIOA and efforts to improve transition outcomes through Pre-ETS

The Workforce Innovation and Opportunity Act (WIOA) of 2014 and the Rehabilitation Act of 1973, as amended by WIOA, mandated a substantial investment of time, money, and effort in transition-age youth with disabilities receiving vocational rehabilitation (VR) services (U.S. Department of Labor, 2016). Prominent among these mandates is the requirement for SVRAs to allocate 15% of their federal funding to provide a set of core pre-employment transition services (Pre-ETS) including (a) job exploration counseling, (b) self-advocacy instruction, (c) work-based learning, (d) counseling on postsecondary enrollment, and (e) workplace readiness training. The Rehabilitation Act Amendments in WIOA also expanded the target population, specifying that Pre-ETS are available to students who are eligible, or potentially eligible for SVRA services, markedly increasing the number of students who stand to benefit from these services. Only students with disabilities ages 14–21 are eligible to receive Pre-ETS unless superseded by the state to serve those younger or older than the federally established age. WIOA mandates were also designed to facilitate increased collaboration between Local Education Agencies (LEAs) and SVRAs in the shared goal of supporting students in achieving competitive integrated employment post high school. As a result, in the years following WIOA implementation, data indicate that VR counselors provide more outreach to LEAs and serve many more students than prior to WIOA (Fabian et al., 2018; Rehabilitation Services Administration, 2022).

The timeline from WIOA passage to implementation was brief, with SVRAs expected to function under the new mandates within months (National Council on Disability, 2017). SVRAs used “general expectations” provided by federal policy and mandates to implement required Pre-ETS, specify the target populations, and to determine VR counselor responsibilities in coordination (Carlson et al., 2020, p. 44). Many of the specifics of how to provide and evaluate Pre-ETS were left to the SVRAs. Carlson et al. (2020) analyzed state plans and other SVRA-supplied policy documents from 2018 to understand how states were implementing Pre-ETS and the degree to which they were meeting the legislative requirements as specified in WIOA. Nearly all states in the sample (n = 38), offered at least some description of the required Pre-ETS core services. However, authors concluded that SVRAs “varied considerably” in how, and to what extent, state policies addressed requirements dictated in the WIOA policy (Carlson et al., 2020, p. 55). These findings were replicated and extended by a later analysis of state plans where researchers found SVRAs “varied greatly” in their descriptions of Pre-ETS, their plans for service delivery, and the skills targeted (Taylor et al., 2022, p. 67). Ten years after WIOA and the establishment of Pre-ETS, the majority of states lack a clear and operational definition of the purpose and desired outcome of each core service. This lack of operationalization has limited SVRAs in their ability to evaluate and monitor services.

1.2Current evaluation and monitoring of Pre-ETS

Enhanced requirements for coordination and funding contained in WIOA directly address longstanding weaknesses in transition services identified in previous iterations of education and workforce legislation (Baer et al., 2011). However, WIOA did not sufficiently operationalize benchmarks for whether services have the intended outcomes or even what those outcomes should be outside of the long-term outcome of employment. One exception in WIOA came in the reporting of measurable skills gains. Measurable skill gains loosely measure the effectiveness of Pre-ETS and transition services without being tied to a specific service. Measurable skill gains are defined as documented progress in one area including, (a) achievement in educational functioning level, (b) attainment of secondary diploma, (c) post-secondary transcript or report card documenting enrollment, (d) satisfactory or better progress towards established training milestone, or (e) successful passage of an exam required for an occupation or progress toward attaining skills (Performance Accountability Under Title I of the WIOA, 2017). Training milestones and skill gains are both defined as enrollment in a post-secondary training or education program that leads to a recognized post-secondary credential. These measurable skill gains are captured alongside case management data that includes whether a student received Pre-ETS, the date Pre-ETS started, and the date exited. Emphasis is also given to whether students receiving Pre-ETS fall in the category of potentially eligible or have enrolled in VR services. Agencies are also required to show that they are spending no less than 15% of their federal allotment on Pre-ETS.

The Rehabilitation Services Administration (RSA) is mandated to monitor SVRA performance, including Pre-ETS (Rehabilitation Services Administration, n.d.). While SVRAs retain responsibility for Pre-ETS coordination and provision, RSA gathers information from SVRAs, LEAs and contracted providers for Pre-ETS in their monitoring activities. These data are used to determine areas of strength and needed improvement for SVRAs related to Pre-ETS (Rehabilitation Services Administration [RSA], 2023). RSA provides the Monitoring and Technical Assistance Guide that establishes the basis for reporting standards. Values guiding the monitoring process include continuous improvement of VR programming, maximizing outcomes for VR participants, and ensuring efficient and effective use of public funds and personnel time as valuable agency resources (RSA, 2023). The most recent guide highlighted a trend of SVRAs returning increasing amounts of unused funds and RSA responded with added emphasis on financial management and investment.

The Monitoring and Technical Assistance Guide includes multiple questions for SVRAs to address in their agency profile. A section devoted to Pre-ETS includes policies and procedures used to meet WIOA requirements regarding target population, resource allocation, and how SVRAs are providing required services. SVRAs provide information through case review that includes policy documents and evidence of how policies are applied in practice. A major limitation, particularly noted in critiques of attempts to use RSA administrative data for program evaluation, is that RSA 911 data does not include any details of the services or training programs beyond a binary indication of whether or not they were received. Current measures of student change resulting from Pre-ETS are limited to reports of employment outcomes, credential attainment, and measurable skill gains that often occur months to years after a Pre-ETS was provided (WIOA, 2014). There is no requirement for states to identify duration of services, what the services included, or what changed in the immediate aftermath of services. As a result, most SVRAs do not collect this data across all students receiving Pre-ETS. Additionally, employment outcomes, credential attainment and skill gain data are not available for students who do not go on to open a case with VR and receive individualized services. Systematic measurement of change from Pre-ETS has the potential to enhance the quality of services and to improve overall outcomes (Mazurek Melnyk Morrison-Beedy, 2018).

1.3Raising the bar on best practice

Recent federal legislation has increased the focus on program evaluation, quality assurance, and data-driven decision making (Anderson et al., 2021). Through WIOA (2014), SVRAs have seen an increase in reporting requirements focusing on “results-driven outcomes” (Vocational Rehabilitation Technical Assistance Center for Quality Management, 2021, p. 1). Section 116(e)(1) of WIOA encourages evaluation that is rigorous enough to “promote, establish, implement, and utilize methods for continuously improving core program activities in order to achieve high-level performance within and high-level outcomes from, the workforce development system.” Section 116(e)(2 3) goes on to state that “evaluations shall use designs that employ the most rigorous analytical and statistical methods that are reasonably feasible . . .  ” with the goal of promoting “the efficiency and effectiveness of the workforce development system.” This language included in the section on evaluation within state agencies invites a level of rigor that far exceeds the basic reporting of what services were received.

This language from Section 116 of WIOA (2014) closely aligns with the more recent Foundations for Evidence-Based Policymaking Act of 2019 (Evidence Act), which was signed into law January 14, 2019. A primary aim of this legislation is to improve the use of evidence and data in the development and implementation of government policies. Among other things, this law mandates (a) each federal agency to designate a Chief Data Officer responsible for managing and promoting the use of data within the agency, (b) the development and implementation of a comprehensive federal data strategy to guide use, management, and sharing of data, and (c) use of rigorous evaluation methodologies that allow agencies to assess the effectiveness of government programs and to facilitate data-driven decision-making. The Evidence Act also encourages the adoption of open data practices where appropriate. The Evidence Act has already increased the emphasis and investment in enhancing equity and employment of people with disabilities through efforts being made both through the Departments of Labor and Education (Vilsack, November 2021).

This trend towards greater accountability and evaluation in government programs has continued since passage of the Evidence Act (2019). For instance, in 2022, the Biden-Harris Administration launched the Year of Evidence for Action with the intent being to strengthen and expand evidence-based government policymaking (White House, 2022). Activities for this Year of Evidence for Action included identifying and creating strategies to promote evidence-based decision-making and involved a partnership between the federal government and several other non-profit organizations and educational institutions (White House, 2022). More information on these more recent efforts can be found at the following website: https://www.evaluation.gov.

In the section that follows we introduce the Transition Readiness Toolkit (TRT) as a mechanism for enhancing transition outcomes for students receiving Pre-ETS. The TRT aligns with actions encouraged by WIOA and the Evidence Act to strengthen the evaluation of services and to provide data necessary for data-driven decision-making. Beyond describing the existing TRT, we describe the processes for its creation that align with the 13 key practices recently recommended by the Government Accountability Office (GAO, 2023), namely to: (a) demonstrate leadership commitment, (b) promote accountability, (c) build and maintain capacity, (d) involve stakeholders, (e) define goals, (f) identify strategies and resources, (g) assess the environment, (h) assess the sufficiency of existing evidence, (i) identify and prioritize evidence needs, (j) generate new evidence, (k) use evidence to learn, (l) apply learning to decision-making, and (m) communicate learning and results.

2.1Initial development and implementation

The idea for the TRT stemmed from efforts of the Utah State Office of Rehabilitation (USOR) to incorporate pre-post assessment in the evaluation of Pre-ETS being offered throughout the state. USOR, like many SVRAs, demonstrated a commitment to improving evaluation of Pre-ETS and sought to promote accountability for their effectiveness; in doing so, they embody two of the foundational key practices put forth by the GAO (2023). USORs efforts to implement pre-post assessment were motivated by clearly defined goals that led to the assessment of the sufficiency of existing evidence and identification of a need for new evidence (GAO, 2023). A collaboration between researchers and USOR began in early 2020 with an initial focus on the key practice of identifying strategies and resources, assessing the environments where Pre-ETS are provided, and involving stakeholders (initially including stakeholders in Utah and expanding in 2022 to include stakeholders from five additional states). This months-long process resulted in a determination that USOR and other SVRAs would benefit from a centralized approach for evaluating change resulting from Pre-ETS and that no existing system adequately did so (Phillips et al., 2022).

A major challenge for SVRAs seeking to evaluate change resulting from Pre-ETS stemmed from definitions that vary across states and are often not conducive to evaluation (Taylor et al., 2022). Creating greater specificity about (a) what constituted each core service, (b) essential elements of each service, (c) how core services were distinguishable, and (d) the immediate outcome from services was necessary in order to create assessments that aligned with each core service. The assessment tools took over a year to create and involved intensive collaborations with USOR, community providers, and other key stakeholders. After initial piloting, USOR began a statewide use of the assessment tools in November 2021, marking initiation of the key practice the GAO refers to as generating new evidence (GAO, 2023).

Through a NIDILRR funded project, in 2022 we engaged SVRAs and their stakeholders from five additional states representing different regions of the country to ensure transferability beyond the USOR. As part of this project, we reviewed the concept map of the assessments, their target outcomes, and each item created for the assessment tools with agency staff and a panel of parents and transition students from each of the five new states. This critical step helped to refine the instruments while also confirming that the assessments and items matched the target outcomes for Pre-ETS in other states. With priority for continual improvement, we have maintained a high level of engagement with stakeholders to refine the assessments over the past three years. This continued engagement with stakeholders from across six states resulted in a revision and improvement to the assessment tools that was implemented November 1, 2023.

Table 1 provides a brief description of the 10 assessments (nine pre-post and one post-only) and how they align with the WIOA core Pre-ETS. While most core services were more readily defined based on the laws, regulations, and guidance documents, workplace readiness required more assessments to capture the multiple facets of this core service as described by stakeholders. The other core service with multiple assessments is work-based learning; the additional post-only assessment aligns with SVRA priorities to gauge student perceptions of their work experiences and track quality metrics including wages and completion rates.

2.2Capacity building and application of newly generated data

Originally, we collected data from the assessments using a popular data collection software. This meant that for USOR and the providers to access the data, members of the research team needed to download the data, analyze it, and create custom reports for USOR and each Pre-ETS provider. This process was labor intensive and only allowed for quarterly reporting of the Pre-ETS evaluation data. Even when USOR was the only SVRA, building and maintaining capacity to use and apply the data adequately to facilitate data-driven decision-making, key practices of the GAO (2023), required a new approach. In February 2022, members of the research team enlisted outside resources to create a software for data collection, data analysis, and real-time reporting of Pre-ETS data. This software is now being used by all SVRAs to allow for the type of use, application, and communication that fosters identification and use of evidence-based decision-making that aligns with the aspirations of WIOA (2014) and the Evidence Act (2019). Through the TRT, SVRAs, in-house Pre-ETS providers, community providers, and educators providing Pre-ETS have access to customized reports and visualizations. This real-time reporting provides intuitive insights that can enhance Pre-ETS through the identification of best practices and the improvement of existing services.

Beyond the collection, analysis, and reporting of Pre-ETS change data, SVRAs often struggle with capacity issues related to training and competency; and the systematic adoption of pre-post assessment is a new concept in most if not all SVRAs. Providing training and technical assistance to produce understanding and approval of the TRT has been a critical aspect of building and maintaining capacity. In our training to providers and administrators, we emphasize the value of pre-post assessment, and how users can leverage this data to improve their services. Maintaining a well-informed and approving set of providers is no small undertaking given the amount of turnover that exists across provider agencies. There is often a need to onboard new staff or re-train existing staff, and technical assistance is available as an ongoing support. Despite the challenges of training, provider agencies (including school staff and community Pre-ETS providers) are among the most supportive stakeholders for the enhanced accountability and insights that come through pre-post assessment. The initial rollout across states suggests that Pre-ETS providers take pride in their services and many welcome the opportunity to measure the immediate impact on students.

2.3Accessibility and usability

Accessibility and usability were central in the creation of the TRT (both the assessment tools and the software). Our team worked closely with stakeholders from across the six states to ensure that the environment where the TRT would be used was understood and that the needs of providers and recipients were met. We made multiple revisions to the assessment tools, the software, and to the process of administration to maximize clarity, ease of use, and customization. Some of the primary features included ensuring that assessments were written at or below a 4^th grade reading level for greater access. We also created an alternative format that can be completed by an observer when self-report is not possible.

All assessments can be completed through an accessible online software or using a paper and pencil version to meet unique agency and recipient needs. By embedding accessibility into the assessments and providing multiple methods for administration, we sought to optimize availability to students regardless of ability or access to technology. Finally, we capped the number of items for each assessment (ranging from 8 –18 items) to make adoption more practical for busy service providers and to reduce the load on students receiving Pre-ETS. A review of submission data suggests that on average, assessments are completed in less than two minutes. The resulting assessment battery is usable by the population, practical to the setting, and tailored to the service expectations (Phillips et al., 2022).

3.1Future directions

The TRT is the result of a multi-year development process, with ongoing and intentional review and feedback from key stakeholders. Nonetheless, instrument development is a lengthy process, and we continue to engage in efforts to validate and strengthen the TRT. Our key efforts to date have focused on content and face validity, primarily through our alignment with WIOA-defined core services and gathering input from key stakeholders to ensure that the instruments are appropriate and useful for the intendent purpose. Our next validation efforts will include applications of item analysis, driven by classical test theory to examine reliability and validity of individual items. We will aim to establish concurrent validity of TRT measures, using a correlational study design against comparable accepted measures and external milestones that correspond with the desired outcome of services.

We are eager to connect this data with case data to understand trends in student outcomes. Merging our TRT data with existing case data (i.e., RSA 911) will allow for understanding trends in pre-post skill gains in Pre-ETS as they relate to accessing post-secondary education and employment. We will test for differential test functioning according to key demographics to ensure usability across our intended population. Finally, we will also be able to explore patterns in service outcomes related to demographic factors, and features of services such as length, ordering, or other elements of training that may elucidate promising practices.

4.1Demonstrated leadership commitment

Demonstrated commitment from agency leadership has been referred to as “the single most important element of successfully managing and improving the performance of federal organizations” (GAO, 2023, p. 39). Commitment can take the form of vocalizing evidence-building as a priority, spending time and resources on evidence-building activities, actively creating buy-in within an agency, and coordinating efforts across teams and agencies.

4.2Promoting accountability

From an evidence-based policy perspective, pursuing accountability demands assignment of responsibility and holding individuals accountable for learning and results (GAO, 2023). SVRAs are already responsible for Pre-ETS delivery and report metrics associated with Pre-ETS to RSA annually (RSA, 2023). As discussed, current data are limited in their utility for understanding quality and impact of services. Pre-post assessment provides the opportunity to garner data that can be used to understand performance in a manner that promotes increased accountability for Pre-ETS.

4.3Building and maintaining capacity

Building and maintaining capacity for evidence-building involves making sure that staff have knowledge, skills, and abilities to carry out the work required. More complex systems or requirements for evidence-building demand greater knowledge, skills, and abilities. Capacity also includes having the time and resources to carry out the necessary activities. Given the significant time constraints experienced by Pre-ETS providers, the resource of time cannot be overlooked. Insufficient funding may also limit capacity to enhance evaluation. Recent research has clearly indicated that limited capacity is a primary challenge among state agencies seeking to build greater evidence for services (Allard et al., 2018).

4.4Involving stakeholders

The importance of involving multiple stakeholders in transition service delivery has long been reflected in legislation (Oertle Seader, 2015), and is most successful when it occurs early and often (GAO, 2023). The last reauthorization of the Individuals with Disabilities Education Act (IDEA, 2004) defined transition services as a coordinated set of activities that are results-oriented and are designed to facilitate movement of a child with a disability from school to employment, post-secondary education, independent living, and/or participation in the community (20 U.S.C. 1401[34]). The 2014 WIOA places considerable emphasis on transition-age students with disabilities and requires states to submit a plan of rehabilitation services that outlines how the state will meet the provisions outlined in the Act. The state plan must contain policies and procedures for cooperation, collaboration, and coordination between the state rehabilitation and state education agencies developed to facilitate transition students from educational services to rehabilitation services (Rehabilitation Act, Title 1 Section [a] [11] [D]). Despite federal legislation for both special education and rehabilitation that emphasizes coordination among state agencies, the coordination often focuses on state level compliance and data sharing, with little emphasis geared toward capacity building and evaluation.