Implementation and impacts of the Substantial Gainful Activity Project demonstration in Kentucky

1.1Background and intervention

As described in Foley et al. (2019), the literature has shown that a very small percentage of SSA beneficiaries return to paid employment for an extended period of time. Roughly 40 percent of SSA beneficiaries who use VR services increase their earnings in the following year, although a large majority (88 percent) do not earn above the SGA level (U.S. Government Accountability Office, 2007a). There is evidence that some VR agency practices are correlated with better employment outcomes among SSA beneficiary clients (U.S. Government Accountability Office, 2007b). For example, VR agencies that provide clients with specialized benefits counseling, job placement services, and faster service delivery are associated with improved VR outcomes (Honeycutt & Stapleton, 2013; Tremblay et al., 2006). Honeycutt and Stapleton (2013) examined the impact of the amount of time beneficiaries must wait between application and receiving VR services; they found that 48 months after application, those who waited longer had fewer months of SGA-level employment and benefits than those served more quickly.

The SGA Project model was designed with insights from these findings and data collected and analyzed by ICI and Mathematica (Foley et al., 2020a). The model consists of four innovation components: a faster pace of services and rapid client engagement, financial and benefits planning, job placement services, and a coordinated team approach.

1.2Kentucky office of vocational rehabilitation service delivery environment

The Kentucky Office of Vocational Rehabilitation (OVR) offered a unique service environment to customize and test the SGA Project model. During the period of the demonstration, Kentucky residents had similar employment rates to the national population but lower levels of educational attainment and higher rates of poverty (U.S. Census Bureau, 2018). Although the Kentucky average household income was less than the national average, the average monthly SSDI benefit amount was almost as high—$1,138 for Kentucky compared with the U.S. average of $1,165 (Social Security Administration, 2015). OVR had a slightly lower share of cases that closed with competitive employment compared to closures across all agencies—54 percent versus 59 percent in fiscal year 2014 (Rehabilitation Services Administration, 2016).

OVR agreed to participate in the SGA Project for several reasons. For many years, OVR had been interested in building its capacity with work incentive counseling, and the SGA Project demonstration provided an opportunity to incorporate benefits counseling into day-to-day operations. The SGA Project demonstration also aligned with asset development efforts in Kentucky’s State Plan for Independent Living. The SGA Project also offered an opportunity to build on and guide the state’s Employment First program goals, which seek to improve the employment rates and quality of life for individuals with disabilities.

1.3SGA Project innovations versus usual VR services

As previously noted, ICI and OVR collaborated to customize the model for implementation in Kentucky. In this section and Table 1, we compare the SGA Project enhanced services with usual VR services received by Kentucky clients. More detailed information about the treatment sites that delivered enhanced services and training and technical assistance activities is available in Sevak et al. (2017).

1.4Pacing and engagement

The aim of the SGA Project enhanced services was to accelerate the VR process for application, eligibility determination, and individualized plan for employment (IPE) development in ways that differed significantly from usual service provision. Under the enhanced services, VR counselors were to use presumptive eligibility guidelines and complete the eligibility process within two days of application. Under usual practice, Kentucky VR counselors coordinated the eligibility determination process within 60 days. Pacing to an IPE was also faster under enhanced services, with counselors encouraged to complete an IPE within 30 days of application; usual VR practice required an IPE within 90 days of eligibility.

1.5Financial counseling

The SGA Project emphasized financial counseling. While under usual service OVR relied on fee-for-service vendors to provide work incentive coordination assistance, the SGA Project allowed for hiring Kentucky work incentive coordinators who were trained and certified to provide a much richer set of early and ongoing financial counseling. The work incentive coordinators were co-located with OVR staff, with each one assigned to two or three districts to serve exclusively SSDI-only beneficiaries enrolled in the SGA Project demonstration. Coordinators completed benefit summary analyses for clients, provided additional supports including the development of financial inventories, and offered asset development counseling and related coaching. These services occurred after IPE development.

1.6Job placement services

The SGA Project had expectations of earlier and more frequent client contact with job placement specialists under enhanced service than usual service. With enhanced services, the specialists also participated in team meetings with clients, VR counselors, and work incentive coordinators before IPE development; provided follow-up contact to support job search activities; and maintained periodic contact with clients about employment issues.

1.7Coordinated team approach

A key component of the SGA Project was a coordinated team approach for staff to discuss and determine the services needed by SSDI-only clients. The team consisted of a VR counselor, a work incentive coordinator, and a job placement specialist, with the VR counselor acting as the team leader for purposes of scheduling and leading the initial meeting. The first team meeting was to occur within five days of the eligibility determination and before IPE completion. By working as a team, these key OVR staff aligned their views of the client’s goals and needs, resulting in a mutual understanding on the part of the client and team about the VR process, work incentives, vocational abilities, and opportunities for a successful and timely competitive employment outcome. Usual service in OVR did not use a team approach.

2.1Random assignment and enrollment

We randomly assigned seven of the 14 OVR service districts in the demonstration to provide enhanced SGA Project services (treatment) and the remaining seven districts to provide usual VR services (control). To divide the districts into groups with similar profiles, we conducted random assignment within pairs of districts matched on geographic region, urban versus rural location, and SSDI-only client employment outcomes before the demonstration began. We randomized districts, rather than individual counselors or clients, to minimize the burden of implementing the demonstration with fidelity.

OVR enrolled all eligible individuals who applied for OVR services between May 1, 2015, and July 29, 2016, into the demonstration. Eligibility criteria included being ages 18 to 64 at application, having a disability other than blindness, receiving SSDI on the basis of one’s own employment, and not receiving SSI at application.

One concern with a treatment and control group design is that contamination can occur if some members of the comparison group receive service changes as a result of the project’s implementation. Based on the qualitative interviews we conducted with staff, it appeared that staff at districts providing usual services were familiar with the SGA Project innovations. Nonetheless, they maintained a business-as-usual service delivery approach. Most staff we interviewed at the control sites expressed little interest in implementing the innovations, such as faster pacing. In addition, staff at the control sites did not have access to technical assistance or enhanced benefits planning and early job placement resources implemented at the treatment sites. Hence, we believe the potential for contamination at the control sites was minimal.

2.2Implementation evaluation

We evaluated the implementation of the demonstration based on: (a) VR administrative data on eligible applicants through April 19, 2017; (b) data collected in Spring 2016 and 2017 during two rounds of site visits featuring multiple interviews with OVR leaders and staff at districts across the state providing enhanced and usual VR services; and (c) interviews with staff at ICI who provided training and technical assistance to OVR throughout the demonstration.

2.3Impact evaluation

Using administrative data from OVRs case management system through April 19, 2017, we examined four pre-specified service and employment outcomes related to the goals of the demonstration. We estimated impacts of the demonstration as a whole, rather than impacts of each of the innovations separately, because all of the enhanced service elements of the SGA Project were available at treatment sites.

To assess whether the innovations had an impact on service pace, we compared the rates at which SSDI-only applicants at the treatment and control sites obtained a signed IPE within 30 days of application. We selected this outcome for pace of services because the IPE is an important service-delivery milestone—services generally do not begin until such a plan is in place and the 30-day window was a goal in the SGA Project innovations. To assess whether the innovations had an impact on client engagement, we examined whether clients disengaged from VR before attaining competitive employment. By this definition, clients whose cases had closed for reasons other than competitive employment were classified as not being successfully engaged in services. We selected this measure for client engagement because it is common for VR applicants determined eligible for services to drop out before services have begun or are completed.

To assess the impact of the innovations on employment, we examined two outcomes. We selected the first, the rate at which applicants closed from OVR with a competitive employment outcome because it is an important goal of VR services. We selected the second, closure with earnings at or above the nonblind SGA level because achieving SGA-level earnings was the demonstration’s ultimate goal. We measured the employment outcomes at the time clients’ cases closed (that is, when they stopped receiving or attempting to receive services); it is possible that employment outcomes could evolve over longer periods of time not covered by our evaluation.

Using a multivariate regression framework, we estimated impacts as the difference in each of these four outcomes between demonstration participants at treatment sites and control sites. We used an “intent-to-treat” design, meaning all applicants meeting the study inclusion criteria at the treatment and control sites are included in the analysis, regardless of whether they received services or disengaged with OVR. While we found little to no differences in client characteristics (Table 2) or pre-demonstration district outcomes, we controlled for pre-demonstration district outcomes and characteristics of the clients, including sex, race, Hispanic ethnicity, age at application, education at application, previous VR application, primary impairment, employment status at application and month of application. The models also include adjustments to standard errors to account for clustering due to office level, rather than individual-level random assignment. We used a wild cluster-bootstrap percentile-t procedure (Cameron et al., 2008). This approach uses bootstrapping to address issues present when estimating cluster-robust standard errors with a small number of clusters (5 to 30); in our analysis each of the 14 offices represented a cluster. The approach is conservative and as such it reduces the probability of falsely concluding the demonstration had significant impacts.

3.1Participant characteristics and validity of comparison group

Treatment and control group clients had similar demographic and background characteristics (Table 2), affirming our confidence in the random assignment and the credibility of the control group. OVR served 522 nonblind SSDI-only clients at treatment sites and 447 nonblind SSDI-only clients at control sites. Just over half of control group members were male, 77 percent were white, about 22 percent were black, and less than 1 percent identified as Hispanic. Two percent of treatment clients were transition age (ages 18 to 24); most were ages 45 to 54 or ages 55 to 64 (32 and 28 percent, respectively). Almost half of treatment group clients had cognitive or psychosocial impairments as the primary impairment. At the time of VR application, 42 percent of treatment group clients had earned a high school diploma, 23 percent had some postsecondary education (but no degree), 10 percent had an associate’s degree, and about 14 percent had earned at least a bachelor’s degree. A majority (53 percent) had previous VR closures prior to the demonstration start. The two groups differed significantly on only three characteristics: treatment group members were roughly half a percentage point more likely to be Hispanic, three percentage points less likely to be transition age, and six percentage points less likely to have an unknown impairment.

3.2Implementation

Data collected during the demonstration revealed that while treatment sites successfully delivered many of the SGA Project innovations, participants did not fully or consistently receive all components of the innovation (Table 3). We summarize implementation findings here; additional findings including office to office differences are included in the interim and final evaluation reports (Martin et al., 2017; Sevak et al., 2017).

While the goal was for eligibility determination to occur within two days of application, that goal was met for about 40 percent of all nonblind SSDI-only applicants and the average time between application and eligibility was 11 business days. Roughly a quarter had a signed IPE within the project goal of 30 days of VR application and the average number of days from application to IPE was 62 days. More than 60 percent received benefits counseling, and one-third received job placement services, each which were to be provided as appropriate. And although over half of clients participated in an initial team meeting, just 27 percent of those who had a meeting completed it within the project goal of 5 business days after application.

Information we learned from site visits and interviews is consistent with these findings. Staff at the treatment sites encountered challenges that affected their clients’ access to the innovations. First, because the pacing innovation was a major shift from their usual, more deliberate approach to IPE development, some counselors resisted adopting the faster approach at first. Second, delays in obtaining documentation from SSA needed for eligibility determinations and benefits analyses prevented some staff from meeting the pacing targets. Large caseloads and staff turnover also made it difficult for staff to meet the SGA Project’s targets for pacing and service delivery for some clients. Finally, some staff struggled to effectively adopt the coordinated team approach because of logistical barriers related to scheduling team meetings and confusion about the roles and responsibilities of the team members. But over time and with technical assistance from ICI, counselors grew more adept and were better able to meet the pacing and service goals.

3.3Impact estimates

We found that the SGA Project had a large and statistically significant impact on the pace of services (Table 4). At treatment sites, 27 percent of applicants obtained a signed IPE within 30 days of application while 8 percent of control group clients met that target. After accounting for client characteristics and site differences in pacing before the demonstration period, this translates to a 17 percentage-point increase, which indicates that the SGA Project innovations more than doubled the rate of IPE development within 30 days of application.

Our findings indicate that 59 percent of clients at the SGA Project’s treatment offices had been successfully engaged with VR as of April 2017. However, after controlling for client characteristics at application and district-level differences before the demonstration, we found that this rate is not statistically different from the 54 percent rate we observed among control group clients.

The SGA Project innovations led to a substantial increase in the percentage of cases that closed with competitive employment. The rate of closure with competitive employment as of late April 2017 was nearly 26 percent among clients at the treatment group sites while it was 14 percent at control group sites. We calculated this rate among all applicants and not just those who closed. The regression-adjusted estimates indicate the demonstration increased competitive, integrated employment at closure over this time period by 8.2 percentage points.

Finally, the rate of closure with SGA earnings was 8.2 percent among treatment group clients compared to 2.2 percent among control group clients. The 5.7 percentage-point regression adjusted impact estimate, calculated for all applicants and not just those who closed with employment, implies that the SGA Project innovations led to a nearly three-fold increase in the rate of closure with SGA-level earnings among applicants whose cases had closed by April 2017.

Because OVR still had open cases (33 percent at treatment sites and 39 percent at control sites), the large estimated impacts on employment and SGA-level earnings are likely to change as more cases close. If we assume that open cases at treatment sites close at the pre-demonstration rate for competitive employment, then the employment rate for open cases at the control sites would need to more than double (to about 35 percent) to eliminate the impact of the SGA Project innovations. The rate for the open control group cases needing to double to eliminate the estimated impact suggests that the SGA Project’s positive impact on competitive employment will likely persist after all cases have closed.

3.4Lessons learned

OVR staff described several key lessons learned while implementing the SGA Project innovations. First, counselors became more accepting and better skilled at delivering the enhanced services over time. VR counselors became aware of presumptive eligibility guidelines and more adept at dynamic IPE development strategies. OVR staff also accepted faster pacing as a best practice when they observed clients who were more engaged and saw the first successful VR closures among clients who received faster pacing. Field staff increased the frequency of their collaborations and knowledge exchange with colleagues, most notably via the coordinated team meetings.

Second, while we cannot separately estimate the quantitative impact of any individual enhanced service, OVR staff praised a number of the enhanced service elements in particular. Staff believed that the faster pace of services and the consistent and early involvement of the work incentive coordinators were the innovations’ most important features. Most executive leadership and local managers praised the use of these coordinators as a valuable innovation that was essential to the project’s success. Work incentive coordinators helped reassure clients who frequently expressed apprehension about how earnings would affect their disability benefits. Some staff believed that team meetings also boosted staff morale and proved beneficial for clients. Because these features represented significant departures from the usual services, they likely contributed to the observed impacts, even if we cannot quantify their contributions apart from the other innovations.

3.5Limitations

Readers should note some limitations of this study when interpreting the findings and considering the applicability of the SGA Project innovations to their agencies’ service delivery practices. First, as mentioned earlier, the large share of demonstration cases that were still open at the time we conducted the evaluation mean that the long-term impacts of the demonstration are currently unknown. Second, because all of the innovations were available in treatment sites, we were unable to assess the impacts of each innovation on its own. Third, while the demonstration could have an impact on many outcomes, our findings are limited to the VR service and closure outcomes documented in OVR administrative data.