ArticlePDF Available

Use of Learning Collaboratives by the Center for Practice Innovations to Bring IPS to Scale in New York State



This column focuses on use of learning collaboratives by the Center for Practice Innovations to help programs implement the evidence-based individual placement and support model of supported employment in New York State. These learning collaboratives use fidelity and performance indicator data to drive the development of program-specific individualized quality improvement plans. As of 2014, 59 (69%) of 86 eligible programs have joined the initiative. Programs are achieving employment outcomes for consumers on par with national benchmarks, along with improved fidelity.
Use of Learning Collaboratives by the Center
for Practice Innovations to Bring IPS to Scale in
New York State
Paul J. Margolies, Ph.D., Karen Broadway-Wilson, Raymond Gregory, Thomas C. Jewell, Ph.D.,
Gary Scannevin, Jr., M.P.S., C.P.R.P., Robert W. Myers, Ph.D., Henry A. Fernandez, J.D., Douglas Ruderman, L.C.S.W.,
Liam McNabb, I-Chin Chiang, M.S., Leslie Marino, M.D., Lisa B. Dixon, M.D., M.P.H.
This column focuses on use of learning collaboratives by the
Center for Practice Innovations to help programs implement
the evidence-based individual placement and support model
of supported employment in New York State. These learning
collaboratives use delity and performance indicator data
to drive the development of program-specic individualized
quality improvement plans. As of 2014, 59 (69%) of 86 eligible
programs have joined the initiative. Programs are achieving
employment outcomes for consumers on par with national
benchmarks, along with improved delity.
Psychiatric Services 2015; 66:46; doi: 10.1176/
The Center for Practice Innovations (CPI) is funded by the
New York State Ofce of Mental Health (OMH) to bring best
practices to mental health programs across the state (1). Large
systems face the challenge of helping practitioners in diverse
programs adopt evidence-based practices. This challenge in-
cludes reaching all programs, regardless of whether they would
be naturally inclined to participate. Because of the large number
of programs and the large geographic area, New York State
(NYS) cannot afford traditional approaches to dissemination,
such as use of consultant trainers for individual agency training.
To encourage community programs to adopt best practices,
OMH developed a program model in 2004 that set a clear ex-
pectation in regard to the implementation of recovery-oriented
evidence-based practices. Through funding policies, OMH
provided incentives for adoption of these practices. These
Personalized Recovery Oriented Services (PROS) programs
are designed for adults living in the community who have
a diagnosis of serious mental illness. The programs help them
identify and achieve personally meaningful recovery goals.
The number of PROS programs has increased signicantly:
in early 2014, 86 programs were serving 10,600 individuals.
OMH leadership identied supported employment as a
key service in PROS programs. Rates of competitive employ-
ment of mental health consumers have historically been very
low. PROS programs across NYS routinely report competitive
employment rates of less than 10% among program partic-
ipants, similar to rates in other states (2,3).
Individual placement and support (IPS) has been recognized
as the evidence-based approach to supported employment
for more than a decade (2,4). A study that focused on 127
sites involved in the Johnson & JohnsonDartmouth Com-
munity Mental Health Program found that the mean quarterly
employment rate for individuals receiving IPS services over
eight years was 41% (5,6). Employment was dened as work-
ing in a competitive job for one day or more in the quarter. IPS
principles include zero exclusion, integration of employment
and mental health services, competitive employment as the
goal, and rapid job search. Key services include developing
a meaningful employment plan that reects the consumers
Learning collaboratives (LCs) have been used in health
care and are increasingly used in behavioral health care. LCs
were initially patterned on the Institute for Healthcare Im-
provement Breakthrough Series (8), and variations have been
developed (9). Fundamental elements include a number of
organizations working together, using quality improvement
methods to close the gap between potential and actual per-
formance, learning from experts as well as from one another,
and using data to track performance (8). Reviews have re-
ported promising results in regard to the impact of LCs, but
they are cautious in reaching rm conclusions because of
difculties comparing studies and other methodological issues
The NYS initiative builds on the LC model designed by
the team that developed IPS with the goals of guiding and
supporting IPS dissemination in 12 states and the District of
Columbia (12). The Johnson & JohnsonDartmouth national
LC used a two-tiered approach. IPS researchers used LCs
to work with mental health and vocational rehabilitation
leaders at the state level, and these leaders in turn worked
with programs in their states. The LC approach has helped Psychiatric Services 66:1, January 2015
states sustain and expand IPS, improving quality and achieving
good outcomes. The NYS LC initiative works directly with
large numbers of programs.
In 2013, CPI worked with 47 of 77 PROS programs across
NYS that were clustered into four regional LCs. In 2014, the
number of participating programs increased to 59 of 86,
clustered into seven regional LCs. OMH strongly encouraged
participation, but no scal or other incentives were provided.
Supported employment is funded through a combination of
Medicaid and state dollars. CPI staff (3.2 full-time-equivalent
[FTE] trainers-consultants) helped these programs by leading
LC meetings, providing guidance concerning the develop-
ment of quality improvement plans, collecting and analyzing
data submitted by the programs, leading informational webinars
and conference calls, and providing onsite program-specic
training and consultation. The LCs provide both face-to-face
and online training and support activities. Programs agree to
attend LC meetings, conduct periodic delity self-assessments,
submit monthly performance indicator data, develop and use
IPS quality improvement plans, and share experiences with
one another.
LCs offer the following training and support activities:
two online training modules, one providing an overview of
IPS and the other focusing on the practitioner skill of job
development; statewide webinar and regional online meet-
ings focusing on important topics (for example, IPS delity
and supervision) and processes (for example, using data to
drive continuous quality improvement efforts) (in 2013, one
webinar and 24 online meetings were conducted, and from
January to May 2014, four webinars and eight online meetings
were conducted); an IPS library, available through CPIs
learning management system, that provides archived webinars,
presentations from past training events, and tools to help IPS
implementation; regional face-to-face meetings where pro-
grams learn from each other (eight meetings in 2013 and 15 from
January to May 2014); individualized consultations, both at
the program site and by telephone that focus on addressing
implementation challenges and enhancing practitioner compe-
tencies (83 site visits in 2013 and 45 from January to May 2014);
and special interest conference calls focusing on issues identi-
ed by participating programs (for example, implementation in
rural settings) (ve calls from January to May 2014).
It is important to note that between LC meetings, par-
ticipants engage in follow-up work, including completion of
online training; submission of performance indicator data,
including caseload size and in-person employer contacts and
employment outcomes; and development and use of quality
improvement plans based on program-specic data.
Data collected in 2013 produced several ndings. In 2013,
a total of 49 (64%) of 77 eligible programs joined the ini-
tiative, and two dropped out before the end of the year. Thus
over 60% of eligible programs chose to join and remain with
the initiative throughout the year, and 96% of programs that
enrolled remained actively involved throughout 2013. By early
2014, 59 (69%) of 86 eligible programs had joined. Improved
IPS delity has been noted. For the 47 programs that began
in 2013 and remained in the LC, 46 had submitted self-
reported delity data, using the Supported Employment Fi-
delity Scale (13) adapted for NYS PROS programs. Fidelity
increased signicantly from January to September 2013 (from
91.3616.7 to 96.1614.7 out of a possible score of 125; p,.05).
(The adapted scale is available online at adobeformscentral.
In addition, the programs have achieved national IPS
benchmarks. In November 2013, CPI revised the LC perfor-
mance indicators, including denitions of key concepts, to be
more consistent with indicators collected in national stud-
ies (indicators are available at
?f5DlAWcIQ1sQVHnkdHaENlfw). For 41 of the 47 programs
that used the revised indicators to provide data for November
2013, we found that IPS caseload per FTE staff (calculated by
dividing the total caseload across the 41 programs by total FTE
staff at those programs) achieved expected national standards:
19.6 consumers receiving IPS services per one FTE. The state-
wide average rate of employment of consumers on the IPS
caseload (calculated at the program level and then averaged
across the state) is within the range reported nationally for ef-
fective IPS programs: 48% over the past month and 44% over
the past three months worked at least one day of competitive
employment. There were 1,045 individuals statewide on the
past-month caseload and 1,596 different individuals on the
caseload over the past three months. These data do not repre-
sent a xed cohort because the IPS caseload uctuates monthly.
Data collected in 2013 indicated one serious limitation
and some continuing challenges. Programs reported a mean
of 1.3 FTE employment staff per PROS program (median51.0,
mode51.0). The overall program census per program ranged
from 41 to 373. The limited availability of employment staff
suggests that a very small number of program participants
can receive IPS services. This has a dramatic impact on the
absolute numbers of program participants who will nd
competitive employment. Across all PROS programs, the
percentage of individuals enrolled on the last day of the year
who were competitively employed increased from 8.6% in
2012 to 9.5% in 2013. Although the trend is in a positive di-
rection, the low number of employment staff severely limits
the achievement of better employment outcomes. In addition,
difculties with specic aspects of delity have been noted.
The September 2013 delity self-assessment identied, on
average across programs, difculties with staff assignments,
including the number of staff assigned to employment duties;
the percentage of time spent in the community providing IPS
services, including job development; and expectations in re-
gard to staff completion of tasks not related to employment.
CPI and OMH are examining ways in which OMH can
strengthen or clarify policies, expectations, monitoring systems,
Psychiatric Services 66:1, January 2015 5
and incentives for achieving better employment outcomes and
increasing program involvement in CPIs IPS initiative. OMH
has looked for opportunities to communicate a clear expectation
that supported employment services are a crucial component of
PROS programs. This communication occurs in meetings and
conference calls with program leaders. In addition, OMH issued
a guidance document that details the manner in which IPS
services can and should be incorporated into PROS program
design. OMH monitors program involvement in the IPS ini-
tiative through detailed monthly reports provided by CPI.
OMH monitoring of implementation and employment outcomes
occurs in several ways: through delity and performance in-
dicator data provided by programs to CPI, through data pro-
vided by programs directly to OMH, and through program
licensing visits, which now focus more fully on IPS activities
and employment outcomes.
OMH is also committed to aligning incentives with de-
sired outcomes. It provides PROS programs with state funds
to assist with IPS implementation. OMH is examining ways
in which these funds can be made contingent on involve-
ment with CPIs IPS initiative, IPS delity, and employment
outcomes. In addition, through the national Ticket to Work
program, programs can now receive funds from the federal
government that are contingent on employment outcomes.
OMH has designed an administrative system that will fa-
cilitate programsaccess to this important incentive.
CPI and OMH are working with program leaders to nd
creative ways to increase staff involvement in employment
activities. Because low current overall employment stafng
levels are likely to continue, employment must become a sig-
nicant focus of other (nonemployment) staff members. CPI
plans to build on the IPS foundation now in place and to look
at creative adaptations and facilitators that incorporate IPS
principles into employment practices designed specically for
the PROS environment. One such facilitator is the Employ-
ment Resource Book (14), which was developed by CPI and
funded in part by the Employment Development Initiative of
the Substance Abuse and Mental Health Services Adminis-
tration and by the National Association of State Mental Health
Program Directors. Informed by IPS principles, the book is
designed for use by consumers on their own as well as with
employment staff members, other practitioners, and peer spe-
cialists. The books 32 topic areas and ten appendices cover
three critical time periods: before the job search, during the
job search, and after getting a job. Information, personalized
activities, and next steps in each topic area are described. We
hope that this resource will inspire and empower consumers
to pursue employment and that it will increase staff involve-
ment with employment activities.
Reports by participating programs suggest that a state mental
health authority (OMH) working closely with a training center
(CPI) can indeed move the bartoward dissemination of IPS
across a large state, resulting in improved delity and out-
comes that are consistent with national benchmarks. The self-
reported nature of these data, however, limits our ability to
reach rm conclusions. Structural and scal barriers have
resulted in difculties with implementation for some pro-
grams (such as the small number of staff assigned to employ-
ment duties and the nature of their assignments). We have
learned that planning efforts must go well beyond training to
include a focus on these barriers.
Dr. Margolies, Mr. Scannevin, and Dr. Dixon are with New York State
Psychiatric Institute and the Department of Psychiatry, Columbia Univer-
sity, New York City (e-mail: Ms. Broadway-
Wilson, Dr. Jewell, Ms. Chiang, and Dr. Marino are with New York State
Psychiatric Institute. Mr. Gregory is with the Department of Psychiatry,
Columbia University. Dr. Myers, Mr. Fernandez, Mr. Ruderman, and
Mr. McNabb are with the New York State Ofce of Mental Health, Albany.
Marcela Horvitz-Lennon, M.D., is editor of this column.
CPI is funded by the NYS OMH. The authors are grateful for the guid-
ance and support provided by Susan Essock, Ph.D.
The authors report no nancial relationships with commercial interests.
1. Covell NH, Margolies PJ, Myers RW, et al: Scaling up evidence-
based behavioral health care practices in New York State. Psy-
chiatric Services 65:713715, 2014
2. Bond GR, Becker DR, Drake RE, et al: Implementing supported
employment as an evidence-based practice. Psychiatric Services 52:
313322, 2001
3. Drake RE, Skinner JS, Bond GR, et al: Social Security and mental
illness: reducing disability with supported employment. Health
Affairs 28:761770, 2009
4. Drake RE, Bond GR, Becker DR: Individual Placement and Sup-
port: An Evidence-Based Approach to Supported Employment.
New York, Oxford University Press, 2012
5. Becker DR, Drake RE, Bond GR: Benchmark outcomes in sup-
ported employment. American Journal of Psychiatric Rehabilitation
14:230236, 2011
6. Becker DR, Drake RE, Bond GR: Corrigendum. American Journal
of Psychiatric Rehabilitation 15:38, 2012
7. Swanson SJ, Becker DR: IPS Supported Employment: A Practical
Guide. Lebanon, NH, Dartmouth Psychiatric Research Center,
8. The Breakthrough Series: IHIs Collaborative Model for Achieving
Breakthrough Improvement. IHI Innovation Series White Paper.
Cambridge, Mass, Institute for Healthcare Improvement, 2003
9. ØVretveit J, Bate P, Cleary P, et al: Quality collaboratives: lessons
from research. Quality and Safety in Health Care 11:345351, 2002
10. Schouten LM, Hulscher ME, van Everdingen JJ, et al: Evidence for
the impact of quality improvement collaboratives: systematic re-
view. British Journal of Medicine 336:14911494, 2008
11. Nadeem E, Olin S, Hill LC, et al: A literature review of learning
collaboratives in mental health care: used but untested. Psychiatric
Services (Epub ahead of print, Sept 1, 2014)
12. Becker DR, Drake RE, Bond GR: The IPS supported employment
learning collaborative. Psychiatric Rehabilitation Journal 37:7985, 2014
13. Becker DR, Swanson SJ, Bond GR, et al: Supported Employment
Fidelity Scale. Lebanon, NH, Dartmouth Psychiatric Research Cen-
ter, 2008. Available at
14. Jewell T, Margolies P, Salerno A, et al: Employment Resource Book.
New York, Research Foundation for Mental Hygiene, 2014 Psychiatric Services 66:1, January 2015
... The New York State Office of Mental Health (NYS-OMH), the New York State Psychiatric Institute and the Department of Psychiatry at Columbia University established CPI in 2007 to promote the widespread use of EBPs developed for adults with serious mental illness throughout New York State (NYS). The Center for Practice Innovations provides training and technical assistance (TA) across eight core recovery-oriented EBPs including: treating co-occurring mental health and substance use disorders, Assertive Community Treatment, 2021, supported employment, Individual Placement and Support [IPS], coordinated specialty care for first-episode psychosis, suicide prevention, cognitive health, and evidence-based care for obsessive-compulsive disorder (Covell et al., 2021;Covell et al., 2014;Dixon & Patel, 2020;Margolies et al., 2021b;Margolies et al., 2015;Margolies et al., 2021a;New York State, 2020;Thorning & Dixon, 2020). The Center for Practice Innovations also supports training in clinical core competencies and state training initiatives to support Medicaid redesign. ...
... Over the past decade, IPS has provided learning collaborative-related webinars and remote meetings for the Individual Placement and Support model of supported employment to programs across NYS including community psychiatric rehabilitation programs and state facility clinics (Margolies et al., 2015). During the period from March 2020-December 2021, a total of 29 collaborative-related webinars and remote meetings were provided, similar to previous years (*redacted 2019). ...
Full-text available
Intermediary and purveyor organizations (IPOs) play a key role in disseminating and implementing behavioral health evidence-based practices. The COVID-19 pandemic created a time of crisis and disruption to behavioral health care delivery. Using the conceptual framework of basic, targeted, and intensive technical assistance (TA) from the Training and Technology Transfer Centers, case studies are used to describe how programs at *redacted,* a state funded-intermediary organization, adapted its training and technical assistance to be delivered entirely remotely, to include content related to COVID-19 and to provide guidance on telehealth-based behavioral health care.
... In addition, learners received the e-learning modules favorably, rating them highly overall and noting that they met stated learning objectives and presented new information. Throughout the development process, data from the e-learning modules were described using the CFIR to identify needs that led to additional e-learning modules as well as strategies for subsequent implementation supports through a learning collaborative statewide (27). ...
... Notably, only half of the providers who completed the evaluation noted an intention to change their practice, and we did not have the capacity to assess practice change at the individual provider level at this stage of IPS implementation (level 3). However, in our subsequent work (27), program fidelity assessments using established measures demonstrated improvement over time, suggesting that level 3 provider practice change and fidelity self-assessed by program sites are shown to be associated with higher employment rates (level 4), which are sustained over time (28). Future research may focus on more rigorous evaluation of knowledge, practice change, mixed-method assessment of how the content from e-learning modules influences practice, and the essential role of care recipients in helping to design training within implementation efforts. ...
Full-text available
Background Implementation science lacks a systematic approach to the development of learning strategies for online training in evidence-based practices (EBPs) that takes the context of real-world practice into account. The field of instructional design offers ecologically valid and systematic processes to develop learning strategies for workforce development and performance support.Objective This report describes the application of an instructional design framework—Analyze, Design, Develop, Implement, and Evaluate (ADDIE) model—in the development and evaluation of e-learning modules as one strategy among a multifaceted approach to the implementation of individual placement and support (IPS), a model of supported employment for community behavioral health treatment programs, in New York State.Methods We applied quantitative and qualitative methods to develop and evaluate three IPS e-learning modules. Throughout the ADDIE process, we conducted formative and summative evaluations and identified determinants of implementation using the Consolidated Framework for Implementation Research (CFIR). Formative evaluations consisted of qualitative feedback received from recipients and providers during early pilot work. The summative evaluation consisted of levels 1 and 2 (reaction to the training, self-reported knowledge, and practice change) quantitative and qualitative data and was guided by the Kirkpatrick model for training evaluation.ResultsFormative evaluation with key stakeholders identified a range of learning needs that informed the development of a pilot training program in IPS. Feedback on this pilot training program informed the design document of three e-learning modules on IPS: Introduction to IPS, IPS Job development, and Using the IPS Employment Resource Book. Each module was developed iteratively and provided an assessment of learning needs that informed successive modules. All modules were disseminated and evaluated through a learning management system. Summative evaluation revealed that learners rated the modules positively, and self-report of knowledge acquisition was high (mean range: 4.4–4.6 out of 5). About half of learners indicated that they would change their practice after watching the modules (range: 48–51%). All learners who completed the level 1 evaluation demonstrated 80% or better mastery of knowledge on the level 2 evaluation embedded in each module. The CFIR was used to identify implementation barriers and facilitators among the evaluation data which facilitated planning for subsequent implementation support activities in the IPS initiative.Conclusion Instructional design approaches such as ADDIE may offer implementation scientists and practitioners a flexible and systematic approach for the development of e-learning modules as a single component or one strategy in a multifaceted approach for training in EBPs.
... 4 While first developed in general health care, learning collaboratives, also called quality improvement collaboratives, have been used frequently in behavioral health by providers of training, technical assistance, and services as a strategy to promote quality improvement and implementation of a range of evidence-based practices (e.g., cognitive behavioral therapy for psychosis, integrated services for co-occurring mental health and substance use disorders, school mental health, supported employment, trauma informed care). [5][6][7][8][9][10][11][12] These collaboratives generally involve bringing together teams from different organizations and using experts to educate and coach them in a quality or implementation project and measure the effects. 13 Sharing of strategies, data, successes, and obstacles among participating teams is central to the approach. ...
Full-text available
Learning collaboratives are increasingly used in behavioral health. They generally involve bringing together teams from different organizations and using experts to educate and coach the teams in quality improvement, implementing evidence-based practices, and measuring the effects. Although learning collaboratives have demonstrated some effectiveness in general health care, the evidence is less clear in behavioral health and more rigorous studies are needed. Learning collaboratives may contain a range of elements, and which elements are included in any one learning collaborative varies widely; the unique contribution of each element has not been established. This commentary seeks to clarify the concept of a learning collaborative, highlight its common elements, review evidence of its effectiveness, identify its application in behavioral health, and highlight recommendations to guide technical assistance purveyors and behavioral health providers as they employ learning collaboratives to improve behavioral health access and quality.
... New York State has one of the largest behavioral healthcare workforces with about 100,000 providers working in over 6000 programs across the state. For the past decade, CPI has incorporated practical approaches in implementation science to increase the knowledge and change the practice of behavioral health providers in NYS, focusing on EBPs for adults diagnosed with serious mental illness (Nossel et al. 2018;Covell et al. 2011Covell et al. , 2014Covell et al. , 2015Covell et al. , 2016Margolies et al. 2015;Dixon and Patel 2020). These practices include supported employment, integrated treatment for people with co-occurring mental health and substance use conditions, assertive community treatment, suicide prevention, early intervention for first episode psychosis, and wellness self-management. ...
Full-text available
Mental health authorities in several states, often working with academic partners, have played important roles in disseminating evidence-based practices (EBPs) for adults diagnosed with serious mental illness. This work has been facilitated by intermediary organizations that work directly with providers to implement EBPs. This report uses two case studies to describe how the Center for Practice Innovations (CPI), an intermediary organization, has used the Active Implementation Research Network’s nine implementation drivers to successfully implement EBPs across the large state of New York. One case study focuses on supported employment and the second on integrated treatment for co-occurring mental health and substance use conditions. We provide these case studies to illustrate how intermediary organizations can use implementation science to organize and select effective support strategies to disseminate and implement a range of EBPs within a state system.
... However, the findings of our study do not provide justification for making adaptations to the IPS model. Learning collaboratives foster a culture of collaboration between IPS programmes through the collection of data concerning programme implementation and outcomes, sharing of knowledge, provision of training and technical support as well as research and innovation (Becker et al., 2014;Margolies et al., 2015). ...
Full-text available
Introduction: we aimed to identify the barriers and facilitators to the implementation of a high fidelity IPS service in a community forensic mental health setting. Method: in-depth interviews were conducted with clinical staff (n=11), patients (n=3), and employers (n=5) to examine barriers and facilitators to implementation of a high fidelity IPS service. Data was analysed using thematic analysis, and themes were mapped onto IPS fidelity criteria. Results: barriers cited included competing interests between employment support and psychological therapies, perceived patients’ readiness for work, and concerns about the impact of returning to work on welfare benefits. Facilitators of implementation included clear communication of the benefits of IPS, interdisciplinary collaboration, and positive attitudes toward the support offered by the IPS programme among stakeholders. Offences, rather than mental health history was seen as a key issue from employers’ perspective. Employers regarded disclosure of offending or mental health history as important to develop trust and to gauge their own capacity to offer support. Conclusion: implementation of IPS in a community mental health forensic setting is complex and requires robust planning. Future studies should address the barriers identified and adaptations to the IPS model are needed to address difficulties encountered in forensic settings.
... Our work facilitating the implementation of IPS in programs across NYS using learning collaboratives has been described previously (13). For this study, 78 of the 98 community sites that joined the initiative in 2016 provided self-reported employment data and self-assessed fidelity data. ...
Objective: A growing body of literature demonstrates that high-fidelity implementation of the individual placement and support (IPS) model of supported employment increases the chances of achieving desired outcomes. This study examined the relationship between IPS fidelity, as self-reported by program sites, and employment outcomes and determined whether this relationship was maintained over time. Methods: A total of 78 outpatient programs in New York State provided data on self-reported IPS fidelity and employment outcomes. Pearson correlations were used to determine the relationship between fidelity scores and competitive employment rates. A mixed-effects model examined the relationship between repeated fidelity and employment measures over time. Results: A significant positive relationship was found between better self-reported IPS fidelity and greater employment. The relationship between IPS fidelity and employment was sustained over time (up to one year). Conclusions: Higher-fidelity implementation of the IPS model, as self-assessed by program sites, was associated with higher employment rates, which were sustained over time.
Objective: This study examined the relationship between individual placement and support (IPS) employment specialists' time spent in the community and employment outcomes in the current digital age, featuring increased technology use and online hiring practices. Methods: The authors examined the relationship between employment outcomes and IPS employment specialists' time spent in the community at 78 sites in 2018 and 84 sites in 2019. Results: The amount of time staff spent in the community was significantly and positively associated with better employment outcomes. Conclusions: These data support the continued importance of employment specialists' spending time in the community with employers and IPS recipients to achieve optimal outcomes for recipients.
The COVID-19 pandemic has had an enormous impact on the provision of behavioral health care services across the United States. This column examines this impact within the context of New York State's supported employment initiative, which involved 89 implementation sites before the start of the pandemic. The pandemic caused changes to the training and implementation supports provided, the number of sites providing these services, and the ways in which sites provided supported employment services. Although mean self-assessed implementation fidelity decreased modestly, employment outcomes that dipped early in the pandemic rebounded quickly to prepandemic levels.
Full-text available
Tobacco use in people with behavioral health conditions remain two to three times higher than the general population causing premature death and impacting recovery negatively across several domains. Intermediary organizations can provide practical tools, training, and technical assistance to help programs improve capacity to treat tobacco use. This report describes the construction and application of the Tobacco Integration Self-Evaluation Tool (TiSET) for behavioral health programs, a 20-item scale inspired by the DDCMHT and additional content from the Facility Tobacco Policy and Treatment Practices Self-Evaluation tool that one of the study authors (JW) used previously with addiction treatment programs. Completing the TiSET is an important step for behavioral health programs to evaluate their ability to effectively treat people that use tobacco. An important next step is to use those results to facilitate a quality improvement process. We include large agency example illustrating how the TiSET can be applied in real-world practice.
Full-text available
Objective: To examine the feasibility of conducting a fully powered randomized controlled trial (RCT) of Individual Placement and Support (IPS). IPS is a form of supported employment which aims to put people into open employment quickly and in accordance with their preferences. It is delivered by employment specialists collocated within clinical teams, and provides time unlimited support for the individual and their employer, along with welfare benefits counselling. Method: A feasibility cluster RCT of treatment as usual (TAU) plus IPS versus TAU alone was conducted over 12 months among patients with offending histories in a community forensic setting in the UK. The feasibility criteria were to achieve 50% recruitment rate; 50% completion rate for IPS; 50% completion rate of all outcome measures; and 80% acceptability rating for IPS. The primary efficacy outcome was the proportion of people in open employment at 12 months. The secondary outcomes were other vocational and educational activities; Brief Psychiatric Rating Scale; Rosenberg’s Self-esteem Scale; Client Service Receipt Inventory; quality of life using the SF12-v2 and EQ5-D3; Social Functioning Questionnaire; Work Limitation Questionnaire; and reoffending. Results: Participants’ mean age was 39.2 years. The majority were male (88.9), White British (72.2), and single (72.2%). Over 72% had no higher qualification beyond secondary education; mean years in education was 10.4. Over one third had schizophrenia, one fifth had depression, and the rest had personality disorder as their primary diagnosis. Participants had a lifetime average of 7.5 convictions for 15.5 offences. The recruitment rate of all referrals was 38.3% (IPS n = 11; TAU n = 7). Completion rate for IPS was 54.5, with 45.5% acceptability rating. Completion rates for outcome measures for the groups at baseline and 12 months ranged from 22.2 to 100%. The proportion of people in open employment at 12 months were 9.1 and 0% for IPS and TAU respectively. Conclusion: It is not feasible to conduct a full RCT of IPS in community forensic settings in the UK owing to recruitment and retention difficulties. Conducting a trial of this kind requires a large pool of patients from multiple sites and longer IPS implementation and recruitment periods than those of this study. Clinical Trial Registration:, identifier NCT02442193.
Full-text available
This column describes the Center for Practice Innovations (CPI), which was created in 2007 by the New York State Office of Mental Health and the Department of Psychiatry at Columbia University. CPI uses innovative approaches to build stakeholder collaborations, develop and maintain practitioners' expertise, and build agency infrastructures that support implementing and sustaining evidence-based practices. CPI's five core initiatives provide training in co-occurring mental and substance use disorders, assertive community treatment, supported employment and education, wellness self-management, and treatment of first-episode psychosis. Central to CPI's activities are award-winning training modules, statewide learning collaboratives, and use of a learning management system.
Full-text available
Objective: Policy makers have increasingly turned to learning collaboratives (LCs) as a strategy for improving usual care through the dissemination of evidence-based practices. The purpose of this review was to characterize the state of the evidence for use of LCs in mental health care. Methods: A systematic search of major academic databases for peer-reviewed articles on LCs in mental health care generated 421 unique articles across a range of disciplines; 28 mental health articles were selected for full-text review, and 20 articles representing 16 distinct studies met criteria for final inclusion. Articles were coded to identify the LC components reported, the focus of the research, and key findings. Results: Most of the articles included assessments of provider- or patient-level variables at baseline and post-LC. Only one study included a comparison condition. LC targets ranged widely, from use of a depression screening tool to implementation of evidence-based treatments. Fourteen crosscutting LC components (for example, in-person learning sessions, phone meetings, data reporting, leadership involvement, and training in quality improvement methods) were identified. The LCs reviewed reported including, on average, seven components, most commonly in-person learning sessions, plan-do-study-act cycles, multidisciplinary quality improvement teams, and data collection for quality improvement. Conclusions: LCs are being used widely in mental health care, although there is minimal evidence of their effectiveness and unclear reporting in regard to specific components. Rigorous observational and controlled research studies on the impact of LCs on targeted provider- and patient-level outcomes are greatly needed.
Full-text available
Social Security Administration disability programs are expensive, growing, and headed toward bankruptcy. People with psychiatric disabilities now constitute the largest and most rapidly expanding subgroup of program beneficiaries. Evidence-based supported employment is a well-defined, rigorously tested service model that helps people with psychiatric disabilities obtain and succeed in competitive employment. Providing evidence-based supported employment and mental health services to this population could reduce the growing rates of disability and enable those already disabled to contribute positively to the workforce and to their own welfare, at little or no cost (and, depending on assumptions, a possible savings) to the government.
Full-text available
Supported employment for people with severe mental illness is an evidence-based practice, based on converging findings from eight randomized controlled trials and three quasi-experimental studies. The critical ingredients of supported employment have been well described, and a fidelity scale differentiates supported employment programs from other types of vocational services. The effectiveness of supported employment appears to be generalizable across a broad range of client characteristics and community settings. More research is needed on long-term outcomes and on cost-effectiveness. Access to supported employment programs remains a problem, despite their increasing use throughout the United States. The authors discuss barriers to implementation and strategies for overcoming them based on successful experiences in several states.
Full-text available
Quality improvement collaboratives are increasingly being used in many countries to achieve rapid improvements in health care. However, there is little independent evidence that they are more cost effective than other methods, and little knowledge about how they could be made more effective. A number of systematic evaluations are being performed by researchers in North America, the UK, and Sweden. This paper presents the shared ideas from two meetings of these researchers. The evidence to date is that some collaboratives have stimulated improvements in patient care and organisational performance, but there are significant differences between collaboratives and teams. The researchers agreed on the possible reasons why some were less successful than others, and identified 10 challenges which organisers and teams need to address to achieve improvement. In the absence of more conclusive evidence, these guidelines are likely to be useful for collaborative organisers, teams and their managers and may also contribute to further research into collaboratives and the spread of innovations in health care.
Full-text available
To evaluate the effectiveness of quality improvement collaboratives in improving the quality of care. Relevant studies through Medline, Embase, PsycINFO, CINAHL, and Cochrane databases. Two reviewers independently extracted data on topics, participants, setting, study design, and outcomes. Of 1104 articles identified, 72 were included in the study. Twelve reports representing nine studies (including two randomised controlled trials) used a controlled design to measure the effects of the quality improvement collaborative intervention on care processes or outcomes of care. Systematic review of these nine studies showed moderate positive results. Seven studies (including one randomised controlled trial) reported an effect on some of the selected outcome measures. Two studies (including one randomised controlled trial) did not show any significant effect. The evidence underlying quality improvement collaboratives is positive but limited and the effects cannot be predicted with great certainty. Considering that quality improvement collaboratives seem to play a key part in current strategies focused on accelerating improvement, but may have only modest effects on outcomes at best, further knowledge of the basic components effectiveness, cost effectiveness, and success factors is crucial to determine the value of quality improvement collaboratives.
Employment is the highest priority for many people with severe mental illness and it is a central aspect of recovery. Over the past two decades, the Individual Placement and Support (IPS) model of supported employment has emerged as the prominent evidence-based approach to vocational rehabilitation. This book synthesizes the research and experience on IPS supported employment: historical context, core principles, effectiveness, long-term outcomes, non-vocational outcomes, cost-effectiveness, generalizability, fidelity, implementation, policy, and future research. This book relates to areas of work with populations with psychiatric disabilities and in community mental health and social service settings. In tracing the evolution of IPS, readers are equipped with an elegant example of the transition from needs assessment, to model development, to testing, and to dissemination.
Objective: Learning collaboratives aim to improve the quality and outcomes of health care. This paper updates the Johnson & Johnson-Dartmouth Community Mental Health Program, a 12-year learning collaborative on supported employment for people with mental illness. Methods: We gathered data from quarterly employment reports, monthly Individual Placement and Support (IPS) meetings, and presentations at the 2013 annual meeting of the learning collaborative. Results: The number of participant states or regions (and sites within these jurisdictions) was expanded to 16 jurisdictions in the United States and 3 in European countries. The quarterly rate of competitive employment has averaged 43% over 11 years in the U.S. sites. The collaborative has spawned numerous interactions, trainings, innovations, and research projects. Conclusions and implications for practice: Long-term learning collaboratives can produce high quality services, good outcomes, sustainability, and innovation.
Mental health program leaders need vocational benchmarks for client outcomes. Outcomes from randomized controlled trials may not generalize to routine practice settings. This paper describes quarterly employment outcomes from a national learning collaborative on supported employment for people with serious mental illnesses. Benchmarks were established using the 25th, 50th, and 75th percentiles. Based on these percentiles, we recommend that state and local mental health and vocational rehabilitation administrators use a minimal benchmark of 33%, a good performance benchmark of 45%, and a high-performance benchmark of 57% for expected rates of competitive employment among active clients who receive evidence-based vocational services.
Supported Employment Fidelity Scale
  • D R Becker
  • S J Swanson
  • G R Bond
Becker DR, Swanson SJ, Bond GR, et al: Supported Employment Fidelity Scale. Lebanon, NH, Dartmouth Psychiatric Research Center, 2008. Available at