Content uploaded by Lisa Dixon
Author content
All content in this area was uploaded by Lisa Dixon on Feb 05, 2018
Content may be subject to copyright.
Use of Learning Collaboratives by the Center
for Practice Innovations to Bring IPS to Scale in
New York State
Paul J. Margolies, Ph.D., Karen Broadway-Wilson, Raymond Gregory, Thomas C. Jewell, Ph.D.,
Gary Scannevin, Jr., M.P.S., C.P.R.P., Robert W. Myers, Ph.D., Henry A. Fernandez, J.D., Douglas Ruderman, L.C.S.W.,
Liam McNabb, I-Chin Chiang, M.S., Leslie Marino, M.D., Lisa B. Dixon, M.D., M.P.H.
This column focuses on use of learning collaboratives by the
Center for Practice Innovations to help programs implement
the evidence-based individual placement and support model
of supported employment in New York State. These learning
collaboratives use fidelity and performance indicator data
to drive the development of program-specific individualized
quality improvement plans. As of 2014, 59 (69%) of 86 eligible
programs have joined the initiative. Programs are achieving
employment outcomes for consumers on par with national
benchmarks, along with improved fidelity.
Psychiatric Services 2015; 66:4–6; doi: 10.1176/appi.ps.201400383
The Center for Practice Innovations (CPI) is funded by the
New York State Office of Mental Health (OMH) to bring best
practices to mental health programs across the state (1). Large
systems face the challenge of helping practitioners in diverse
programs adopt evidence-based practices. This challenge in-
cludes reaching all programs, regardless of whether they would
be naturally inclined to participate. Because of the large number
of programs and the large geographic area, New York State
(NYS) cannot afford traditional approaches to dissemination,
such as use of consultant trainers for individual agency training.
To encourage community programs to adopt best practices,
OMH developed a program model in 2004 that set a clear ex-
pectation in regard to the implementation of recovery-oriented
evidence-based practices. Through funding policies, OMH
provided incentives for adoption of these practices. These
Personalized Recovery Oriented Services (PROS) programs
are designed for adults living in the community who have
a diagnosis of serious mental illness. The programs help them
identify and achieve personally meaningful recovery goals.
The number of PROS programs has increased significantly:
in early 2014, 86 programs were serving 10,600 individuals.
OMH leadership identified supported employment as a
key service in PROS programs. Rates of competitive employ-
ment of mental health consumers have historically been very
low. PROS programs across NYS routinely report competitive
employment rates of less than 10% among program partic-
ipants, similar to rates in other states (2,3).
IPS AND LCs
Individual placement and support (IPS) has been recognized
as the evidence-based approach to supported employment
for more than a decade (2,4). A study that focused on 127
sites involved in the Johnson & Johnson–Dartmouth Com-
munity Mental Health Program found that the mean quarterly
employment rate for individuals receiving IPS services over
eight years was 41% (5,6). Employment was defined as work-
ing in a competitive job for one day or more in the quarter. IPS
principles include zero exclusion, integration of employment
and mental health services, competitive employment as the
goal, and rapid job search. Key services include developing
a meaningful employment plan that reflects the consumer’s
wishes,aswellasjobdevelopmentandjobsupports(7).
Learning collaboratives (LCs) have been used in health
care and are increasingly used in behavioral health care. LCs
were initially patterned on the Institute for Healthcare Im-
provement Breakthrough Series (8), and variations have been
developed (9). Fundamental elements include a number of
organizations working together, using quality improvement
methods to close the gap between potential and actual per-
formance, learning from experts as well as from one another,
and using data to track performance (8). Reviews have re-
ported promising results in regard to the impact of LCs, but
they are cautious in reaching firm conclusions because of
difficulties comparing studies and other methodological issues
(9–11).
The NYS initiative builds on the LC model designed by
the team that developed IPS with the goals of guiding and
supporting IPS dissemination in 12 states and the District of
Columbia (12). The Johnson & Johnson–Dartmouth national
LC used a two-tiered approach. IPS researchers used LCs
to work with mental health and vocational rehabilitation
leaders at the state level, and these leaders in turn worked
with programs in their states. The LC approach has helped
4ps.psychiatryonline.org Psychiatric Services 66:1, January 2015
BEST PRACTICES
states sustain and expand IPS, improving quality and achieving
good outcomes. The NYS LC initiative works directly with
large numbers of programs.
CPI’S DISSEMINATION OF IPS
In 2013, CPI worked with 47 of 77 PROS programs across
NYS that were clustered into four regional LCs. In 2014, the
number of participating programs increased to 59 of 86,
clustered into seven regional LCs. OMH strongly encouraged
participation, but no fiscal or other incentives were provided.
Supported employment is funded through a combination of
Medicaid and state dollars. CPI staff (3.2 full-time-equivalent
[FTE] trainers-consultants) helped these programs by leading
LC meetings, providing guidance concerning the develop-
ment of quality improvement plans, collecting and analyzing
data submitted by the programs, leading informational webinars
and conference calls, and providing onsite program-specific
training and consultation. The LCs provide both face-to-face
and online training and support activities. Programs agree to
attend LC meetings, conduct periodic fidelity self-assessments,
submit monthly performance indicator data, develop and use
IPS quality improvement plans, and share experiences with
one another.
LCs offer the following training and support activities:
two online training modules, one providing an overview of
IPS and the other focusing on the practitioner skill of job
development; statewide webinar and regional online meet-
ings focusing on important topics (for example, IPS fidelity
and supervision) and processes (for example, using data to
drive continuous quality improvement efforts) (in 2013, one
webinar and 24 online meetings were conducted, and from
January to May 2014, four webinars and eight online meetings
were conducted); an IPS library, available through CPI’s
learning management system, that provides archived webinars,
presentations from past training events, and tools to help IPS
implementation; regional face-to-face meetings where pro-
grams learn from each other (eight meetings in 2013 and 15 from
January to May 2014); individualized consultations, both at
the program site and by telephone that focus on addressing
implementation challenges and enhancing practitioner compe-
tencies (83 site visits in 2013 and 45 from January to May 2014);
and special interest conference calls focusing on issues identi-
fied by participating programs (for example, implementation in
rural settings) (five calls from January to May 2014).
It is important to note that between LC meetings, par-
ticipants engage in follow-up work, including completion of
online training; submission of performance indicator data,
including caseload size and in-person employer contacts and
employment outcomes; and development and use of quality
improvement plans based on program-specific data.
Data collected in 2013 produced several findings. In 2013,
a total of 49 (64%) of 77 eligible programs joined the ini-
tiative, and two dropped out before the end of the year. Thus
over 60% of eligible programs chose to join and remain with
the initiative throughout the year, and 96% of programs that
enrolled remained actively involved throughout 2013. By early
2014, 59 (69%) of 86 eligible programs had joined. Improved
IPS fidelity has been noted. For the 47 programs that began
in 2013 and remained in the LC, 46 had submitted self-
reported fidelity data, using the Supported Employment Fi-
delity Scale (13) adapted for NYS PROS programs. Fidelity
increased significantly from January to September 2013 (from
91.3616.7 to 96.1614.7 out of a possible score of 125; p,.05).
(The adapted scale is available online at adobeformscentral.
com/?f5zO40AePRbfe73yRJIGhRZg.)
In addition, the programs have achieved national IPS
benchmarks. In November 2013, CPI revised the LC perfor-
mance indicators, including definitions of key concepts, to be
more consistent with indicators collected in national stud-
ies (indicators are available at adobeformscentral.com/
?f5DlAWcIQ1sQVHnkdHaENlfw). For 41 of the 47 programs
that used the revised indicators to provide data for November
2013, we found that IPS caseload per FTE staff (calculated by
dividing the total caseload across the 41 programs by total FTE
staff at those programs) achieved expected national standards:
19.6 consumers receiving IPS services per one FTE. The state-
wide average rate of employment of consumers on the IPS
caseload (calculated at the program level and then averaged
across the state) is within the range reported nationally for ef-
fective IPS programs: 48% over the past month and 44% over
the past three months worked at least one day of competitive
employment. There were 1,045 individuals statewide on the
past-month caseload and 1,596 different individuals on the
caseload over the past three months. These data do not repre-
sent a fixed cohort because the IPS caseload fluctuates monthly.
Data collected in 2013 indicated one serious limitation
and some continuing challenges. Programs reported a mean
of 1.3 FTE employment staff per PROS program (median51.0,
mode51.0). The overall program census per program ranged
from 41 to 373. The limited availability of employment staff
suggests that a very small number of program participants
can receive IPS services. This has a dramatic impact on the
absolute numbers of program participants who will find
competitive employment. Across all PROS programs, the
percentage of individuals enrolled on the last day of the year
who were competitively employed increased from 8.6% in
2012 to 9.5% in 2013. Although the trend is in a positive di-
rection, the low number of employment staff severely limits
the achievement of better employment outcomes. In addition,
difficulties with specific aspects of fidelity have been noted.
The September 2013 fidelity self-assessment identified, on
average across programs, difficulties with staff assignments,
including the number of staff assigned to employment duties;
the percentage of time spent in the community providing IPS
services, including job development; and expectations in re-
gard to staff completion of tasks not related to employment.
PROMOTING SUSTAINABILITY AND HIGH FIDELITY
CPI and OMH are examining ways in which OMH can
strengthen or clarify policies, expectations, monitoring systems,
Psychiatric Services 66:1, January 2015 ps.psychiatryonline.org 5
BEST PRACTICES
and incentives for achieving better employment outcomes and
increasing program involvement in CPI’s IPS initiative. OMH
has looked for opportunities to communicate a clear expectation
that supported employment services are a crucial component of
PROS programs. This communication occurs in meetings and
conference calls with program leaders. In addition, OMH issued
a guidance document that details the manner in which IPS
services can and should be incorporated into PROS program
design. OMH monitors program involvement in the IPS ini-
tiative through detailed monthly reports provided by CPI.
OMH monitoring of implementation and employment outcomes
occurs in several ways: through fidelity and performance in-
dicator data provided by programs to CPI, through data pro-
vided by programs directly to OMH, and through program
licensing visits, which now focus more fully on IPS activities
and employment outcomes.
OMH is also committed to aligning incentives with de-
sired outcomes. It provides PROS programs with state funds
to assist with IPS implementation. OMH is examining ways
in which these funds can be made contingent on involve-
ment with CPI’s IPS initiative, IPS fidelity, and employment
outcomes. In addition, through the national Ticket to Work
program, programs can now receive funds from the federal
government that are contingent on employment outcomes.
OMH has designed an administrative system that will fa-
cilitate programs’access to this important incentive.
CREATIVE PLANNING FOR THE FUTURE
CPI and OMH are working with program leaders to find
creative ways to increase staff involvement in employment
activities. Because low current overall employment staffing
levels are likely to continue, employment must become a sig-
nificant focus of other (nonemployment) staff members. CPI
plans to build on the IPS foundation now in place and to look
at creative adaptations and facilitators that incorporate IPS
principles into employment practices designed specifically for
the PROS environment. One such facilitator is the Employ-
ment Resource Book (14), which was developed by CPI and
funded in part by the Employment Development Initiative of
the Substance Abuse and Mental Health Services Adminis-
tration and by the National Association of State Mental Health
Program Directors. Informed by IPS principles, the book is
designed for use by consumers on their own as well as with
employment staff members, other practitioners, and peer spe-
cialists. The book’s 32 topic areas and ten appendices cover
three critical time periods: before the job search, during the
job search, and after getting a job. Information, personalized
activities, and next steps in each topic area are described. We
hope that this resource will inspire and empower consumers
to pursue employment and that it will increase staff involve-
ment with employment activities.
CONCLUSIONS
Reports by participating programs suggest that a state mental
health authority (OMH) working closely with a training center
(CPI) can indeed “move the bar”toward dissemination of IPS
across a large state, resulting in improved fidelity and out-
comes that are consistent with national benchmarks. The self-
reported nature of these data, however, limits our ability to
reach firm conclusions. Structural and fiscal barriers have
resulted in difficulties with implementation for some pro-
grams (such as the small number of staff assigned to employ-
ment duties and the nature of their assignments). We have
learned that planning efforts must go well beyond training to
include a focus on these barriers.
AUTHOR AND ARTICLE INFORMATION
Dr. Margolies, Mr. Scannevin, and Dr. Dixon are with New York State
Psychiatric Institute and the Department of Psychiatry, Columbia Univer-
sity, New York City (e-mail: margoli@nyspi.columbia.edu). Ms. Broadway-
Wilson, Dr. Jewell, Ms. Chiang, and Dr. Marino are with New York State
Psychiatric Institute. Mr. Gregory is with the Department of Psychiatry,
Columbia University. Dr. Myers, Mr. Fernandez, Mr. Ruderman, and
Mr. McNabb are with the New York State Office of Mental Health, Albany.
Marcela Horvitz-Lennon, M.D., is editor of this column.
CPI is funded by the NYS OMH. The authors are grateful for the guid-
ance and support provided by Susan Essock, Ph.D.
The authors report no financial relationships with commercial interests.
REFERENCES
1. Covell NH, Margolies PJ, Myers RW, et al: Scaling up evidence-
based behavioral health care practices in New York State. Psy-
chiatric Services 65:713–715, 2014
2. Bond GR, Becker DR, Drake RE, et al: Implementing supported
employment as an evidence-based practice. Psychiatric Services 52:
313–322, 2001
3. Drake RE, Skinner JS, Bond GR, et al: Social Security and mental
illness: reducing disability with supported employment. Health
Affairs 28:761–770, 2009
4. Drake RE, Bond GR, Becker DR: Individual Placement and Sup-
port: An Evidence-Based Approach to Supported Employment.
New York, Oxford University Press, 2012
5. Becker DR, Drake RE, Bond GR: Benchmark outcomes in sup-
ported employment. American Journal of Psychiatric Rehabilitation
14:230–236, 2011
6. Becker DR, Drake RE, Bond GR: Corrigendum. American Journal
of Psychiatric Rehabilitation 15:38, 2012
7. Swanson SJ, Becker DR: IPS Supported Employment: A Practical
Guide. Lebanon, NH, Dartmouth Psychiatric Research Center,
2013
8. The Breakthrough Series: IHI’s Collaborative Model for Achieving
Breakthrough Improvement. IHI Innovation Series White Paper.
Cambridge, Mass, Institute for Healthcare Improvement, 2003
9. ØVretveit J, Bate P, Cleary P, et al: Quality collaboratives: lessons
from research. Quality and Safety in Health Care 11:345–351, 2002
10. Schouten LM, Hulscher ME, van Everdingen JJ, et al: Evidence for
the impact of quality improvement collaboratives: systematic re-
view. British Journal of Medicine 336:1491–1494, 2008
11. Nadeem E, Olin S, Hill LC, et al: A literature review of learning
collaboratives in mental health care: used but untested. Psychiatric
Services (Epub ahead of print, Sept 1, 2014)
12. Becker DR, Drake RE, Bond GR: The IPS supported employment
learning collaborative. Psychiatric Rehabilitation Journal 37:79–85, 2014
13. Becker DR, Swanson SJ, Bond GR, et al: Supported Employment
Fidelity Scale. Lebanon, NH, Dartmouth Psychiatric Research Cen-
ter, 2008. Available at dms.dartmouth.edu/prc/employment/
14. Jewell T, Margolies P, Salerno A, et al: Employment Resource Book.
New York, Research Foundation for Mental Hygiene, 2014
6ps.psychiatryonline.org Psychiatric Services 66:1, January 2015
BEST PRACTICES
