Defending and managing the pipeline: Lessons for running a randomized experiment in a correctional institution

Abstract

Objectives
To discuss the challenges faced in an experimental prisoner reentry evaluation with regard to managing the pipeline of eligible cases.

Methods
This paper uses a case study approach, coupled with a review of the relevant literature on issues of case flow in experimental studies in criminal justice settings. Included are recommendations for researchers on the management of case flow, reflections on the major research design issues encountered, and a listing of dilemmas that are likely to plague experimental evaluations of prisoner reentry programs.

Results
Particularly in a jail setting, anticipating the timing of release of a prisoner to the community is probably impossible given the large number of issues that impact release, many of which will be unanticipated. A detailed pipeline study is critical to the success of an experimental study targeting returning prisoners. Pipeline studies should be conducted under what will be the true conditions and context for enrollment, given all eligibility criteria.

Conclusions
With continued and systematic documentation of enrollment challenges in future experimental evaluations of reentry programs, as well as other experimental evaluations that involve individuals, academics can build a deep literature that would help facilitate future successful randomized experiments in the criminal justice field.

Full text

Defending and managing the pipeline: lessons
for running a randomized experiment
in a correctional institution
Caterina G. Roman &Jocelyn Fontaine &
John Fallon &Jacquelyn Anderson &Corinne Rearer
Published online: 22 May 2012
#Springer Science+Business Media B.V. 2012
Abstract
Objectives To discuss the challenges faced in an experimental prisoner reentry
evaluation with regard to managing the pipeline of eligible cases.
Methods This paper uses a case study approach, coupled with a review of the relevant
literature on issues of case flow in experimental studies in criminal justice settings.
Included are recommendations for researchers on the management of case flow,
reflections on the major research design issues encountered, and a listing of dilemmas
that are likely to plague experimental evaluations of prisoner reentry programs.
Results Particularly in a jail setting, anticipating the timing of release of a prisoner to
the community is probably impossible given the large number of issues that impact
release, many of which will be unanticipated. A detailed pipeline study is critical to
the success of an experimental study targeting returning prisoners. Pipeline studies
should be conducted under what will be the true conditions and context for enroll-
ment, given all eligibility criteria.
Conclusions With continued and systematic documentation of enrollment challenges
in future experimental evaluations of reentry programs, as well as other experimental
J Exp Criminol (2012) 8:307–329
DOI 10.1007/s11292-012-9155-y
C. G. Roman (*)
Department of Criminal Justice, Temple University, 1115 Polett Walk, Gladfelter Hall, 5th Floor,
Philadelphia, PA 19010, USA
e-mail: croman@temple.edu
J. Fontaine
The Urban Institute, Washington, DC, USA
J. Fallon
Corporation for Supportive Housing, Chicago, IL, USA
J. Anderson
Corporation for Supportive Housing, Oakland, CA, USA
C. Rearer
Trilogy, Inc., Chicago, IL, USA

evaluations that involve individuals, academics can build a deep literature that would
help facilitate future successful randomized experiments in the criminal justice field.
Keywords Challenges .Evaluation .Experimental design .Permanent supportive
housing .Prisoner reentry
Undertaking an experimental evaluation in criminal justice is ambitious, even when
any initial reluctance by program and agency leaders is overcome. In 1997, Robert
Boruch, in his seminal text Randomized Experiments for Planning and Evaluation,
wrote: “the main lesson from these [the Spouse Assault Replication Program] and
other experiences (e.g. Conner 1977) is that the possibility of failure is real. It must be
planned for”(p.184). This statement cannot be underestimated in its importance not
only with regard to initial planning and implementation of an experimental evaluation
but also with regard to maintaining random assignment and ensuring subject enroll-
ment goals have been met. Given the complexity of randomized experiments, plan-
ning is not done in a vacuum—researchers usually rely on their past research
experiences or those of their colleagues—usually passed on orally and sometimes
found in project reports (Lum and Yang 2005). Although a handful of general
descriptive texts have been written about randomized experiments for social science
evaluation, and another few dozen articles have been published about specific
challenges, there remain very few resources, published and unpublished, available
to help guide researchers wishing to develop randomized experiments in the social
sciences, and even fewer resources specifically focused on the field of criminal
justice.
1
Even within criminal justice, because the range of project types is wide
(e.g., policing, courts, corrections, etc.), and aspects of random assignment within an
experimental evaluation are varied, it is unlikely that the extant literature covers
pertinent topics and challenges in the depth needed to help inform future planning.
In this paper, we discuss the lessons learned during the implementation and
enrollment phase of the evaluation of the Cook County Frequent Users of Jail and
Mental Health Services Enhancement Project (“Chicago FUSE”), a housing reentry
program. Although many of the lessons learned will be relevant for criminal justice
experiments conducted in all types of settings, because enrollment into the evaluation
occurred while individuals were incarcerated in jail, our issues will resonate squarely
with researchers conducting experiments in correctional and community correctional
settings. In particular, this article highlights the challenges of implementing research
in jails that offer post-release services initiated in the institution.
In our review of the literature on conducting experiments in criminal justice settings,
we found that the majority of the extant literature highlights the difficulties with gaining
and maintaining cooperation from agencies, how to overcome the perceived ethical and
legal issues of random assignment, issues with attrition from treatment, and how to
maintain the integrity of the experimental and control conditions (see, for example,
Boruch 1997; Clarke and Cornish 1972; Farrington 1983; Petersilia 1989; Weisburd
1
Others have also lamented the paucity of resources and literature discussing the non-technical aspects of
the research specifically related to planning and study enrollment (see Rezmovic et al. 1981; Petersilia
1989).
308 C.G. Roman et al.

2003). The literature does not provide a full picture of the wide range of possible
enrollment or “pipeline”issues that, together, can sink a rigorously designed evalu-
ation (Boruch 1997; Petersilia 1989). The pipeline is the flow of eligible individuals
into the study over time (Boruch 1997). Warnings about the likelihood of encounter-
ing pipeline issues exist (see, for example, National Research Council 2005), but
specific descriptive examples and narrative accounts that can serve as exemplary
lessons are rare. For reentry programs, the pipeline poses unique challenges because
there is not a large population of eligible individuals waiting to be randomly assigned
on any given day; enrollment most often is a rolling process where individuals are
identified and recruited during their period of incarceration or at the time of their
release, but only truly enroll in or “attach”to the program upon release. Depending on
how program eligibility is defined—and it is often clearly set before the researcher
comes into the picture—and how the treatment slots operate (e.g., all available at
once versus as beds/slots become available), the enrollment time period can range
from a few months to a few years. As the months add up, the potential for problems
increase (Hatry and Newcomer 2004; Morell 2010).
In the Chicago FUSE evaluation, we did not encounter some of the typical hurdles
during the pre-planning and initial implementation phase, such as staff unwillingness
to support randomization due to legal and ethical issues or program staff overriding
random assignment. However, we were struck head on with challenges as we sought
to manage enrollment over a lengthy period. Hence, we focus our discussion mostly
on the study enrollment period, and, in particular, defending and managing the
pipeline—the process and practice of how cases flow into and out of the study and
are managed as the evaluation is progressing. Defending and managing the pipeline
involves much more than carefully crafting eligibility criteria to ensure a sufficient
number of cases will be attained, and, in turn, provide sufficient statistical power. In
other words, issues of the pipeline are not solely design issues; how one manages the
pipeline throughout the enrollment period and anticipates hurdles will be fundamental
to the success of the evaluation, and ultimately, to producing useful research.
To provide context for the issues encountered in the FUSE evaluation and lessons
discussed, we first describe Chicago FUSE, the evaluation, and the enrollment process.
This context is important, as the information learned and resulting decisions made in the
pre-planning stages had a huge impact on the resources dedicated to defining, managing,
and defending the pipeline and the final size of the sample for evaluation. After
discussing the context, we describe our experiences, with emphasis on the challenges
encountered, field techniques, and strategies utilized to try to overcome pipeline issues.
We highlight lessons learned in the evaluation management process and reflect on
(given hindsight) major research design issues and dilemmas that are likely to continue
to plague experimental evaluation in the field of prisoner and jail reentry.
2
The Chicago FUSE program
Chicago FUSE is a jail-based reentry program designed to serve frequent users
of jail and hospital services who have a current AXIS I mental health diagnosis
2
We do not report on final project outcomes.
Defending the pipeline 309

and a history of chronic homelessness. The demonstration provides permanent
supportive housing
3
to individuals leaving the Cook County Department of Correc-
tions (CCDOC), the largest single-site jail in the country. Chicago FUSE is run by the
Corporation for Supportive Housing (CSH) in a joint collaboration with the Cook
County Sheriff’s Office and Cermak Health Services of Cook County, an affiliate of
the Cook County Health and Hospital System (CCHHS). The program is part of the
CSH’s nationwide Returning Home Initiative (RHI) funded by the Robert Wood
Johnson Foundation. RHI, which was the foundation for Chicago FUSE, has three
main goals, to: (1) reduce recidivism; (2) reduce homelessness/decrease shelter usage;
and (3) decrease the costs associated with multiple service use across the criminal
justice, housing/homelessness services, and mental health services systems. RHI
funds supported a full-time RHI program manager in Chicago (who is also the FUSE
program manager) who devoted 100 percent of his time to achieving the goals of
RHI, with a particular emphasis on getting Chicago FUSE off the ground (for a brief
summary of RHI activities, see Roman et al. 2009).
Although Chicago FUSE was based somewhat on a small demonstration taking place
in New York City, called New York FUSE, the idea was to create a program in Chicago
that would identify and enroll individuals while they were incarcerated and, hence,
begin the linkage to services while individuals were still in jail. To our knowledge,
developing FUSE as a true reentry housing program (i.e., enrolling and serving only
those leaving a correctional institution as opposed to individuals in shelters with recent
incarceration histories) was a novel idea. Although it is notoriously difficult for service
providers to secure permanent housing vouchers for returning prisoners,
4
the Chicago
program manager was dedicated to the idea of creating an innovative permanent
supportive housing reentry program designed to serve persons with severe mental
illness who were chronically homeless and transitioning out of jail.
The interest and commitment of local elected officials was central to the develop-
ment and operation of a reentry housing program modeled after New York FUSE.
The Chicago program manager obtained the commitment of the President of the Cook
County Board of Commissioners to facilitate momentum in the jail for a program that
would require the cooperation of various city, county, and state agencies—given the
focus on frequent users—including the Illinois Department of Human Services,
Office of Mental Health, the City of Chicago, and the Director of the Cook County
Department of Corrections.
The program manager received written agreement from the City of Chicago,
through the Low-Income Housing Trust Fund using state-supported rental housing
support program subsidies, that the program would secure 120 vouchers dedicated to
permanent housing. The city agreed that the rental subsidies could be used for the
long-term homeless leaving the jail provided they met the federal definition of
3
Supportive housing is the combination of permanent, affordable housing with supportive services aimed
at helping residents maintain housing stability. While not all supportive housing programs are the same,
shared components include affordability (tenants generally do not pay more than 30–50 % of their income
in rent) and a range of services, including coordinated case management, health and mental health services,
substance use treatment and recovery, vocational and employment services, money management, life skills,
household establishment, and tenant advocacy.
4
In many jurisdictions, vouchers are tied to Federal definitions of chronic homelessness. A long stay in jail
or prison does not qualify a person as chronically homeless.
310 C.G. Roman et al.

chronic homelessness upon entry into the jail. The subsidies required that a
community-based provider offer long-term support in the community. The subsidies
were landlord-based and available from the city for a limited amount of time. Given
the limited resources secured for the housing—only 120 units—enrollment into the
program was contingent on participation in the research study. That is, there were not
enough housing slots to provide housing to a group of individuals who did not
consent to participate in the research study.
To be eligible to receive permanent supportive housing (the treatment), a detainee
had to meet each of four criteria: (1) have had at least two stays in a mental health
inpatient program since 1980; (2) have had at least two admissions to the jail since
January 2005; (3) have a current Axis I mental health diagnosis; and (4) have had at
least four episodes of homelessness in the last three years and/or one year of
consecutive homelessness (the federal definition of chronic homelessness). These
criteria were established to ensure that the project was focused on those most in need
of housing and services, thereby maximizing the utility of the intervention’s resources
as well as the long-term sustainability of the project. In addition to the stated criteria,
detainees were ineligible if they were convicted of and/or arrested for arson, sex, or
child molestation offenses. These restrictions were imposed by the local rental
subsidies used for the program.
The evaluation
The evaluation proposal was a product of broader systems change research that the
Urban Institute was conducting for CSH on the RHI initiative, which was operational
in three cities at the time. The research team suggested that there may be stand-alone
projects within RHI that could support a rigorous evaluation, leading to a series of
phone conversations with RHI program managers in the three cities and site visits to
assess project suitability for evaluation.
By the conclusion of the site visit to discuss Chicago FUSE (in development, but
not yet implemented), the research team believed FUSE was a strong candidate for an
experimental study. Fortunately, the Chicago RHI manager was a proponent of
experimental studies and it did not take long for him to gain the support of political
leaders in Chicago and agency administrators. Over the next two months, supported
by RHI funds, researchers conducted a full evaluability assessment
5
and brief pipe-
line study to prepare for a proposal submission to the National Institute of Justice’s
(NIJ) investigator-initiated solicitation in early 2007.
In late 2007, the NIJ research grant was awarded to the Urban Institute. The
research team proposed a three and a half year experimental study that would rely
on: (1) the collection of in-person baseline surveys with all research subjects in the
treatment and control group, prior to assignment and occurring while individuals
were still incarcerated, and (2) administrative data from the CCDOC, Illinois State
Police, Illinois Department of Human Services, Cermak Health Services (jail health
5
Evaluability assessment (EA) is a process for assessing the overall feasibility of an evaluation before the
evaluation takes places. EA helps researchers select designs for evaluations that are feasible, relevant, and
useful. EA is described in detail in Wholey (2004).
Defending the pipeline 311

care), and homeless management information systems (HMIS) collected at one-year
post release. The evaluation did not include follow-up in-person interviews with the
research subjects since the probability of successfully tracking and interviewing
individuals with long histories of homelessness, mental illness, and jail use was
deemed very low, regardless of resources.
We concluded from the evaluability assessment that, to ensure maximum gener-
alizability of the research findings, the most appropriate design would involve
random assignment occurring within the jail before detainees were released. There
was some discussion about trying to randomize eligible persons on the day of release
rather than on the day of intake. However, enrollment at release would have been
much more disruptive to the jail operations, requiring late night evaluations of
eligibility for multiple research criteria under considerable pressure, and it would
have been logistically complicated to house people that leave the jail after midnight or
in the early morning hours. Following some discussion, the decision was made to
evaluate people as soon as they arrived to the jail and to engage and follow them as
they proceed through the court process.
In addition, the decision was made against community-based enrollment because it
was likely that only a small portion of potentially eligible individuals would make it
through a provider’s door, thereby drastically narrowing the type of individuals who
would be served by the program—limiting generalizability as well as the overall
policy usefulness of the program (Morell 2010). There was also strong interest in
ensuring the program was a reentry program, which sought to facilitate a smooth
transition from the jail to the community among a group of individuals with histories
of cycling between systems. Hence, jail “in-reach”was an important component of
the treatment.
Randomization occurred shortly after a person was identified as eligible, con-
sented to participate in the research study (and potentially receive housing), and
completed the baseline survey. While randomization occurred before a person was
released from jail, with the exception of a few possible contacts between client and
provider during jail in-reach, all the services occurred in the community. As Gueron
indicates (2002, 27), the longer the route from the point of random assignment to
actual enrollment in services, the lower the percentage of individuals assigned to the
treatment will actually get the treatment. We acknowledged that we would have to
address the many ways detainees could be lost in the pipeline, and were particularly
confident we had estimated the correct percentages that would not be released during
the study/program enrollment period. The evaluability assessment that had been
conducted a few months before the grant was written had provided us the opportunity
to examine data on jail admissions and mental health services, from which we could
ascertain that we would not have much trouble finding detainees who met the AXIS I
diagnosis and would still be released to the community a short time after jail
admission. New York City’s experience with a FUSE program further confirmed that
a significant portion of individuals who are frequent users of jail are also frequent
users of homeless shelters (within any given month at least a few hundred individuals
matched both criteria) (Moser 2007). But past experience in reentry evaluation and
reinforcement from the National Research Council’s (NRC) report on improving
evaluation of anticrime programs (NRC 2005), prodded us to conduct a more detailed
pipeline study of case flow immediately after the grant was awarded. The pipeline
312 C.G. Roman et al.

study involved using newly available data tools in the 60 days after the grant was
awarded to estimate how many potential clients would be lost as the funnel narrowed
toward community-based services.
More specifically, we had access to an electronic database system in the jail,
known as DataLink, which identifies detainees with histories of mental illness.
DataLink, developed by the Illinois Department of Human Services, Division of
Mental Health (DMH), is a cross-match between the jail census and open cases with
local mental community mental health providers. The Chicago project manager
provided funding to enhance the functionality of DataLink to identify and prioritize
the most frequently jailed and hospitalized in the state psychiatric system, who may
or may not have had services from mental health providers. The enhanced DataLink
system revealed that on a Monday in October 2007, 202 people of the 11,016 total
detainees in the jail on that day matched the program/study 2 by 2 eligibility criteria
(at least two jail stays since 2005 and two inpatient mental health hospitalizations
since 1980). We then used the DataLink system to chronicle how many detainees
entered the jail the remainder of the week and met the 2 by 2 eligibility criteria (9 new
detainees over 4 days). A week later, we examined DataLink again and confirmed that
roughly 9 to 10 individuals who met the 2 by 2 eligibility criteria entered the jail
weekly. Then, using statistics produced by colleagues working on New York FUSE,
we estimated that 50 percent of those eligible on the frequent user criteria would be
chronically homeless. We discussed the pipeline flow with medical staff and medical
social workers in the jail and they agreed with our estimates.
Basedontheknowledgethat,onanygivendayinjail,therewouldbe
roughly 200 individuals that met the 2 by 2 criteria, reduced by half to meet
the homelessness criteria, and another 15 percent who, according to Cook
County jail statistics, will go on to prison or never exit to street, we estimated,
on the conservative side, that at least five individuals would be in the jail each
day who met all of the eligibility criteria and could be approached. With other
possible issues (e.g., jail stay of one or two days—too short to be identified
and interviewed, incoherent subjects, and refusals), we estimated we could easily
recruit 400–500 detainees over 16 months. Our goal was to consent 30 individuals
each month. However, because the number of housing vouchers was limited and
providers indicated they only had the capacity to carry small caseloads due to the
intensive nature of services, we were faced with some important decisions about the
ratio of random assignment.
After a number of discussions with UI’s Institutional Review Board, we
proposed allocation to the treatment group at 33 percent. Individuals had a
33 percent chance of being placed in the treatment group. The assignment ratio
was not 1 to 1 because of serious concerns with the availability of permanent
housing vouchers in Chicago that could be used to serve single adults. We were
also concerned that having a relatively small sample of treatment and control
cases would not permit a robust test of statistically significant differences
between the cases, given that many evaluations of reentry programs have found
rather small effect sizes. With a 1 to 2 ratio for random assignment, after
16 months of recruitment, we anticipated a treatment group sample of approx-
imately 160 individuals and a control group sample of 304 individuals for a
total study sample of 464.
Defending the pipeline 313

The pipeline from recruitment to program entry
During the pipeline study, it became clear that evaluation researchers would be
needed at the jail everyday to identify, recruit, and manage enrollment. Even if the
jail had available staff willing to recruit for the program/study, past experience, the
research literature (e.g., Petersilia 1989; Rezmovic et al. 1981), and overall jail budget
constraints suggested that a research-funded staff person, not a program staff person,
should be responsible for screening and baseline data collection. This was budgeted
into the grant. After the grant was awarded, the partners agreed that having a locally-
based full-time person familiar with the population to be served and affiliated with
local providers already working within the jail would greatly benefit the program and
evaluation. CSH funded two staff positions—one as the “enrollment case manager”
and one as the “linkage coordinator.”Although the positions were hired and super-
vised by one of the four treatment providers involved in Chicago FUSE, the staff
hired were considered part of the research team. The enrollment case manager was
responsible for identifying, verifying, consenting/surveying eligible clients, and
notifying clients of evaluation group assignment. Fig. 1depicts the process used to
identify, consent, and assign clients. The enrollment case manager had a number of
steps to take before a detainee was approached for consent and enrollment in the
program/study:
1. Check DataLink for eligibility on the two hospitalizations and two incarcerations
criteria;
2. Check eligible persons’booking charges using the jail databases for those
charges that were non-eligible and to determine the individual’s likelihood of
being released within the study period;
3. Check the jail medical case files to ascertain whether detainee has a current Axis I
disorder;
4. Meet with the jail medical social workers to ensure that the individual was not
already linked to a non-FUSE service provider;
5. Locate the detainee in the jail facility and administer the coherency checklist
developed specifically for the study and;
6. Administer the chronic homelessness “checklist”developed for the study.
Following a determination of eligibility in step six, the enrollment case manager
called an off-site researcher to obtain the group assignment. After random assign-
ment, treatment group clients were referred to the linkage coordinator who then
connected individuals through a discharge plan with one of four possible providers
of permanent supportive housing connected to the program. The linkage coordinator
worked with the providers to ensure that efforts were developed to make initial
contact with the detainee while incarcerated (jail in-reach), and to track individuals
in the treatment group who might “disappear”before they were placed into housing.
Control group subjects were randomly assigned to “business as usual”and were
referred to the jail medical social workers to determine a housing plan for them. They
received information and incentives to encourage them to stay in touch with the
researchers, including postcards from the research team with reminders of how to
secure housing and supportive services upon release and a $20 gift card incentive six
314 C.G. Roman et al.

months after release to remain in contact with the research team. Remaining in
contact with the control group by providing information and incentives was used as
a method to reduce the potential for harm to the individuals who did not receive the
treatment. Using this approach for control subjects, the research team was able to
overcome the typical hurdle lodged against experimental research about the ethical or
moral issues with providing services to some while not providing services to others
(Rossi et al. 2004; Petersilia 1989).
Fig. 1 Chicago FUSE evaluation screening and enrollment process
Defending the pipeline 315

Preparing the pipeline process
We assumed that hiring enrollment staff would be easy given that we were placing
little burden on jail staff for this task and providing additional resources to the jail.
Yet, once CSH agreed to hire staff, it took nearly six months for the county and state
to decide how the funding for the positions should flow, who would supervise the
new staff, and how the staff persons would separate their roles as clinicians, court
advocates, and researchers. These decisions had to be made before an application
could be submitted to the local human subjects’review board since the staff would
hold key positions within the evaluation team. Issues such as the total count of
temporary grant positions are important in tight economic times in an era of budget
uncertainty. In addition, there were confidentiality issues as to whether a person could
view statewide mental health data if employed by a clinical, non-governmental
agency.
These issues were resolved after considerable discussion among multiple partners
with multiple agendas. One of the four clinical community-based providers was
selected to hire, train, and supervise both staff persons. The direct clinical supervision
of the enrollment case manager (a masters-level clinician with research skills) was
provided by the Urban Institute and Cermak Hospital (i.e., the jail medical center)
staff. For purposes of viewing the data systems linked to eligibility, this staff person
was able to view the statewide roster of potential clients for the study and to view data
for both experimental and control subjects after random assignment.
Defending the pipeline
The decision to hire a new staff person to conduct enrollment also brought other
issues concerning the appropriate timing and training of new staff and obtaining jail
clearance. The enrollment case manager and the linkage coordinator were hired at the
time we estimated that the human subjects’research protocol would be cleared (May
2008). We could not begin training until the protocol was approved, because the
enrollment case manager could not view any data until passage of the protocol.
Training was complicated because of the multi-tiered process to identify eligible jail
detainees (see Fig. 1). As such, we anticipated tremendous effort on the part of the
enrollment case manager and planned for weekly trouble shooting teleconference
meetings between the research and program team. We expected issues to arise that
might interrupt the flow of information to the case manager or make it difficult to
approach a detainee for consent; yet, we did not anticipate the number and stubborn-
ness of some of the issues that narrowed the pipeline and led to far-below-expected
weekly enrollment. Below, we discuss these issues and the lessons learned.
Opening the pipeline
Successfully managing the pipeline begins with approval of the human subjects’
protocol that would provide permission for the eligibility, enrollment, and consent
process as developed by the research team. In addition to the Urban Institute’s
Institutional Review Board (IRB), the human subjects’protocol for the Chicago
316 C.G. Roman et al.

FUSE evaluation had to pass four IRBs, including the Cook County Bureau of Health
Services’IRB, the Illinois Department of Human Services’Office of Mental Health
IRB, and the IRBs of two of the four service providers participating in the project. While
the Urban Institute’s IRB approved the protocol in November 2007, it took over a year
for the additional four IRBs to approve the protocol, which considerably delayed the
commencement of enrollment. Sample recruitment did not begin until January 2009,
almost one full year after the study’s projected timeline for starting enrollment. This
delay meant that considerable resources were expended by the research team managing
the IRB process—resources that were needed to manage the project’s enrollment
process. Looking back on the process, we suggest a number of strategies to mitigate
potential hurdles that can arise early in the pre-implementation phase:
&Long before implementation of your evaluation, cultivate connections with
researchers who have successfully implemented experimental evaluations in the
local jurisdiction where your study is located to help navigate the potential
hurdles you might face. Even more important, perhaps, with regard to evaluations
that involve research subjects with an extra layer of protections (in this case
“prisoners”which includes detainees, prisoners and parolees), build relationships
with local researchers who have recently had protocols involving prisoners
approved. “Recently”is key because the makeup of IRBs change over time.
&Collaborate with a site-based researcher familiar with the local IRB process who
is interested and supportive of your research project. Having this person become a
paid part of your research team can further solidify support. In our case, the
CCBHS (the agency that oversees studies that take place in the jail) mandated a
“local”principal investigator (PI) employed by the County be responsible for all
tasks associated with IRB submission.
&For evaluations involving services to individuals with mental illness, ensure that the
enrollment staff has proper training and experience with individuals with mental
illness to help build a human subjects’protocol that demonstrates the appropriate
skills to ensure proper protection of potential research subjects. Targeting detainees
with serious mental illness adds a layer of complexity to the enrollment process and
can have serious implications for the pipeline (Wolff 2000).
&When writing grant proposals, build in at least six months of time in the project
timeline to clear IRB. However, six months will be too short if your protocol must
pass through more than one IRB. Starting the IRB process before grant funds
arrive certainly can be helpful, but not every grantee agency has this luxury,
depending on the particulars of how IRBs are funded and managed.
&Strategize carefully about the timing of hiring of research/enrollment staff for
study implementation. The timing is critical to careful spending of valuable
resources. As Rossi and colleagues (2004) stated: “Experienced evaluators also
know that one of the most precious resources is time. The time allotted for
completion of the evaluation and flexibility of deadlines are essential consider-
ations in evaluation planning but are rarely determined by the evaluator’s prefer-
ences”(p.47). Because we anticipated that IRB approval would take six months,
we had the case manager hired and trained for six months without an approved
IRB. This had a serious impact on resources and our ability to extend the timeline
as the study progressed.
Defending the pipeline 317

Managing the pipeline
When the pipeline study was conducted, we had reviewed the extant criminal justice
literature on experimental evaluations to glean the enrollment lessons from past
experiences. There was limited information to be found in academic journals, but a
few problems were presented in final grant reports. The predominant issues that
interrupted or slowed enrollment were related to enrollment staff turnover (Gondolf
2010; Petersilia 1989) and agency cooperation (Feder et al. 2000; Gondolf 2010), but
many studies just noted that the authors “over anticipated”the case flow (Bickman
1985; Devine et al. 1994; Petersilia 1989), or noted that enrollment took longer than
expected and did not provide any details behind this statement. Other, more rigorous
academic scans for enrollment lessons found similarly slim advice. For instance,
Davis and Auchter (2010), in their review of implementation issues in successful
experimental studies of interventions to prevent violence against women, found that
in 5 of the 19 evaluations no information was provided on the expected or actual
length of the intake period. Eight studies noted slower than expected intake rates, but
with few details on the problems.
Because our estimates of case flow were based on the results of the pipeline study,
we were surprised by many of the issues that arose. On average, 8 total detainees
were recruited for the study each month (ranging from 2 to 15 per month), when we
anticipated 30. However, we kept track of all potential detainees who entered the jail
as frequent users of jail and mental health services to determine where we “lost”
potential cases in order to assess and confront head on any issues with the pipeline as
they arose. Figure 2shows the percentage of detainees lost to various factors as they
moved through the pipeline. We describe these factors below and discuss the deci-
sions we confronted in working to overcome these hurdles.
The first issue that severely impacted the pipeline occurred before enrollment even
began. At the time of the evaluability assessment, the Cook County Jail was imple-
menting a new screening tool that would include screening for homelessness. The
2.4%
0.7%
1.1%
2.7%
3.9%
4.6%
5.2%
6.0%
10.1%
17.7%
25.8%
0% 5% 10% 15% 20% 25% 30%
Refusal
Referred to sta te inpatient MH center
Released to suburbs
Not eligible according to med ical doctors
Inappropriate diagnosis
Unable to interview due to symptoms
Prison/extradited
Already engaged with an outside agency
Exclusiona ry charges
Not chronically hom eless
Gone before able to approach
Fig. 2 Percentage of detainees lost in pipeline during enrollment period, Chicago FUSE
318 C.G. Roman et al.

evaluation was depending on this information to screen for study/program eligibility
related to the homelessness criteria. In the nine months between proposal submission
and grant award, jail administrators dropped the finalization of the new screening
protocol due to budget cuts and staff changes across the county that affected the jail.
These cuts were the first of its type in nearly 30 years. As shown in Fig. 1, the case
manager had to approach detainees and screen for chronic homelessness after all
other criteria had been met. As shown in Fig. 2, approximately 18 percent of the
detainees who appeared eligible based on their hospitalization and jail histories were
determined to be ineligible based on the homelessness eligibility criteria. This
percentage translated to 202 clients over a two-year period, a significant amount of
resources expended by the sole enrollment case manager on cases that would turn out
to be ineligible.
Number of ineligible cases on mental health eligibility criteria underestimated
Program staff, researchers, and provider staff had originally calculated that all the
Axis I clients identified by DataLink would be eligible for the demonstration. After
the study began, it came to our attention that the rental subsidies could only be used
for a subset of Axis I diagnoses that were designated as “severe.”Unfortunately,
DataLink was designed to include persons with substance abuse as their primary
diagnosis that had been hospitalized. These individuals are not eligible for the
services provided in the treatment. According to our pipeline tracking database,
roughly 7 percent of potentially eligible cases were excluded from the study because
they had ineligible mental health diagnoses (3.9 percent combined with 2.7 percent
identified as ineligible by jail medical staff).
Clients already attached to a service provider
Roughly seven percent of potentially eligible cases were already working with a
mental health service provider that was not one of the four providers participating in
the program. After the program was implemented, jail medical social workers
indicated that they did not want clients who were already working with another
mental health service provider to be part of FUSE. They believed it would be
disruptive to the client to change providers if he or she had already established a
relationship with another service provider. We pushed to tackle this on a case by case
basis, given that FUSE services were more comprehensive than services offered
through many, though not all, other providers, but the long-time medical social
workers in the jail did not want to risk disrupting client–provider relationships.
Clients lost before approached
Although some clients were lost before they could be enrolled simply because they
had very short jail stays of one or two days, sometimes there were institutional or
extra-local issues that impacted access to the detainees. Once the enrollment case
manager consulted DataLink to determine and prioritize a list of potentially eligible
frequent users of jail and hospital services, she was instructed by jail staff to consult
with the two long-time jail medical social workers responsible for developing case
Defending the pipeline 319

plans for the reentry of detainees with mental illness. The medical social workers
often knew without having to review files whether a detainee would meet all of the
eligibility criteria. However, the medical social workers, who carry extremely large
caseloads, had only agreed to a one-hour meeting each week to review the list of
potentially eligible detainees. It was from this “approved”list that the enrollment case
manager prioritized enrollment tasks for the remainder of the week. Essentially, this
meant that new, potentially eligible detainees who entered the jail later in the week
would not be vetted by the medical social workers until the following week, increas-
ing the likelihood that any potentially eligible detainee would be released from jail
before he/she was even approached.
Clients lost to prison
In the decade before this study began, Chicago providers and jail medical social
workers had a fairly good estimate of who would be going to prison within the mental
health population. The percentage sentenced to prison from jail remained generally
consistent. However, during the current evaluation period, the providers were sur-
prised to see unusually harsh prison sentences for persons with mental illness. Judges
appeared less open to plea bargains in court or probation. This meant the enrollment
case manager had to engage detainees for longer periods of time and service providers
had to follow persons through the prison system. Sentences determined by the court
are quite variable. Despite having the Chief Judge of the Criminal Courts as a primary
supporter of the program and the intake coordinator conducting advocacy on behalf
of FUSE treatment clients in court, individual judges make the final determination of
sentences.
Data availability
Other issues related to managing the pipeline included data and client access issues.
At least a handful of times each month, and sometimes for more than a week, the
enrollment case manager was unable to access the live data-matching system due to
system or computer problems in the jail or at the Division of Mental Health. In
addition, in early 2010, the jail began using a new data system and the transition to
the new system hampered all access to the data matching system and arrest, charge,
and court information needed to determine eligibility. It also disrupted the jail’s
ability to get people to court. The inaccessibility of jail data lasted almost one month
during this process.
Institutional context of jail reentry
Jail riots, system-wide lock downs for counts or behavior, and the size of the facility
also limited access to clients by the enrollment case manager. The challenges of
reentry in the jail context have been documented in the literature (see Solomon et al.
2008); the unique context of jails creates a complex landscape for reentry planning
(e.g., some individuals are detained that are not serving a sentence, others are
detained that are likely to serve a long sentence in prison, frequent cyclers, unpre-
dictability of release dates, etc.). In addition, there were staff changes in the jail
320 C.G. Roman et al.

including the retirement of a staff person of several decades who helped establish the
mental health program in the jail. New staff oversaw many changes including the new
data system, staff layoffs, and an internal move of all of the detainees within the jail
with mental illness to new locations in the jail, which necessitated change in various
staff offices, personnel, procedures, meal and count times as well as relationships
between staff. These issues and changes were beyond the direct control of the case
manager.
Furthermore, the enrollment case manager had six steps to take before someone
was even considered eligible to be in the study/program. These six steps, each quite
discrete and based in the context of a jail, meant that only one or two detainees could
be approached on any given day, let alone confirmed as eligible and consented.
State budget cuts
State budget problems also severely impacted case flow. From July through August
2009, in response to budget cuts in mental health services in Illinois, recruitment was
slowed considerably at the request of the service providers and program manager.
Illinois state budget cuts for mental health services meant that providers were forced
to lay off a significant number of their staff (80 staff at one of the four agencies) and
would only be reimbursed by the state for serving clients who were Medicaid-eligible (a
funding stream that was untouched by the budget cuts). As most people leaving the jail
lose their Medicaid eligibility, this caused significant disruption among the providers,
the enrollment case manager could only recruit detainees who were Medicaid-eligible.
Over these two months, only 11 detainees were recruited and consented.
Expand, defend or admit defeat?
Boruch (1997) addresses pipeline issues and offers three strategies to mitigate prob-
lems: (1) alter eligibility criteria, (2) extend the study timeline, and (3) intensify
outreach/recruitment of subjects. These strategies cannot always be employed be-
cause they are often dependent on available resources or how the pipeline was
constructed. For the current study, altering the eligibility criteria was not an option
due to the funding requirements for the housing and treatment services arranged
during the program planning stage. The different funding requirements for the mental
health services and rental subsidies, which added complexity to the eligibility pro-
cess, were inflexible. Therefore, we were unable to make adjustments to the eligibility
criteria. The funding for mental health services required that the person have a
diagnosis of schizophrenia, schizoaffective disorder, severe major depression, or
bipolar disorder. These diagnoses permitted a person to qualify for both social
security and state Medicaid benefits quickly, which was beneficial to the providers
for reimbursement of services. The funding for the rental subsidies from the state and
federal government required that the person be defined as chronically homeless using
the HUD definition. Each of these systems tightened their criteria as the external
world economy worsened; the project slowed; and the need for adjustment in the
research increased.
We explored the idea of recruiting women with co-occurring disorders housed in a
substance above treatment program in the jail or to work with people on the mental
Defending the pipeline 321

health probation unit. The research and program team decided these options would
hurt the homogeneity of the treatment group that we fought so hard to maintain, add
time consuming steps to the long enrollment process, and likely result in the enroll-
ment of clients not appropriate for treatment services.
Extending the timeline for the program/study was also not an option the group
could seriously entertain. Extending the timeline was costly to the evaluation; we
could not afford to add more than six months to the evaluation timeline without
severely jeopardizing resources that would be available for follow-up data collection
and other evaluation tasks. Extending the timeline could also jeopardize access to the
remaining housing vouchers; the vouchers had been intended for use within a specific
fiscal year. The lengthy IRB process and slow matriculation into the treatment group
given the randomization ratio were jeopardizing the program’s access to the rental
subsidies. As a result, the evaluation team was under pressure from program staff and
providers to change the randomization ratio to 1 to 1, which we did in the second year
of enrollment. Although this increased the number of individuals who received the
treatment, it had no immediate benefits to the evaluation, as the same total number of
subjects would be enrolled.
In the jail setting, intensifying outreach meant simply utilizing more resources to
access the same possible base of clients. The enrollment case manager was already
having a number of issues accessing clients in the jail, which made intensifying
outreach difficult to envision. We gingerly inquired as to whether the jail medical
social workers could take on new responsibilities or give greater access and freedom
to the case manager, but the state budget cuts had severely impacted jail staffing,
making any request for research assistance seem extreme.
Although we did not anticipate the breadth and stubbornness of enrollment issues,
we had set forth (and were following) a careful project monitoring strategy as
previous evaluation researchers have suggested (see, for example, Bell 2004). We
conducted standing weekly telephone conferences among the research and program
team expressly designed to identify possible enrollment issues and problem solve
before the issues became insurmountable. Each week, at least a half dozen researchers
and practitioners floated ideas about how to increase the pipeline. The time expended
by the research team on trouble shooting, conducting site visits and meetings at the
jail and with program partners was significant.
In the end, the research team was not able to increase enrollment. We were
beginning to jeopardize the strong relationship between the research and the project
team. Study procedures were draining program resources, particularly because CSH
was providing direct funding for a research position and a sizable number of housing
slots had not been filled. Final attempts to secure additional federal funding after
detailing enrollment issues resulted in the grant budget being cut, which, in turn, led
to the joint decision among the research and project team to terminate enrollment at
that time (October 2010). The research and program team decided to continue
tracking the 160 individuals in the study (59 in treatment group)—a sample size
unlikely to provide sufficient power for rigorous quantitative analyses.
As we revisit the issues that arose in the pipeline process and reflect on our failure
to reach the targeted sample size, we are struck by how typical some of the issues
appeared, but more so by the lack of extant literature chronicling these issues that
perhaps could have helped guide us in making better decisions in the planning phases.
322 C.G. Roman et al.

We take some consolation in reading Boruch’s concluding thoughts in his chapter
titled “Population, Power and Pipeline”(1997):
Terminating a controlled study rarely is a palatable choice for the study’s
sponsors, the experiment’s designers, and other stakeholders. Nevertheless, it
might be sensible when the necessary sample size cannot be attained. The
resources that then become available can be used to investigate and explain
why the difficulties were insurmountable and to understand how a less ambi-
tious (but more feasible) study might be designed. There is no substantial body
of literature on the orderly termination of experiments. (p.95)
Discussion
Criminological experts on random assignment suggest that raising the rate of big
successes in experimental evaluation would have positive implications for the field.
Sherman (2007), and others (Conner 1977; Shadish et al. 2002) suggest a number of
reasons why randomized experiments fail or show weak program effects. These
include weaknesses due to weak programs (which include poor theory, poor program
fidelity, or low dosage), samples which are too heterogeneous, samples that constitute
low-harm (e.g., minor drug offenders), and randomization controlled by program
agency staff rather than researchers. We believe the evaluation of the Chicago FUSE
program was on the path to avoid all or most of these problems. FUSE is theory-
based; has had strong program fidelity; and provides high dosage in the package of
permanent housing bound tightly to a full suite of intensive supportive and enhanced
services. Providers serving treatment clients could access supplemental client-based
project funding that supported the provision of more quality services than would be
available otherwise, including jail pick-ups and emergency funds for what are
typically non-billable/reimbursable services. Providers have strong incentives to
utilize the full range of services and enhancements offered to clients.
In addition, FUSE clients are at high-risk of recidivism as crafted by the strict,
multi-dimensional eligibility criteria. FUSE clients consume a highly disproportion-
ate share of the total resources and, hence, have the potential to generate large effects
in terms of public safety, shelter avoidance, and the reduced costs to state and local
governments associated with fewer returns to jail and shelter. Yet, we could not meet
our projected enrollment and satisfy the evaluation grant requirements, and we were
left to make tough decisions about whether and how to continue all of the data
collection tasks for follow-up. The enrollment challenges we faced and the decisions
we made forced us to reflect on some of the larger issues facing experimental
evaluations in criminal justice, which we touch on below.
Major lessons
The inherent challenges evaluating a “new”program
The lure of conducting a rigorous, timely experimental evaluation of a new
and innovative program can be great, but the costs and benefits must be
Defending the pipeline 323

carefully weighed. New programs need time to develop, particularly those that
draw on a diverse set of partners and foster new collaborations or processes
that have not been in place previously. This lesson is not a new one (Hatry
and Newcomer 2004; Rossi et al. 2004), and other innovative reentry programs
have experienced challenges due to the “newness”of the partnerships being
created (see Fontaine et al. 2011). However, with regard to Chicago FUSE, the
groundwork had been laid by partners that supported the opportunity for a random-
ized experiment. The research team was impressed by extent of the Chicago partner-
ship and wanted to capitalize on the opportunity at the time. The logic behind the
program was carefully articulated and processes for client and data access were
apparent.
While the program logic was clear, it took time to figure out how to identify the
appropriate data needs and processes most effective for readily identifying those who
are appropriate for supportive housing—this was not unforeseen given the layered
criteria for program participation (Rossi et al. 2004) and the fact that each criterion
was related to different sources of data. Looking back, this should have been a
warning sign, as Hatry (2004) and others (Morell 2010; Nightingale and Rossman
2004; Rossi et al. 2004) have stressed how the complexity of program operations can
compound data collection problems and undermine evaluation. We would have had a
greater chance at success had we carefully worked through the targeting and enroll-
ment procedures and documented program inputs, outputs, and performance meas-
ures to ensure they were in place and operating before attempting a rigorous impact
evaluation. The best option, in hindsight, would have been to set aside the first few
months of the evaluation period—post-IRB approval—to incorporate a new pipeline
study that mimicked actual enrollment (see point below). In the case of Chicago
FUSE, the funding constraints and timeline just did not permit enough time for these
implementation activities to occur, mostly due to the delays in the IRB process. We
were also under particular pressure to start the evaluation at the time of program
implementation because housing vouchers for the reentry population were in limited
supply.
Pipeline study should mimic field procedures to be used for enrollment
It is too obvious to state that a pipeline study is critical to the success of an
experimental study targeting returning prisoners. Thus, what we emphasize is that a
pipeline study should be conducted under what will be the true conditions and context
for enrollment, given all eligibility criteria. This is applicable for evaluation studies
involving individuals incarcerated in prisons as well as jails. Similarly, eligibility
criteria should be crafted using available, existing data systems. It is worth noting that
we believed we closely followed the recommendations from the NRC’s report on
evaluation of anticrime programs—we conducted both an evaluability assessment
using the checklist questions outlined in Chapter 3 of the report and executed a
pipeline study using real-time data from Chicago, but we did not have all the data
pieces necessary for a true pilot test of our pipeline. Researchers should set aside a
significant amount of funds (as well as time) for the evaluability assessment and
pipeline study. This recommendation is of particular importance for evaluators
considering evaluating programs that have not yet been implemented. In a perfect
324 C.G. Roman et al.

world, funders would cover the pipeline study, particularly when there is some
evidence that random assignment may be possible. In addition, we suggest funders
consider setting aside a small amount of funds in their portfolios that researchers
could request for evaluability assessments. Hence, the onus would be on the inves-
tigator to identify promising programs and to convince funders to invest in a multi-
step process for possible experimental evaluation. Seconding the NRC report (2005,
48), we would suggest that a preliminary year of funding for a pipeline study and full
testing of study enrollment procedures would strengthen evaluation studies that
involve random assignment of individuals.
Innovation and related research processes should be adapted to the local context
Petersilia (1989) suggests that innovation is most likely to succeed if the innovation
is adapted to the local context instead of forcing agencies to fit the innovation.
For Chicago FUSE, the enrollment criteria, informed by the program’s funding
sources, resulted in an entirely new process for client identification. The
enrollment case manager had to use several sources to identify clients as well
as impose on the jail medical social workers to confirm client eligibility. As
Eck (2002) has mentioned, organizations are reluctant to engage in experiments
because it is an intrusion into the agencies’existing processes. One could argue that
the creation of new systems or processes for identifying and serving individuals for
FUSE was not the most efficient way to get detainees into housing (and the research
study). That is, the pipeline could have been identified solely using existing systems,
in the manner that echoed daily routine activities of the jail medical staff, such as
having the jail medical social workers refer individuals to the program, as they came
in contact with them, as the sole means for program/research enrollment. However,
with this approach, it would be likely that not all referred clients would meet set
eligibility criteria.
Anticipate and be responsive to concerns of the line staff
When asking existing line staff—who are doing their own work and responding to
their own sets of pressures—to take on new tasks, adapt new operating procedures, or
interact with new research staff, researchers should be cautious. Echoing sentiments
made previously by Feder et al. (2000), we suggest that, while having champions at
the elected official/public administrator level is necessary to get most projects off the
ground, it is important to recognize that it may be the line staff who have most
influence on the pipeline. It is critical that line staff are included in the project
and that rapport and trust are built with line staff before the project is
implemented. This takes time.
Anticipate disruption from uncontrollable events
Even with a more accurate pipeline study, the disruption in our study from the
state funding cuts, changes in court sentencing practices, termination of the
implementation of the new intake screen that would have incorporated home-
lessness, and later data system upgrades could not have been easily anticipated
Defending the pipeline 325

or controlled. We believed we had balanced research ambition with operational
reality when we estimated over 100 detainees each month would meet the
criteria, and then conservatively planned for a monthly enrollment of 30. Even
with conservative estimates of enrollment, researchers should build in additional
resources and time whenever possible. The inclusion of internal or external
advisory boards or review panels could also provide unbiased advice on how to
balance evaluation resources and make tradeoffs when problems arise (NRC
2005). Also important is carefully relaying enrollment issues to funders as they occur.
We suggest formal quarterly updates on enrollment progress for all program partners
and funders.
Scrutinize options (at the outset of the study) for switching to a quasi-experimental
design
No one managing an experimental evaluation wants to admit defeat. But defeat
might be more palatable if options to “fall back on,”such as a strong quasi-
experimental design, are discussed and prioritized at the outset of the experi-
mental study (Wholey 2004). It is likely that some quasi-experimental options are
worthy in and of themselves, and would help save the resources already devoted to
the evaluation.
Think carefully about the point of randomization
In the jail context, it is tricky to estimate precisely when randomization should
occur, keeping in mind the desire to ensure that those randomized actually
receive the intended treatment (Goldkamp 2008; Gueron 2002). In FUSE, the
process for determining eligibility began upon an individual’s entry into the jail.
Randomization occurred shortly after the enrollment case manager met with the
research subjects, but before they were released from the jail (the point where services
were offered). The time between randomization and release could be days, weeks,
months, or even years.
Because the release of detainees is controlled by the courts, release dates are
therefore subject to many external variables such as politics impacting sentenc-
ing, criminal justice press coverage, and state and local cost cutting. This can
result in dramatic changes in the flow from the jail. It might also lead to a
sudden release of people ahead of a predicted schedule. Basically, anticipating
the timing of release is probably impossible given the large number of issues
that impact release, many of which, as we discovered, will be unanticipated. A
target population that includes individuals with mental illness adds an additional
level of complexity to estimating case flow (Wolff 2000). A pipeline study that
uses real time, real life data (i.e., not simulated), and is conducted at a time proximate
to the start of the evaluation should minimize some of the problems we faced.
However, even with a solid pipeline study, decisions about point of randomization
require careful analysis of how resources for enrollment will be best spent and who is
best set up to manage the process. These decisions should be made with senior
institutional and line staff regardless of whether a researcher is the person enrolling
subjects.
326 C.G. Roman et al.

Concluding thoughts
Perhaps with continued and systematic documentation of enrollment challenges in
future experimental evaluations of reentry programs as well as other experimental
evaluations that involve individuals who are assigned, academics can build a deep
literature that would help facilitate future successes. We believe that detailed narrative
accounts of both successful and failed experimental studies can be particularly useful
for strengthening the management of experiments in the criminal justice field. It is
possible that we could have still encountered insurmountable problems even if we
had conducted a lengthier preliminary study using exact enrollment procedures. A
number of academics have stated that even the best laid plans may not work out
(Boruch 1997; Hatry and Newcomer 2004; National Research Council 2005). Given
the large investment of resources to conduct one single-site evaluation, future re-
search discussion on evaluation management should go beyond general recommen-
dations and checklists and move toward the creation of decision trees in evaluation
feasibility, formalized contingency plans in evaluation implementation, and academic
panels of experts who would be available to reflect and advise at routine interim
progress points. There is no doubt that the difficulties implementing experimental
studies in criminal justice contribute significantly to their sparseness. But progress
can be made from building the literature, sharing experiences, and maintaining
optimism that with careful planning and management, obstacles can be successfully
overcome.
Acknowledgments Part of the research was funded by grant 2007-IJ-CX-0022 from the National
institute of Justice. The research was also supported by the Corporation for Supportive Housing. The
authors gratefully acknowledge the support of Dr. Eileen Couture, Dr. Carlos Quezada-Gomez, and Terri
Marshall in their role as local investigator for the study, and wholeheartedly thank Doris Weiland of Temple
University for administering the random assignment protocol and helping us troubleshoot problems with
the pipeline. The authors also wish to acknowledge the contributions of the reviewers whose comments
improved this paper.
References
Bell, J. B. (2004). Managing evaluation projects. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.),
Handbook of practical program evaluation (2nd ed., pp. 571–603). San Francisco: Jossey-Bass.
Bickman, L. (1985). Randomized field experiments in education: implementation issues. New Directions
for Program Evaluation, 28,39–53.
Boruch, R. (1997). Randomized experiments for planning and evaluation. Thousand Oaks: Sage.
Clarke, R., & Cornish, D. (1972). The controlled trial in institutional research: Paradigm or pitfall for
penal evaluators? London: HMSO.
Conner, R. F. (1977). Selecting a control group: an analysis of the randomization process in twelve social
reform programs. Evaluation Quarterly, 1, 194–244.
Davis, R. L., & Auchter, B. (2010). National Institute of Justice funding of experimental studies of violence
against women: a critical look at implementation issues and policy implications. Journal of Experi-
mental Criminology, 6, 377–95.
Devine, J., Wright, J., & Joyner, L. (1994). Issues in implementing a randomized experiment in a field
setting. New Directions for Program Evaluation, 63,27–40.
Eck, J. (2002). Learning from experience in problem oriented policing and crime prevention: The positive
function of weak evaluations and the negative functions of strong ones. In N. Tilley (Ed.), Evaluation
for crime prevention: Crime prevention studies, 4 (pp. 93–117). Monsey: Criminal Justice Press.
Defending the pipeline 327

Farrington, D. P. (1983). Randomized experiments on crime and justice. In M. Tonry & N. Morris (Eds.),
Crime and justice ( Vol. 4, pp. 257–308). Chicago: University of Chicago Press.
Feder, L., Jolin, A., & Feyerherm, W. (2000). Lessons from two randomized experiments in criminal justice
settings. Crime and Delinquency, 46, 380–400.
Fontaine, J., Gilchrist-Scott, D., & Horvath, A. (2011). Supportive housing for the disabled reentry
population: The District of Columbia frequent users service enhancement pilot program. Washington,
DC: Urban Institute.
Goldkamp, J. (2008). Missing the target and missing the point: “Successful”random assignment but
misleading results. Journal of Experimental Criminology, 4,83–115.
Gondolf, E. W. (2010). Lessons from a successful and failed random assignment testing batterer program
innovations. Journal of Experimental Criminology, 6, 355–376.
Gueron, J. M. (2002). The politics of random assignment: Implementing studies and affecting policy. In F.
Mosteller & R. Boruch (Eds.), Evidence matters: Randomized trials in education research (pp. 15–49).
Washington, DC: Brookings Institution Press.
Hatry, H. T. (2004). Using agency records. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.),
Handbook of practical program evaluation (2nd ed., pp. 396–411). San Francisco: Jossey-Bass.
Hatry, H. T., & Newcomer, K. E. (2004). Pitfalls of evaluation. In J. S. Wholey, H. P. Hatry, & K. E.
Newcomer (Eds.), Handbook of practical program evaluation (2nd ed., pp. 547–570). San Francisco:
Jossey-Bass.
Lum, C., & Yang, S. M. (2005). Why do evaluation researchers in crime and justice choose non-
experimental methods? Journal of Experimental Criminology, 1, 191–213.
Morell, J. A. (2010). Evaluation in the face of uncertainty. NY: Guilford Press.
Moser, R. (2007). Reentry housing: Systems, programs and policy. Presentation to the North Carolina
Department of Corrections (May). New York, NY: Corporation for Supportive Housing.
National Research Council. (2005). Improving evaluation of anticrime programs. Committee on Improving
evaluation of anti-crime programs. Washington, DC: The National Academies Press. Committee on
Law and Justice, Division of Behavioral and Social Sciences and Education.
Nightingale, D. S., & Rossman, S. B. (2004). Collecting data in the field. In J. S. Wholey, H. P. Hatry, & K.
E. Newcomer (Eds.), Handbook of practical program evaluation (2nd ed., pp. 363–395). San Fran-
cisco: Jossey-Bass.
Petersilia, J. (1989). Implementing random experiments: lessons from BJA’s intensive supervision project.
Evaluation Review, 13, 435–458.
Rezmovic, E. L., Cook, T. J., & Dobson, L. D. (1981). Beyond random assignment: factors affecting
evaluation integrity. Evaluation Review, 5,51–67.
Roman, C., Fontaine, J., & Burt, M. (2009). System change accomplishments of the Corporation for
Supportive Housing’s Returning Home Initiative- Summary Brief. Washington, DC: Urban Institute.
Rossi, P., Lipsey, M., & Freeman, H. (2004). Evaluation: A systematic approach (7th ed.). Thousand Oaks:
Sage.
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for
generalized causal inference. Boston: Houghton Mifflin.
Sherman, L. W. (2007). The power few: experimental criminology and the reduction of harm. Journal of
Experimental Criminology, 3, 299–321.
Solomon, A. L., Osborne, J. W. L., LoBuglio, S. F., Mellow, J., & Mukamal, D. (2008). Life after lockup:
Improving reentry from jail to the community. Washington, DC: Urban Institute.
Weisburd, D. (2003). Ethical practice and evaluation of the interventions in crime and justice: the moral
imperative for randomized trials. Evaluation Review, 27, 336–354.
Wholey, J. S. (2004). Evaluability assessment. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.),
Handbook of practical program evaluation (2nd ed., pp. 33–62). San Francisco: Jossey-Bass.
Wolff, N. (2000). Using randomized controlled trials to evaluate socially complex services: prob-
lems, challenges and recommendations. The Journal of Mental Health Policy and Economics, 3,97–
109.
Caterina Roman joined the faculty in the Department of Criminal Justice at Temple University in Fall
2008 after nearly two decades with the Urban Institute. Her research interests include policy and program-
ming related to prisoner and jail reentry; the role of community organizations and institutions in crime
prevention and neighborhood well being; the effectiveness of community justice partnerships; and the
relationship between social networks, neighborhoods, and violence. She has been evaluating reentry
programs and studying reentry partnerships for almost two decades.
328 C.G. Roman et al.

Jocelyn Fontaine is a senior research associate in the Urban Institute’s Justice Policy Center, where her
research is focused primarily on evaluations of reentry initiatives and community-based crime initiatives.
Ms. Fontaine has a PhD in Justice and Public Policy from American University’s School of Public Affairs.
John Fallon is a Senior Program Manager at the Corporation for Supportive Housing working to create
supportive housing opportunities for those cycling through the criminal justice system and emergency
services. He has been involved in direct services for nearly 30 years including working with adolescents,
persons with mental illness, persons leaving correctional facilities, and those living on the streets.
Jacqueline Anderson Jacquelyn Anderson is Senior Program Manager for Policy Research at the
Corporation for Supportive Housing (CSH), where she is responsible for managing all research and
evaluation for the organization.
Corinne Rearer LCSW, holds the position of Supervisor of the Frequent Users of Jail and Mental Health
Project and Intake Coordinator at Trilogy Inc. She is a clinician and advocate for those with severe and
persistent mental illness within community mental health and correctional centers.
Defending the pipeline 329