Content uploaded by Samuel DeWitt
Author content
All content in this area was uploaded by Samuel DeWitt on Oct 12, 2022
Content may be subject to copyright.
SMART PROBATION GRANT: RISK BASED
SUPERVISION
ORGANIZATIONAL IMPACT EVALUATION
REPORT
RUTGERS UNIVERSITY SCHOOL OF
CRIMINAL JUSTICE
SEPTEMBER 30, 2016
Rutgers University School of Criminal Justice Page 1 of 17
Smart Probation Grant: Risk Based Supervision September 30, 2016
EXECUTIVE SUMMARY
Background
The Smart Probation Project, Risk-Based Supervision (RBS) program reallocates
probation resources in the Burlington, Mercer, and Passaic Probation Divisions through
the use of findings from an actuarially-based risk assessment instrument. The instrument
gauges the level of risk that probationers pose to public safety according to three
categories: Reduced (Low), Regular (Medium), and Close (High). Risk profiles are
subsequently used to inform caseload assignments for probation officers. Officers are
paired to certain risk categories (i.e., officers only supervise low, medium, or high risk
caseloads), and caseload sizes are intended to follow a natural progression of small
(maximum of 50 clients), normal (range of 90 to 120 clients), and large (maximum of
500 clients) in accordance with the respective high, medium, and low risk classifications.
The theoretical framework of the project is built upon the risk principal within the tenets
of effective correctional interventions: allocate a greater number / intensity of
correctional resources towards high risk clients that pose the greatest risk to public safety,
and dedicate relatively fewer resources to lower risk clients. The Rutgers University
School of Criminal Justice is the research partner for this project, and will use statistical
analyses of Probation’s data to explore key process, outcome, and organizational goals.
Purpose
This report communicates findings from the Rutgers University School of
Criminal Justice’s organizational impact evaluation of the Smart Probation Project, Risk-
based Supervision.
Objectives
The objectives of the organizational impact evaluation are to use Probation’s data,
as well as a survey of probation officers, to measure four key organizational impact
outcomes of the Smart Probation Project, Risk-based Supervision:
1. Did officers have sufficient time to supervise the high risk caseload?
2. Did officers have sufficient time to supervise the low risk caseload?
3. Were the risk score ranges correctly determined or adjusted?
4. Were the high and low risk caseloads maintained at the intended volumes?
Process
Explorations of the research questions entail measuring whether RBS program
probation officers were able to sufficiently supervise their caseloads given the shifts in
caseload sizes. A survey was developed and administered to probation officers to gauge
their views of how the RBS program impacted their workloads as well as their
impressions of what tasks took the most time to perform, and potential areas for
improvement. The survey administration began on June 27, 2016 and was closed on
Rutgers University School of Criminal Justice Page 2 of 17
Smart Probation Grant: Risk Based Supervision September 30, 2016
August 8, 2016. The survey was administered through a web-based program called
Qualtrics, and a link to the survey was emailed to all probation officers (both RBS
program officers and non-project officers). A total of 587 responses were received.
After cleaning the data for partial responses and duplicate submissions, a total of 394
viable responses were garnered. Of these responses, 74 officers indicated that they
worked in a county that was participating in the RBS program. Data from these 74
responses were used to answer the research questions and provide additional context. In
addition to the survey responses, probation data were analyzed to determine whether the
risk scores were correctly determined and whether, during the course of the project, the
high and low risk caseloads were maintained at the intended volumes. As it regards the
former, risk assessments were analyzed to examine how often they were overridden for
incoming clients under the new program. For the latter, monthly and quarterly counts of
unique clients per officer were estimated using all clients, including those who were
transitioned into the program in October 2013 from pre-existing probation terms. These
estimates were then compared to existing caseload standards for Reduced, Regular, and
Close supervision categories.
Findings
1. Did officers have sufficient time to supervise the high risk caseload?
Yes, for the most part. Survey results indicated that about 70 percent of high risk
caseload probation officers believed that they had a similar amount of time to supervise
their caseloads when compared to prior to the implementation of the RBS program.
Approximately 63 percent of the high risk caseload officers indicated that they had
enough time to adequately supervise their caseloads.
2. Did officers have sufficient time to supervise the low risk caseload?
Yes, for the most part. About 60 percent of the low risk caseload officers indicated
that they had less time to supervise their caseloads when compared to prior to the
implementation of the RBS program. However, 60 percent believed they had enough
time to supervise their cases despite having less time than they previously had.
3. Were the risk score ranges correctly determined or adjusted?
No. Survey results indicated that probation officers that participated in the RBS
program estimated that they or a supervisor adjusted the risk classification score for 35
percent of the probationers that were assessed. But, the reasoning behind the adjustments
did not strictly adhere to the tenets of the RBS program (e.g., “age made them riskier”,
“face to face experience”, “to make the results match what I had in mind”).
An analysis of CAPS data reaffirms the results from the surveys. In total, of 4,164
risk assessments completed throughout the course of the program 1,363 (32.7%) were
overridden in some manner. More specifically, a substantial number of overrides were
from “Regular” to “Close” (n=216) and from “Regular” to “Reduced” (n=489),
suggesting that officers perceive the risk score range for the “Regular” classification to be
too wide, encapsulating clients with too low and too high risks.
Rutgers University School of Criminal Justice Page 3 of 17
Smart Probation Grant: Risk Based Supervision September 30, 2016
4. Were the high and low risk caseloads maintained at the intended volumes?
Yes, for high risk; and no for low risk. Average estimates of quarterly caseload
sizes within risk category (Reduced, Regular, Close) suggest that most officers fall
within recommended caseloads for their assigned risk level. There are only a small
contingent of officers within each risk category (about 5%) that had caseloads that
exceeded the standards set forth within the program manual.
While high and medium risk officers maintained caseload sizes of around 40 and
90, which aligned well with the numbers set forth by the project, analyses of CAPS
data indicate that low risk caseloads were much lower than the volumes intended
through the program. Average caseload sizes for officers supervising “Reduced” risk
clients were in the teens or low twenties, this number of clients is significantly lower
than the maximum 500 that was allowable in the RBS program.
Recommendations
A recurring theme throughout the organizational impact assessment was that
probation officers felt that their caseloads were generally too high. Those officers that
participated in the RBS program indicated that caseloads were too high for the low risk
caseload; and high risk caseload officers indicated that violation rates were high, so much
of their time was spent in court. Data from the outcomes evaluation support the
contentions of the high risk caseload officers: high risk probationers were violated at a
rate of 30 percent compared to approximately 1 percent for low risk probationers.
Whether this is a factor of supervision being higher for high risk and resulting in higher
violation rates – and low risk probationers simply not being caught due to the high
caseloads of low risk officers is not clear.
However, results from the outcomes report also indicate that low risk probationers
were rearrested and reconvicted at substantially lower rates when compared to high risk
probationers. As these criminal justice actions are typically the province of the police
and other non-probation law enforcement officials, the RBS program appears to be a
meaningful way to decrease recidivism.
Probation should adopt a more meaningful risk assessment instrument that
communicates the criminogenic needs of probationers. This process should be paired
with a more nuanced analysis of risk category assignments (i.e., score cutoffs for risk
bands), so that the risk category assignments more closely align with the intended
caseload sizes. As indicated by the analyses of CAPS data, the risk assessment
instrument that was used in the RBS project classified few probationers as low risk,
which resulted in small caseload sizes for low risk officers. Many of these cases, in fact,
were the result of ex-post modifications to risk classifications. Simultaneously, the
agency should continue to gather data that can be used by researchers in order to connect
criminal justice outcomes to cases. Through this process, risk assessment instruments
can be normalized to the population of interest, and caseload size decisions and risk score
cutoffs that communicate low, medium, and high risk can be informed by outcomes
rather than clinical judgments.
Action Steps
This report concludes the Rutgers University School of Criminal Justice’s
analyses of the Risk-Based Supervision Pilot Project. Probation should ensure that
Rutgers University School of Criminal Justice Page 4 of 17
Smart Probation Grant: Risk Based Supervision September 30, 2016
similar data elements used in this report as well as the process and outcomes evaluations
continue to be gathered so that longer-term impacts of the Risk-Based Supervision Pilot
Project can be analyzed. From a policy perspective, the results from the reports
communicate that the project had positive impacts upon outcomes for probationers, was
followed according to plan, and did not have a deleterious impact on the workloads of
probation officers. Thus, it is advised that Probation expand the program to other
counties.
Rutgers University School of Criminal Justice Page 5 of 17
Smart Probation Grant: Risk Based Supervision September 30, 2016
INTRODUCTION
The following document evaluates the organizational impact of New Jersey
Probation’s (NJP) Risk-Based Supervision (RBS) program, focusing specifically on
whether officers had sufficient time to supervise their caseloads and whether risk scores
were correctly determined and maintained at intended volumes. Briefly, the RBS
reallocates probation resources in the Burlington, Mercer, and Passaic Probation
Divisions through the use of findings from an actuarially-based risk assessment
instrument. The instrument gauges the level of risk that probationers pose to public
safety according to three categories: low, medium, and high risk. Risk profiles are
subsequently used to inform caseload assignments for probation officers. Officers are
paired to certain risk categories (i.e., officers only supervise low, medium, or high risk
caseloads), and caseload sizes are intended to follow a natural progression of small
(maximum of 50 clients), normal (range of 90 to 120 clients), and large (maximum of
500 clients) in accordance with the respective high, medium, and low risk classifications.
The theoretical framework of the project is built upon the risk principal within the tenets
of effective correctional interventions: allocate a greater number / intensity of
correctional resources towards high risk clients that pose the greatest risk to public safety,
and dedicate relatively fewer resources to lower risk clients.
The organizational impacts of these revised supervision standards are the topic of
this evaluation. Specifically, the following sections will explore the following research
questions: 1) did the RBS program officers have sufficient time to supervise the high risk
caseload, 2) did the RBS program officers have sufficient time to supervise the low risk
Rutgers University School of Criminal Justice Page 6 of 17
Smart Probation Grant: Risk Based Supervision September 30, 2016
caseload, 3) were the risk score ranges correctly determined or adjusted, and 4) were the
high and low risk caseloads maintained at the intended volumes?
DATA COLLECTION
Two major data sources were used for these analyses: Probation’s CAPS data
management information system and a survey of all probation officers. Probation
provided the researchers with a dataset that contained information on all probationers that
were included in the Risk-based Supervision Pilot Project, as well as probationers that
were not supervised within pilot counties. Information about the risk levels of the
probationers was included as well as a JUID for the supervising officer. This allowed for
analyses of caseload sizes in relation to risk level. With this information, determinations
of whether caseload sizes were maintained according to the project’s intended volumes
could be explored.
The research team developed a survey to ascertain probation officer attitudes
towards the project. The survey included identifying information about whether
probation officers were program participants as well as the risk level of their caseload. In
addition, the survey gauged views about whether the officers had sufficient time to
supervise their caseload, what activities were the most time consuming, and what areas
should potentially be adjusted as the project moves forward. The data were analyzed
using the Stata statistical software package, and results are communicated through the use
of frequency distributions and mean response rates for a given question. The survey was
made available to all probation officers through a web-based administration system called
Qualtrics. The survey was open and able to be taken from June 27, 2016 to August 8,
Rutgers University School of Criminal Justice Page 7 of 17
Smart Probation Grant: Risk Based Supervision September 30, 2016
2016. A total of 587 responses were received. After cleaning the data for partial
responses and duplicate submissions, a total of 394 viable responses were used for the
analyses. Of these 394 responses, 74 officers indicated that they worked in a county that
was participating in the RBS program. Of the participating officers, 8 indicated that they
supervised a low risk caseload, 27 supervised a medium risk caseload, and 39 indicated
that they supervised a high risk caseload.
RESULTS
Did officers have sufficient time to supervise the high risk caseload?
Yes, for the most part. Approximately 70 percent of the officers that participated
in the RBS program and supervised a high risk caseload indicated that they had “the
same” amount of time to supervise their caseloads when compared to prior to the
initiation of the program. About 9 percent indicated they had “a lot more time” to
supervise their cases, an equal proportion indicated they had “more” time. Twelve
percent indicated they had “less” time and none of the high risk caseload officers
indicated that they had “a lot less” time to supervise their cases. When asked if they
believed they had enough time to supervise all of the probationers on their caseload, 33
percent of the high risk caseload probation officers indicated “definitely yes”; 30 percent
indicated “probably yes”; 18 percent indicated “probably not”; and 18 percent indicated
“definitely not”.
When high risk caseload probation officers were asked to consider all of their
obligations as an officer and indicate how appropriate they believed their caseload size to
be, 18 percent indicated “I have way too many probationers to supervise”; 26 percent
Rutgers University School of Criminal Justice Page 8 of 17
Smart Probation Grant: Risk Based Supervision September 30, 2016
indicated “I have slightly too many probationers to supervise”; 56 percent indicated “I
have an appropriate number of probationers to supervise”; and none indicated “I have too
few probationers to supervise”.
Did the officers have sufficient time to supervise the low risk caseload?
Yes, for the most part. Approximately 60 percent of the officers that participated
in the RBS program and supervised a low risk caseload indicated that they had “less”
time to supervise their caseloads when compared to prior to the initiation of the program.
About 20 percent indicated they had “more” time; 20 percent indicated they had “the
same” amount of time. None of the officers indicated that they had “a lot less time” to
supervise their cases. When asked if they believed they had enough time to supervise all
of the probationers on their caseload, 29 percent of the low risk caseload officers
indicated “definitely yes”; 29 percent indicated “probably yes”; 14 percent indicated
“probably not”; and 29 percent indicated “definitely not”.
When low risk caseload probation officers were asked to consider all of their
obligations as an officer and indicate how appropriate they believed their caseload size to
be, 33 percent indicated “I have way too many probationers to supervise”; 16 percent
indicated “I have slightly too many probationers to supervise”; 33 percent indicated “I
have an appropriate number of probationers to supervise”; and 16 percent indicated “I
have too few probationers to supervise”.
Rutgers University School of Criminal Justice Page 9 of 17
Smart Probation Grant: Risk Based Supervision September 30, 2016
Were the risk scores ranges correctly determined or adjusted?
No. Survey results indicated that probation officers that participated in the RBS
program estimated that they or a supervisor adjusted the risk classification score for 35
percent of the probationers that were assessed. However, the reasons behind the
overrides did not strictly coincide with the tenets of allowable overrides for the RBS
project. Overrides are allowed in situations where the dynamic risk score instrument
results indicate a change is needed or if the probationer a) is a current gang member; b)
has a criminal record more serious than the risk score indicates; c) has significant,
untreated mental health issues; d) has recent drug, gambling or alcohol abuse; e) has
recent probation or parole violations; or f) has an unverifiable residence or employment
information.
While most of the responses to the survey question “When the result of the risk
assessment was overridden, what are some examples of factors that, in your opinion,
justified the change?” coincided with the above factors that would justify an override,
several did not. For example, some responses included: “based on the individual’s
character”; “their age made them riskier”; “needed more treatment”; “level of education”;
“high profile case like Megan’s Law”.
Further, of the 4,164 risk assessments conducted over the course of the RBP
program (as indicated by CAPS data made available to the researchers) a full third (1,363
or 32.7%) of these risk classifications were overridden in some way. Though some of
these overrides were due to the client entering a secure facility (i.e., being classified as
“INC” or “INP”) these only account for a small minority of the total number of risk
classification overrides. Rather, “Regular” clients, as originally indicated by their risk
Rutgers University School of Criminal Justice Page 10 of 17
Smart Probation Grant: Risk Based Supervision September 30, 2016
score, appear to have their classification modified most frequently. Specifically, of the
1,363 overrides in the CAPS data, 216 (15.8%) stem from “Regular” clients being
overridden to the “Close” classification, and 489 (35.9%) “Regular” clients are
reclassified as “Reduced” risk, collectively accounting for over 50% of all overrides
made during the RBP program. This tells us that officers perceive the risk score range for
“Regular” classification as being too wide, encompassing clients that are evidently too
low or high a risk to be treated as a “Regular” risk.
Were the high and low risk caseloads maintained at the intended volumes?
Yes, for high risk; and no for low risk. An analysis of CAPS data indicates that
average caseload sizes within risk categories fell within or very near programmatic
standards for high and medium risk caseloads. However, low risk caseloads were quite
small in comparison to the intended caseload size of 500. These findings remained true
for most time periods within the RBP pilot project. See, for example Figure 1, which
plots average caseload sizes by quarter and within the risk classification officers are
responsible for (“Close”, “Reduced”, and “Regular”).
Rutgers University School of Criminal Justice Page 11 of 17
Smart Probation Grant: Risk Based Supervision September 30, 2016
Figure 1. Average Client Counts by Quarter
As indicated in Figure 1, most quarterly averages (estimated from monthly counts within
officers) fall below existing thresholds for caseload sizes. For example, in all quarters the
average caseload size for officers supervising “Close” clients did not exceed 40, while the
existing standard is 50 or less. Examining Figure 1, we can see that the highest average
caseload size for officers supervising “Close” clients was 39, which occurred in the first
quarter of 2015 (2015Q1). Further, while existing caseload standards for officers
supervising “Regular” risk clients is 90 to 120, most average client counts for these
officers fell under the accepted minimum in each quarter of the RBP program. As an
example, Figure 1 indicates that the only quarter where average caseload sizes exceeded
Rutgers University School of Criminal Justice Page 12 of 17
Smart Probation Grant: Risk Based Supervision September 30, 2016
the minimum was the first quarter of the program (2013Q4), where the average caseload
size was 91 clients.
However, caseload sizes for “Reduced” clients were the lowest, on average,
among all three groups – in no quarter did average caseload sizes exceed 25 clients (see,
for example, the bar for the 3rd quarter of 2014 (2014Q3) where the average is at its
highest – 21 clients). This finding is in stark contrast to the intentions that low risk
caseloads be the highest within the RBS program. According to the CAPS data, low risk
officers seemingly were paired with both the lowest risk probationers as well as the
lowest caseload sizes. This finding somewhat contradicts survey responses from officers
supervising “Reduced” risk clients, as a sizable proportion indicated that their caseloads
were too large to be managed (assuming that officers felt that the risk assessment
instrument accurately ascertained risk levels). In light of this, caseload sizes at the 95th
percentile are plotted in Figure 2, below.
Rutgers University School of Criminal Justice Page 13 of 17
Smart Probation Grant: Risk Based Supervision September 30, 2016
Figure 2. 95th Percentile Client Counts by Quarter
Figure 2 illustrates estimated client counts situated at the 95th percentile of all
officer client caseloads, meaning that 5% of officers have caseloads larger than those
depicted here within quarter, and by risk category. Here we see different results than
those depicted in Figure 1 that more closely align with officer perspectives on caseload
sizes as indicated by survey data. Specifically, it appears that average caseload sizes for
officers supervising “Reduced” risk clients were estimated as lower due to a relatively
large subsample of officers with very low caseload sizes (see, for example, the 1st quarter
of 2015, where officers at or above the 95th percentile of caseload sizes have at least 204
“Reduced” clients at a time).. Here, when we examine the 95th percentile, we see that
Rutgers University School of Criminal Justice Page 14 of 17
Smart Probation Grant: Risk Based Supervision September 30, 2016
caseload sizes can be substantially higher than average estimates, but they still fall within
an acceptable range in accordance with existing caseload size standards. Additionally,
opinions of “Close” client supervising officers are also, at least in part, identified here – it
appears that the upper 5% of officers serving “Close” clients have caseloads either
meeting the recommended maximum (50) or exceeding it. This is evident because most
bars for “Close” supervision officers come quite close to 50 in many quarters and sits at
50 in one quarter (Quarter 2, 2016). Finally, “Regular” client caseloads appear to fall in
line with existing standards, as 95th percentile values generally align with recommended
caseload sizes (90 to 120 clients).
CONCLUSION
The results from the process evaluation of the RBS program indicated that
probation officers and supervisors followed the tenets of the program with fidelity. Risk
assessments were completed in a timely manner, and almost all probationers had a
completed risk assessment instrument. The outcomes evaluation results indicated that the
RBS program had significant and substantial impacts on recidivism including new
arrests, convictions, and violations of probation. The organizational impact evaluation
results indicated that, while risk scores were not necessarily adjusted according to the
tenets of the project, officers indicated that the time they had to supervise probationers
was largely unchanged. Interestingly, low risk caseloads had the lowest number of
clients – a finding that ran counter to the intended volumes set forth by the RBS
designers.
Rutgers University School of Criminal Justice Page 15 of 17
Smart Probation Grant: Risk Based Supervision September 30, 2016
High risk officers spent much of their time in court processing violations of
probation. Low risk officers indicated that they felt their caseloads were quite high,
which resulted in not as close of supervision when compared to prior to the
implementation of the RBS program. However, this lower supervision for lower risk
probationers likely played a part in the lower rate of violations of probation for low risk
probationers. Low risk probationers were also rearrested and reconvicted at very low
rates. This is a promising result given that rearrests and reconvictions are typically the
province of other non-probation law enforcement agencies that would not have been
effected by the implementation of the RBS program.
Because the RBS program significantly reduced recidivism and officers
seemingly adjusted to the program quite well, as indicated through the results of the
process and organizational impacts evaluations, Probation should be encouraged to
expand the use of the program to other counties. During the course of expanding the
program, Probation should ensure that data are gathered that will allow for further
research about outcomes to be conducted. Upon adopting a new risk assessment
instrument to communicate risk profiles of clients, Probation should conduct more
nuanced analyses of risk category cutoff scores in order to more closely align with project
goals. While caseload sizes were all below the intended thresholds, low risk caseload
sizes were significantly divergent from the intended maximum of 500. The cutoff
decisions for the RBS project essentially entailed that probationers being assessed as low
risk would be rare events rather than common occurrences – which resulted in very low
caseload sizes for probation officers with supervisory responsibilities for low risk clients.
Rutgers University School of Criminal Justice Page 16 of 17
Smart Probation Grant: Risk Based Supervision September 30, 2016
By conducting a more nuanced analyses with Probation data, caseload sizes and risk
score cutoffs can be further modified in accordance with actuarial findings.
Rutgers University School of Criminal Justice Page 17 of 17
Smart Probation Grant: Risk Based Supervision September 30, 2016