Content uploaded by Carl B. Gacono
Author content
All content in this area was uploaded by Carl B. Gacono on Nov 28, 2020
Content may be subject to copyright.
SIS J. Proj. Psy. & Ment. Health (2021) 28: 01-00
Carl B. Gacono, PhD, ABAP, Private Practice, Asheville, NC, drcarl14@aol.com; Jason M. Smith, PsyD, ABPP, Adjunct Faculty,
Pierpont Community & Technical College, Fairmont, WV.
Key Words: Rorschach, R-PAS, Comprehensive System, Court testimony, assessment
Issues to Consider Prior to Using the R-PAS in a Forensic Context
Carl B. Gacono and Jason M. Smith
After the sudden death of John Exner (developer of the Comprehensive System; CS; Exner, 1969, 2003), a new scoring
system, the Rorschach Performance Assessment System (R-PAS; Meyer, Viglione, Mihura, Erard, & Edberg, 2011) was
introduced. While the CS was developed with extensive research over approximately 40 years, this new system has only been
studied for approximately fifteen years. When considering its forensic usage, it is essential to understand that CS validity
research does not translate directly to R-PAS. One primary reason for the lack of translation is the differing administration
procedures. Other factors such as the absence of comparison descriptive group data (non-patient, clinical, & forensic) and
internal validity problems with studies utilized in the primary meta-analyses which supports the R-PAS variable selection
should be considered in determining the methods admissibility. In this article, we discuss essential issues to consider prior to
using the R-PAS in an applied context.
After the premature death of John Exner
(developer of the Comprehensive System; CS;
Exner, 1969, 2003), a new scoring system, the
Rorschach Performance Assessment System (R-
PAS; Meyer, Viglione, Mihura, Erard, & Edberg,
2011) was introduced. While the CS was
developed with extensive research over
approximately 40 years, this new system has only
been studied for around fifteen years. For several
reasons CS validity research does not translate
directly to the R-PAS. A primary reason for the
lack of translation is the differing administration
procedures.
The R-PAS has been criticized related to its
administration procedures (Khadivi, & Evans,
2012; Mattlar, 2011; Ritzler & Sciara, 2010;
Smith, 2012), the lack of descriptive comparative
data (this article), its use of the international
Rorschach norms (Andronikof, 2019; Ritzler &
Sciara, 2009; Sciara & Ritzler, 2018), the
declaration that it has no cross cultural differences
(Andronikof, 2019; Fontan, 2014, 2019; Fontan &
Smith, 2018), problems with the indices used (i.e.,
Complexity, Suicide Constellation; Fontan, 2014;
Fontan & Smith, 2018), and the lack of forensic
admissibility (Gurley, Piechowski, Sheehan, &
Gray, 2014; Kivisto, Gacono, & Medoff, 2013). At
this time, these criticisms have not been adequately
addressed.
Altered Administration Procedures:
R-PAS administration procedures have created a
“Rorschach” distinct from the CS Rorschach.
Reportedly, Meyer et al. (2011) conducted initial
research on this new procedure and stated the
following reasons to accept their new
administrations: 1) reduced short and long records
(smaller SD for R), 2) eased the administration
task (stating no need to re-administer), 3) very
little effect on variables, 4) better fit for protocols
to the norms, and 5) potential for less examiner
variation across sites. R-PAS administration
requires cards to be “pulled” after four responses.
This procedure allows for 40 (range 20-40) to be
the highest number of responses possible.
The need for limiting lengthy protocols, is quite
likely a straw person “concern.” In administering
hundreds of Rorschachs, we have never had more
than about 1% that required the CS instructions
related to lengthy protocols, and not many more
that were too “short” (4%). While rates for
experienced clinicians who routinely administer
the CS Rorschach deserves study, this does not
seem to be a legitimate reason for modifying the
CS’ well thought out administration procedures1.
As noted by Gacono and Meloy (1994) related to
assessing antisocial and psychopathic subjects:
We have found, contrary to our expectations,
that psychopaths generally produce a normative
amount of Rorschach responses, at least in a
research setting. We have purged our group
samples of protocols with less than 14 responses
without great heartache, because they were few
in number (p. 5)2.
Though Exner (2003) slightly altered the
administration through different versions of the CS
(and the five American Rorschach systems prior to
00: Gacono & Smith
the CS: Klopfer, Beck, Hertz, Piotrowski, and
Rapport-Schafer), his administration instructions
were based on an assimilation of the best
procedures from the five systems. Exner (1974)
stated the CS “closely parallels that of the
traditional systems … [and] it is designed to
provide the Rorschach community with a common
language and a common methodology” (p. 17).
The R-PAS procedures, on the other hand, were,
based primarily on concerns with the problems
caused by the number of responses influencing the
“distributions of other Rorschach variables and
confounds their interpretation” and a comparison
to the CS instructions (Meyer et al., 2011, p. 409).
Specifically, “R-PAS was developed with one of
its aims being to better manage the number of
responses given to protocols, and thus improve its
psychometrics” (Meyer et al., 2020, p. 144).
Managing R may or may not improve the general
psychometrics (reliability) of the Rorschach. In
some cases, it may decrease its validity.
Rather than just the open-ended CS instruction of
“What might this be?” the R-PAS also includes
these implicit instructions “Try to give two
responses… or maybe three, to each card. That is,
for each card try to see two different things;
possibly three” (Meyer et al., 2011, p. 29). This
directive approach is a significant departure from
the open-ended CS instructions. The impetus of the
R-PAS instructions is to “control” or “manage”
rather than to facilitate a client directed process.
The existing Holtzman inkblot test provides an
alternative Inkblot method for examiners who
prefers a more directive, “controlled” approach.
Ironically, in their CS validity meta-analyses,
Mihura, Meyer, Dumitrascu, and Bombel (2013),
endorse the impact that different administration
procedures have on data production as they
“excluded results derived from studies employing
Rorschach test administration procedures that
markedly deviated from standard procedures” (p.
560). R-PAS authors have acknowledged the need
to have uniform Rorschach administration
procedures (Meyer et al., 2020). Yet, the
implications of this concern are seemingly ignored
when utilizing statistically manipulated CS
administered protocols for promoting the R-PAS
validity.
Statistics and meta-analyses have been utilized to
justify an absence of meaningful differences
among Rorschach administration procedures
(Hosseininasab et al., 2019; Meyer et al., 2020;
Pianowski, Meyer, de Villemor-Amaral, Zuanazzi,
& do Nascimento, 2019). These studies have found
some statistical differences which were largely
ignored when statistical comparisons of select
variables did not produce significantly different
quantities. It should be noted that six studies in the
meta-analysis did not use exact R-PAS
administration manual guidelines (Hosseininasab
et al., 2019; Meyer et al., 2020). Also statistics
deal with ranges and probabilities and fail to
capture, when non-patient protocols are utilized,
the multitude of differences that become apparent
when the Rorschach is used in clinical samples.
This is one reason why understanding the history
of a variable’s development is essential to selecting
variables for inclusion in any system as opposed to
using a statistical chopping block.
A study by Hartmann (2001) altered the Exner
administration by providing longer instructions
prior to administering the test. The instructions
were:
I am going to show you some pictures. There are
10 in all, and none of them resemble anything in
particular. They are made by putting ink on a
piece of paper, folding it, and pressing the sides
together, so that they become figures. Now I
would like you to tell me what they look like,
remind you of, or make you think of. There is no
right or wrong answers. Different people see
different things, and I want to know what you
think they might resemble. Tell me everything
you see in each picture, and when you have
finished with a card, you may put it down
(Hartmann, 2001, p. 462).
At no time did Hartmann instruct the examinee to
give a certain amount of responses a la R-PAS
administration. In response to this lengthy
introduction (instructions) fewer brief protocols
were produced when compared to the standard
Exner instructions (“What might this be?”);
however, she found no significant group
Rorschach variable differences between the two
instructions. Pianowski et al. (2019) looked at two
groups of non-patient protocols and found no
differences in CS Rorschach variables when
Using the R-PAS: 00
administering protocols with the CS (N = 50) or R-
PAS (N = 50) administration methods. In this
study, 53 statistical analyses were done between
Rorschach variables from an R-PAS and CS
administration. They found that a CS
administration had higher means for FQ-% and the
R-PAS equivalent of the Affective ratio and lower
means on R and FQu% compared to an R-PAS
administration. Three other differences were found
on variables used only in R-PAS. Instead of
specifically hypothesizing what could be different
in the two administrations, the large number of
comparisons may have resulted in lost data.
Statistical approaches absent the historical
rationale (sound theory) utilized in developing the
CS instructions are limited by many factors.
Significant differences can manifest in clinical
samples that were not evident with non-patients.
The extent to which alterations in administration
procedures impact results in a range of clinical
samples, nor the impact of altered administration
on important interpretative approaches such as
sequence analysis, have not yet been fully assessed
(see Meloy, Acklin, Gacono, Murray & Peterson,
1997).
Directive versus opened ended administration
instructions initiate distinctly different cognitive
test taking sets which can interact with a given
personality style in creating significantly different
results. The validity of the Rorschach rests on its
ability to elucidate different variable pattern of
variables dependent on the patients’ clinical issues
and personality style. How might an individual
with a perfectionistic style be impacted by the R-
PAS instructions? Does the direct instructional set
create a template for judging their performance?
What is the impact on a paranoid individual who
now has a template for responding? Does a
directive approach reduce ambiguity and
subsequently impact their anxiety (Y, m)? Will it
negate the need for them to end Cards with a Dd; a
pattern where the stimulus field is narrowed to
make the previously ambiguous, testing experience
manageable? Will the Obsessive-Compulsive
person feel pressure to conform to the demands of
the R-PAS instructions? Or will the structure
reduce their anxiety? While the answers to these
questions are unknown, what is known is that the
CS open ended administration procedures have
weathered the test of applied usage, and “work”
effectively in eliciting valuable patterns consistent
with the theoretical understanding of a multitude of
clinical and non-clinical populations (Gacono,
2019; Mattlar, 2011; Ritzler, 2014; Smith, Gacono,
Fontan, Cunliffe, & Andronikof, 2020; Smith et
al., 2018). The impact of a more directive approach
has not yet been demonstrated.
The Rorschach’s sensitivity to variations in
administration, such as altered lighting, using dirty
cards, seating position, etc. has also been
demonstrated (Exner, 2003). There is a plethora of
social science literature concerning the impact of
instructional sets on subsequent performance
(McDaniel, Hartman, Whetzel, & Grubb, 2007;
McDaniel & Nguyen, 2001). One example from
Gacono and Meloy (1994) demonstrates how
changing even a single word can significantly
impact results. In her dissertation, Margolis (1992)
attempted to replicate Gacono and Meloy’s (1994)
list for scoring AgC. She, however, changed the
original research instructions from “Please rate
those objects popularly perceived as predatory,
dangerous, malevolent, injurious, or harmful”, to,
“Please rate those objects you perceive to be
predatory, dangerous, malevolent, injurious or
harmful” (p. 273). By doing so, she evoked a
distinctly different cognitive set in the survey
participants. Gacono and Meloy’s (1994) choice of
popular was intentional and meant to be
ambiguous. It allowed subjects to rate objects as to
what most or many people consider to be
aggressive. Respondents could externalize
responsibility for their choice, be less likely to
assume their answers has any relationship to their
own aggression (essential in non-patients where
aggression is likely ego-dystonic) and would
decrease any concerns about being judged for their
answers. Not surprisingly, Margolis’ altered
instructions produced different results. Statements
by the respondents supported the presence of
censoring, “other(s) might see these words as
being aggressive, I don’t see them in that way,”
and “I must be really aggressive because I checked
a lot of the words (Margolis, 1992, p. 41; Gacono
& Meloy, 1994, p. 273). In a similar vein
comments made by clients taking the R-PAS, “Oh
wait, you said I have to give two or three
responses” support the contention that a distinct
00: Gacono & Smith
and different (from CS) cognitive set is elicited by
R-PAS instructions.
Several early studies examined the impact of
altered Rorschach administration procedures on
Rorschach production. Blais, Norman, Quintar,
and Herzog (1995) found the CS administration
showed more color, blends, and shading responses
than a Rapaport Rorschach administration.
Bornstein, Bonner, Kildow, and McCall (1997),
studied an undergraduate population and found no
differences between individual and group
administered Rorschach protocols; however, they
only focused on their non-CS derived ROD score.
Might their results have been different in clinical
samples? How about psychopathic offenders who
might be motivated to conform (females) or rebel
(males)? In considering our recent study of
psychopathic women, any instructional set that
provides a template for comparing one’s
performance is likely to increase their need to
interact with others to aid in regulating emotions
and issues with self-worth that manifest during the
Rorschach administration3. Would these altered
instructions create another point from which they
can evaluate themselves negatively? Would it
increase the production of PERs, Cgs, projection,
and other indications of defensiveness? These
issues have not been adequately addressed by the
R-PAS research. The meaning from the many
years of study (with links to theory) that went into
developing the CS administration procedures
seems to have been ignored. This is just one of the
common fallacies (bias) in psychology research
that stems from a failure to understand fully, or
even ignoring, the history of what precedes one’s
current “discoveries” (see Chapter 2 in Smith,
Gacono, & Cunliffe, 2021).
The R-PAS lacks adequate descriptive group data:
Currently, the normative data for R-PAS consists
of statically manipulated CS administered
protocols (Meyer et al., 2011). A statistical
procedure that has been called R-Optimized. This
method intends to mimic how a Rorschach
protocol would look if it were administered using
R-PAS instructions. This method is in contrast to
developing descriptive group data based on using
R-PAS administration procedures.
The R-PAS manual indicates that the International
CS non-patient normative data was utilized
(Meyer, Erdberg, & Shaffer, 2007; N = 1396), with
a statistical modelling procedure, to mimic a non-
patient normative sample that was administered
with R-PAS guidelines (N = 640). Sciara and
Ritzler (2018) also discussed cautionary notes
about the international sample itself:
(a) the use of inexperienced examiners; (b) the
ways sampling might affect generalizability (i.e.,
questionable selection procedures, insufficient
exclusion criteria, inadequate sample size); and
(c) the use of less than ideal administration
procedures, including the lack of a warmup
phase as part of the administration (p. 343).
In order to model the normative data to look like
R-Optimized administration protocols 123
protocols administered by students primarily under
Don Viglione’s supervision using R-PAS
administration of patients and non-patients were
used in the statistical procedure. This sample
consisted of “30 non-patients, 30 chronic but stable
schizophrenia patients, 15 adult and 15 adolescent
sex offenders, and 33 outpatients” (Meyer et al.,
2011, p. 470). Of note, however, there are no
comprehensive descriptive data sets of R-PAS
administered normative, clinical, or forensic
samples.
This can be contrasted with descriptive data
contained in The Rorschach Workbook for the
Comprehensive System (4th & 5th edition; Exner,
1995, 2001) which includes descriptive CS data for
Nonpatient Adults (N=700; also listed by
Introversive, Extratensives, & Ambitents),
Nonpatient Children and Adolescents (N=1390;
also listed by age), Inpatient Schizophrenics
(N=320), Inpatient Depressives (N=315), and
Character Disorders (N=180). Exner (2007) also
provided a new 450 non-patient normative sample.
Directly related to using the Rorschach in a
forensic context, descriptive data is available for
Conduct Disordered Children (N=72; Gacono,
Gacono, & Evans, 2008), Conduct Disordered
Adolescents (N=179; Gacono et al., 2008),
Antisocial Personality Disordered Women
(N=232; Smith et al., 2021), Female Psychopaths
(N=115; Smith et al., 2021), Antisocial Personality
Disordered Males (N=108; Gacono et al., 2008),
Non-Sexually Offending Psychopathic Males
Using the R-PAS: 00
(N=32; Gacono, Meloy & Bridges, 2008), Non-
violent Male Pedophiles (N=39; Gacono, Meloy &
Bridges, 2008), Sexual Homicide Perpetrators
(N=38; Gacono, Meloy & Bridges, 2008), Sexually
Offending Females (N=39; Smith et al., 2021),
Antisocial Disordered Schizophrenic Men (N=100;
Gacono et al., 2008), Inmates in Forensic
Psychiatric Treatment (N=211; Young, Erdberg, &
Justice, 2008), three groups of Forensic
Outpatients (paranoid schizophrenics N=90,
undifferentiated schizophrenics, N=38, Bipolar
patients N=40; Gacono & Gacono, 2008) and
Child Custody Litigants (N=728, also presented by
gender, Singer, Hoppe, Lee, Olesen, & Walters,
2008).
This translates to 2061 forensic protocols
administered with CS procedures (see also Gacono
& Evans, 2008; Gacono & Meloy, 1994;
Piotrowski, 1996a, 1996b, 1996c, 2007, 2017).
The R-PAS has two forensic samples (N=30), 15
adult and 15 adolescent sex offenders (Meyer et
al., 2011). Forensic “case” studies, not a substitute
for comparative descriptive data, were provided in
edited texts by Erard and Evans (2017) and Mihura
and Meyer (2018).
Issues with Mihura et al. (2013) Meta-analyses:
The Mihura et al. (2013) meta-analyses played a
significant role in determining which variables
would be included in the R-PAS. The following
guidelines guide the inclusion of a variable in R-
PAS if 1) the notion that there needed to be a
conceptual association between the code and the
interpretation, 2) parsimony, and 3) clinical
experience based on experienced practitioners
(Meyer et al., 2011). These meta-analyses
contained 210 studies; however, not all were used
to demonstrate validity for the CS variables.
Specifically, Mihura stated, “associations between
the Rorschach and self-report measures were not
used to determine the validity of Rorschach
variables” because “Rorschach studies using self-
report measures for validity criteria are typically
published in less rigorous journals” (i.e., journal
impact factor, Mihura, 2019, p. 171).
When examining all 210 studies within the Mihura
et al. (2013) meta-analyses using five criteria
(IQ/Education level, R, Lambda/F%, inter-rater
reliability, sample size), every article had
methodological or internal validity issues that
made a full interpretation of their findings
problematic (Smith et al., 2018). Four articles only
had one issue (1.9%), 17 had two (8.1%), 85 had
three (40.5%), 87 had four (41.7%) and 17 had five
(8.1%). For example, 51.4% did not provide any
statistics for R, 67.1% did not provide this data for
Lambda/F%, 9.5% used IQ scores < 80, 34.2% did
not include an inter-rater reliability statistic, and
26.7% had a sample size < 20.
Twenty-two (10.5%) of the studies included
protocols with less than 14 responses. These older
CS Rorschach studies with shortened protocols
(current CS guidelines consider that < 14 response
protocols may be statistically invalid/unreliable)
were included in the Mihura et al. (2013) meta-
analyses (Smith et al., 2018). Besides the problem
with including low R studies in this type of
analyses it should be noted that R-PAS instructions
would preclude these protocols from being
produced (push for 2 and pull after 4; Meyer et al.,
2011).
Smith et al. (2018) also found that of the 210
studies in the Mihura et al. (2013) meta-analyses,
only 104 of the 210 (49.5%) studies were research
on the Rorschach (validation studies). None met all
five Rorschach criteria assessed (appropriate inter-
rater reliability, R, Lambda, sample size, &
IQ/educational level). Internal validity issues with
90% of the studies examined had three or more
issues related to the above criteria. Without this
information, one cannot determine whether the
statistical procedures in the studies were
appropriate and/or whether the findings can be
generalized beyond the sample studied.
Contradictions were also noted in how the meta-
analyses results were used. For example, the HVI
variable was included in R-PAS with little support
found in the meta-analyses while the DEPI which
had moderate support in the meta-analyses was
excluded from R-PAS. Of note, the CS Human
Experience score (Hx) was removed from R-PAS
based on a statistical chopping block. We found
this to be an important indicator of primitive
hysteria in psychopathy (female offenders M =
0.84, 35%; female psychopaths M = 1.03, 38%;
male offenders M = 0.15; 11%; male psychopaths
M = 0.18, 12%, Chapter 3 in Smith et al., 2021).
00: Gacono & Smith
This suggests there is some clinical utility for this
variable.
Failure to adequately address critics of the R-PAS:
Criticisms of R-PAS have generally been met with
statistical analyses that fail to address the
conceptual points raised (Smith et al., 2020).
Applying statistics without putting them into
context is never justified (i.e., Simpson paradox). It
may lead to large effect sizes and statistical
significance, but the results can still be
meaningless (lacking in practical significance and
logical soundness; see Chapter 2 in Smith et al.,
2021; Smith et al., 2020). This over-evaluation of
statistical prowess at the expense of conceptual
understanding has resulted in the tail (statistics)
wagging the dog (sound conceptual driven
methodology; Gacono 2019). It can lead to
important variables being eliminated without
adequate consideration of their historical value
within certain clinical populations. There is some
truth to the notion that, “There are three kinds of
lies: lies, damned lies, and statistics” (unknown).
Or, as a colleague, Taka Endo, has stated, “The
study of statistics is a method to measure if we are
in the right direction not a method of finding the
truth.” Perhaps, Sigmund Freud said it best,
“Those critics who limit their studies to
methodological investigations remind me of people
who are always polishing their glasses instead of
putting them on and seeing with them” (in Reik,
1956, p. 54).
In the interest of clarifying some of the issues with
the R-PAS, we requested that the data from the
Mihura et al. (2013) study be shared with us so that
we might verify the hypothesized Rorschach
predictor–criterion associations database, validity
coefficients decisions, and interrater reliability for
the predictor– criterion associations (Mihura et al.,
2013). As Mihura noted in her commentary (2019,
p. 179), in the spirit of ethical guidelines and
scientific cooperation, we provided her with our
data from Smith et al. (2018) for her review and
analysis. However, we were denied access to her
meta-analyses data. Mihura stated “I (the first
author) am using some of the secondary data in
new analyses and—in some cases (not all)—it
would be difficult or impossible to accurately trace
these databases back to their original format and
reconfigure a new database.” She also stated it was
more than five years after publication and that the
data we requested would be explained in her reply
(it was not; Joni Mihura, Personal Communication,
March 17, 2019). This first point is of significant
concern--that data used in a significant way to
support and promote R-PAS is no longer available.
Unlike the issue with sharing the original CS
protocols, where there are confidentiality issues,
the data from the Mihura et al. meta-analyses have
no such constraints.
Finally, the lead creator of R-PAS (Meyer) has
discussed similar issues related to other
researchers. Meyer (2000) has been critical of
“arguments … designed to convince readers of a
conclusion, regardless of its accuracy” (p. 77) and
stated,
[others] were not sufficiently careful in their
scholarship (e.g., the erroneous citations), were
not aware of some key literature on a topic (e.g.,
the composite variables), presented a limited and
slanted portrayal of relevant issues and evidence,
… and repeatedly dismissed corrective feedback
(p. 77).
Meyer also stated,
As the Rorschach evidence base continues to
grow and develop, sound and balanced criticism
of the literature will help advance scientific
knowledge and applied practice. Conversely,
publishing assertions that are known to be wrong
or misleading can only serve political purposes
that thwart the goals of science and retard
genuine evolution in the field (2000, p. 78).
In addition the above issues, Meyer et al. (2020)
have now advocated that during the COVID-19
pandemic, it is acceptable to administer the
Rorschach virtually or remotely using the above
mentioned Rorschach administration studies,
supported by three dissertations on virtual
Rorschach administrations (one dissertation was
not complete; Berry, 2020; Logid, 2018;
Whitehouse, 2019). The numerous modifications
to standardized procedures that they are needed to
do this make it inappropriate to administer the R-
PAS virtually4. Remote Rorschach administration
is also likely unethical (CSIRA/ARISI, 2020). The
results have certainly not been proven to be valid
(CSIRA/ARISI5, 2020). It should be noted that two
prominent R-PAS creators, Meyer and Viglione
(2008), have previously stated that the “standard
Using the R-PAS: 00
CS test administration does not lend itself to
automated, computer-adapted administration or to
computer automated scoring” (p. 306) in direct
contradiction to offering or suggesting virtual
administration. This represents another
inconsistency between ethical and competent
practice and the promotion of R-PAS.
The above information offers a blueprint for
challenging the R-PAS should it be used in any
forensic context.
1. There are a number of criticisms of R-PAS that have
not been as of yet adequately addressed. These
include its altered administration procedures
(Khadivi, & Evans, 2012; Mattlar, 2011; Ritzler &
Sciara, 2010; Smith, 2012), its use of the
international Rorschach norms (Andronikof, 2019;
Ritzler & Sciara, 2009; Sciara & Ritzler, 2018), the
declaration that it has no cross cultural differences
(Andronikof, 2019; Fontan, 2014, 2019; Fontan &
Smith, 2018), problems with the indices used (i.e.,
Complexity, Suicide Constellation; Fontan, 2014;
Fontan & Smith, 2018), and the lack of forensic
admissibility (Gurley et al., 2014; Kivisto et al.,
2013).
2. The meta-analysis on R-PAS administration
included studies with instructions different from
those found in the R-PAS manual (Hosseininasab et
al., 2019; Meyer et al., 2020).
3. The R-PAS altered “directive” administration
procedures to create a different test than the CS
“open ended” administration procedures.
Consequently, CS validity does not translate directly
to the R-PAS.
4. The R-PAS validity lacks the necessary R-PAS
administered comparative descriptive data to be used
clinically or forensically. Other than 15 adult and 15
adolescent sex offenders and a few forensic case
studies, no forensic samples exist from R-PAS
administered protocols.
5. The majority (90%) of the Mihura et al. (2013) CS
meta-analyses studies had internal validity issues
(Smith et al., 2018, 2020) such as studies did not
include IQ/Educational level, did not report R or
Lambda statistics, had Rorschach protocols < 14, did
not report inter-rater reliability, and had sample sizes
less than 20.
6. Statistical approaches used by R-PAS proponents
fail to address the conceptual issues raised by R-PAS
critics (Smith et al., 2020).
7. The data from the Mihura et al. (2013) meta-
analyses that has been used to promote R-PAS is,
according to Mihura, no longer available in “their
original format.”
8. Administering the R-PAS virtually is likely to be
unethical (CSIRA/ARISI, 2020).
Given the weight of the above issues, the applied
usage of R-PAS is premature and will likely add
legitimate grounds for critics of the Rorschach to
devalue the test (Piotrowski, 1996a, 1996b, 1996c,
2007, 2017). It will aid in creating one more “straw
person” or “apparent” controversy that detractors
will use to discredit the Rorschach Method and
further influence practitioners to avoid using the
Rorschach in forensic contexts. Rather than
discarding the use of the Rorschach and the
valuable information it provides, however, we
recommend that psychologists continue to use the
CS in clinical and forensic contexts and increase
their sophistication for presenting testimony
(Piotrowski, 2007; Wiener, 2008). This is really
not a difficult task (Wiener, 2008).
As outlined by Andronikof (2019, p. 9) “a) the
validity of the CS has a solid evidence-base which
has not been disproved, b) the attempt to discredit
its creator is produced by promoters of a
competing system, [and] c) the R-PAS, although
founded on Exner’s work, deviates from and
betrays Exner’s interpretation principles and is
therefore not a continuation or development of the
CS.” It should also be noted that while many of the
CS Rorschach critics of the past two decades were
individuals who were not experts in assessment,
the Rorschach, or even clinical practice, the
majority of those who now criticize R-PAS are
Rorschach experts; many of whom have conducted
original Rorschach research.
1An “experienced clinician” might be defined as a licensed
psychologist who has administered several hundred Rorschach
protocols in applied contexts over a reasonable length of time
(say 10 years). It would not include student administrators
working only in a research setting. Proper rapport frequently
reduces the number of protocols that might conclude with < 14
responses.
2While “at least in research settings” was included in this
statement, most of the protocols throughout the forensic
samples where administered in clinical contexts and then
included as archival data (Gacono & Evans, 2008). A more
accurate statement would be “at least within our forensic
samples.”
3We have found these women (N=115) to manifest primarily
extratensive and ambitent styles (Smith et al., 2021).
00: Gacono & Smith
4 i.e., size of the image, ability to turn the card/image, needing
the examiner to see what the person is seeing, possibility using
a layperson to help administer, etc.
5CS International Rorschach Association/Association
Rorschach Internationale Système Intégré; https://www.csira-
arisi.org/
References:
Andronikof, A. (2019). What future for the
Comprehensive System? Paper presented at the
3rd Congress of the Comprehensive System
International Rorschach Association
(CSIRA/ARISI).
https://www.researchgate.net/publication/33774
1439
Berry, B. A. (2020). The effects of manipulating visual
processes on Rorschach performance,
Unpublished doctoral dissertation. University of
Toledo, Toledo, OH.
Blais, M. A., Norman, D. K., Quintar, B., & Herzog, D.
B. (1995). The effect of administration method:
A comparison of the Rapaport and Exner
Rorschach systems. Journal of Clinical
Psychology, 51(1), 100–107.
Bornstein, R. F., Bonner, S., Kildow, A. M., & McCall,
C. M. (1997). Effects of individual versus group
test administration on Rorschach Oral
Dependency scores. Journal of Personality
Assessment, 69, 215–228.
CSIRA/ARISI. (2020). CSIRA position paper on
psychological tele assessment.
https://www.csira-arisi.org/
Erard, R. E., & Evans, F. B. (Eds.). (2017). The
Rorschach in multimethod forensic assessment:
Conceptual foundations and practical
applications. Routledge/Taylor & Francis Group.
Exner, J. E. (1969). The Rorschach systems. New York:
Grune & Stratton.
Exner, J. E. (1974). The Rorschach: A Comprehensive
System (1st ed.). New York: Wiley.
Exner, J. E., Jr. (1995). A Rorschach workbook for the
Comprehensive System (4th ed.). Asheville, NC:
Rorschach Workshops.
Exner, J. E., Jr. (2001). A Rorschach workbook for the
Comprehensive System (5th ed.). Asheville, NC:
Rorschach Workshops.
Exner, J. E. (2003). The Rorschach: A Comprehensive
System (4th ed.). New York: Wiley.
Exner, J. E. (2007). A new US adult nonpatient sample.
Journal of Personality Assessment, 89(Suppl. 1),
S154–S158.
Fontan, P. (2014). Approche Psychométrique du test de
Rorschach, Unpublished doctoral dissertation,
Université Paris Ouest Nanterre, Paris, France.
Fontan, P. (2019). The Future of the Rorschach test.
Journal of Japan Rorschach Society for the
Comprehensive System, 23(1), 2-8.
Fontan, P., & Smith, J.M. (2018). A Critical Review of
R-PAS, Symposium conducted at the Society for
Personality Assessment at the 2018 Society of
Personality Assessment Annual Convention,
Washington DC, USA.
https://www.researchgate.net/publication/32874
3530
Gacono, C.B. (2019). The importance of Lambda to the
generalizability of Rorschach findings reported
in the literature. SIS Journal of Projective
Psychology & Mental Health, 26(2), 104-106.
Gacono, C. B., & Evans, F. B. (Eds.). (2008). The LEA
series in personality and clinical psychology.
The handbook of forensic Rorschach
assessment. (N. Kaser-Boyd & L. A. Gacono,
Collaborators). Routledge/Taylor & Francis
Group.
Gacono, C. B., Gacono, L. A., & Evans, F. B. (2008).
The Rorschach and antisocial personality
disorder. In C. B. Gacono, F. B. Evans (Eds.) &
N. Kaser-Boyd, L. A. Gacono (Collaborators),
The LEA series in personality and clinical
psychology. The handbook of forensic
Rorschach assessment (p. 323–359).
Routledge/Taylor & Francis Group.
Gacono, C.B., & Meloy, J.R. (1994). The Rorschach
assessment of aggressive and psychopathic
personalities. Hillsdale, NJ: Erlbaum.
Gacono, C. B., Meloy, J. R., & Bridges, M. R. (2008). A
Rorschach understanding of psychopaths, sexual
homicide perpetrators, and nonviolent
pedophiles. In C. Gacono, B. Evans, N. Kaser-
Boyd, & L. Gacono (Eds.), The handbook of
forensic Rorschach assessment (pp. 379–402).
Mahwah, NJ: Lawrence Erlbaum.
Gacono, L. A., & Gacono, C.B. (2008). Some
considerations for the Rorschach Assessment of
forensic Psychiatric outpatients. In C. B.
Gacono, F. Evans, N. Kaser-Boyd, & L. Gacono
(Eds.), The Handbook of Forensic Rorschach
Using the R-PAS: 00
Assessment (pp. 421-444). New York, NY US:
Routledge/Taylor & Francis Group.
Gurley, J. R., Sheehan, B. L., Piechowski, L. D., &
Gray, J. (2014). The admissibility of the R-PAS
in court. Psychological Injury and Law, 7(1), 9–
17. https://doi.org/10.1007/s12207-014-9182-2
Hartmann, E. (2001). Rorschach administration: A
comparison of the effect of two instructions.
Journal of Personality Assessment, 76(3), 461–
471.
Hosseininasab, A., Meyer, G.J., Viglione, D.J., Mihura,
J.L., Berant, B., Resende, A.R., Reese, J., &
Mohammadi, M.R. (2019). The effect of CS
administration or an R-Optimized alternative on
R–PAS variables: A meta-analysis of findings
from six studies. Journal of Personality
Assessment, 101(2), 199-212,
DOI:10.1080/00223891.2017.1393430
Khadivi, A., & Evans, F. B. (2012). The brave new
world of forensic Rorschach assessment:
Comments on the Rorschach special section.
Psychological Injury and Law, 5(2), 145–149.
doi:10.1007/s12207-012-9134-7
Kivisto, A. J., Gacono, C. B., & Medoff, D. (2013).
Does the R-PAS meet standards for forensic
use? Considerations with introducing a new
Rorschach coding system. Journal of Forensic
Psychology Practice, 13(5), 389-410.
Logid, G. M. (2018). Presentation of Rorschach Inkblot
stimuli via the iPad®: An equivalence
investigation. Unpublished doctoral dissertation,
California School of Professional Psychology,
San Diego, CA.
Margolis, J. (1992). Aggressive and borderline-level
content on the Rorschach: An exploratory study
of some proposed scoring criteria. Unpublished
doctoral dissertation, California School of
Professional Psychology, Fresno, CA.
Mattlar, C. E. (2011). The issue of an evolutionary
development of the Rorschach Comprehensive
System (RCS) versus a revolutionary change (R-
PAS). Retrieved from
http://www.rorschachtraining.com/category/artic
les.
McDaniel, M.A., & Nguyen, N.T. (2001). Situational
judgment tests: A review of practice and
constructs assessed. International Journal of
Selection and Assessment, 9, 103– 113.
McDaniel, M. A., Hartman, N. S., Whetzel, D. L., &
Grubb, W. L., III (2007). Situational judgment
tests, response instructions, and validity: A
meta-analysis. Personnel Psychology, 60, 63-91.
Meloy, R., Acklin, M., Gacono, C., Murray, J., &
Peterson, C. (1997). Contemporary Rorschach
Interpretation. Mahwah: NJ: Lawrence Erlbaum
Publishers.
Meyer, G. J. (2000). On the science of Rorschach
research. Journal of Personality Assessment,
75(1), 46–81.
Meyer, G. J., Erdberg, P., & Shaffer, T. W. (2007).
Toward international normative reference data
for the Comprehensive System. Journal of
Personality Assessment, 89(Suppl1), S201–
S216.
Meyer, G.J., Hosseininasab, A., Viglione, D.J., Mihura,
J.L., Berant, E., Resende, A.C., & Reese, J.
(2020). The effect of CS administration or an R-
Optimized alternative on potential projective
material in Rorschach responses from six studies
and a meta-analysis of their findings. Journal of
Personality Assessment, 102 (1), 135-146. DOI:
10.1080/00223891.2018.1492926
Meyer, G. J., & Viglione, D. J. (2008). An introduction
to Rorschach assessment. In R. P. Archer & S.
R. Smith (Eds.), Personality assessment (p. 281–
336). Routledge/Taylor & Francis Group.
Meyer, G. J., Viglione, D. J., Mihura, J. L., Erard, R. E.,
& Erdberg, P. (2011). Rorschach Performance
Assessment System: Administration, coding,
interpretation, and technical manual. Toledo,
OH: Rorschach Performance Assessment
System.
Meyer, G.M., Viglione, D.J., Mihura, J.L., Erdberg, P.,
Bram, A., Giromini, L., Grønnerød, C., Kleiger,
J., Lipkind, J., de Ruiter, C., Pianowski, G., &
Vanhoyland, M. (2020). Recommendations
concerning remote administration of the
Rorschach. https://r-
pas.org/Docs/Remote%20Administration%20of
%20the%20Rorschach.pdf
Mihura, J. L. (2019). Correcting Smith et al.’s (2018)
criticisms of all Rorschach studies in Mihura,
Meyer, Dumitrascu, and Bombel (2013).
Rorschachiana, 40(2), 169-186.
Mihura, J. L., & Meyer, G. J. (Eds.). (2018). Using the
Rorschach Performance Assessment System®
(R-PAS®). The Guilford Press.
Mihura, J. L., Meyer, G. J., Dumitrascu, N., & Bombel,
G. (2013). The validity of individual Rorschach
variables: Systematic reviews and meta-analyses
00: Gacono & Smith
of the comprehensive system. Psychological
Bulletin, 139, 548–605. doi:10.1037/a0029406
Pianowski, G., Meyer, G.J., de Villemor-Amaral, A.E.,
Zuanazzi, A.C., & do Nascimento, R.S.G.F.
(2019). Does the Rorschach Performance
Assessment System (R-PAS) differ from the
Comprehensive System (CS) on variables
relevant to interpretation?. Journal of Personality
Assessment, DOI:
10.1080/00223891.2019.1677678
Piotrowski, C. (1996a). The Rorschach in contemporary
forensic psychology. Psychological Reports,
78(2), 458.
Piotrowski, C. (1996b). The status of Exner’s
Comprehensive System in contemporary
research. Perceptual and Motor Skills,
82(3_suppl), 1341–1342.
Piotrowski, C. (1996c). Use of the Rorschach in forensic
practice. Perceptual and Motor Skills, 82,254.
Piotrowski, C. (2007). Forensic psychological testing as
a function of affiliation and organizational
setting. Organization Development Journal,
25(1), 94-98
Piotrowski, C. (2017). Rorschach research through the
lens of bibliometric analysis: Mapping
investigatory domain. SIS Journal of Projective
Psychology & Mental Health, 24(1), 34-38.
Reik, T. (1956). The search within; the inner experiences
of a psychoanalyst. Farrar, Strauss, and Cudahy.
Ritzler, B. A. (2014). Society for Personality
Assessment (SPA). Rorschach Training
Programs, Inc. Newsletter, 6(2). Retrieved from
http://www.rorschachtraining.com/category/new
sletters/page/2/
Ritzler, B., & Sciara, A. (2009). Rorschach
Comprehensive System international norms:
Cautionary notes.
Ritzler, B., & Sciara, A. (2010). Rorschach Training
Programs comments on RPAS.
Sciara, A.D. & Ritzler, B.A. (2018). Comment on
“Addressing Issues in the Development and Use
of the Composite International Reference Values
as Rorschach Norms for Adults” (Meyer,
Shaffer, Erdberg, & Horn, 2015), Journal of
Personality Assessment, 100 (3), 343-344.
Singer, J., Hoppe, C. F., Lee, S. M., Olesen, N. W., &
Walters, M. G. (2008). Child custody litigants:
Rorschach data from a large sample. In C. B.
Gacono, F. B. Evans (Eds.) & N. Kaser-Boyd, L.
A. Gacono (Collaborators), The LEA series in
personality and clinical psychology. The
handbook of forensic Rorschach assessment (p.
445–464). Routledge/Taylor & Francis Group.
Smith, B. L. (2012). Clinical interpretations using the CS
and R-PAS: Discussion. This is not a horserace:
Comparison of clinical interpretation using the
CS and R-PAS. A symposium presented at the
Society for Personality Assessment Annual
Meeting, March 16, 2012, Chicago, IL.
Smith, J.M., Gacono, C.B., & Cunliffe, T.B. (2021).
Understanding Female Offenders: Psychopathy,
Criminal Behavior, Assessment, & Treatment.
San Diego, CA: Elsevier.
Smith, J.M., Gacono, C.B., & Fontan, P., Cunliffe, T.B.,
& Andronikof, A. (2020). Understanding
Rorschach research: Using the Mihura (2019)
commentary as a reference. SIS Journal of
Projective Psychology & Mental Health, 27(2),
73-84.
Smith, J.M., Gacono, C.B., Fontan, P., Taylor, E.E.,
Cunliffe, T.B., & Andronikof, A. (2018). A
scientific critique of Rorschach research:
Revisiting Exner’s Issues and Methods in
Rorschach Research (1995). Rorschachiana,
39(2), 180-203.
Whitehouse, J. (2019). The Rorschach in the 21st
century: A tablet-based administration
equivalence study, Unpublished doctoral
dissertation, William James College, Newton,
MA.
Wiener, I. (2008). Presenting and defending Rorschach
testimony. In C. B. Gacono, F. B. Evans (Eds.)
& N. Kaser-Boyd, L. A. Gacono (Collaborators),
The LEA series in personality and clinical
psychology. The handbook of forensic
Rorschach assessment (p. 121–137).
Routledge/Taylor & Francis Group.
Young, M. H., Erdberg, P. S., & Justice J. (2008).
Inmates in prison psychiatric treatment: A
multimethod description. In C. B. Gacono, F.
Evans (Eds.), The handbook of forensic
Rorschach assessment (pp. 403-421). New York,
NY US: Routledge/Taylor & Francis Group.
