ArticlePDF Available

A Scientific Critique of Rorschach Research: Revisiting Exner’s Issues and Methods in Rorschach Research (1995)

Authors:
  • Consultant/author/artist
  • International Rorschach Institute
  • taylor psychological innovators

Abstract and Figures

Exner’s (1995a) Issues and Methods in Rorschach Research provided a standard of care for conducting Rorschach research; however, the extent to which studies have followed these guidelines has not been examined. Similarly, meta-analytic approaches have been used to comment on the validity of Exner’s Comprehensive System (CS) variables without an evaluation as to the extent that individual studies have conformed to the proposed methodological criteria (Exner, 1995a; Gacono, Loving, & Bodholdt, 2001). In this article, 210 studies cited in recent meta-analyses by Mihura, Meyer, Dumitrascu, and Bombel (2013) were examined. The studies were analyzed in terms of being research on the Rorschach versus research with the Rorschach and whether they met the threshold of validity/generalizability related to specific Rorschach criteria. Only 104 of the 210 (49.5%) studies were research on the Rorschach and none met all five Rorschach criteria assessed. Trends and the need for more stringent methods when conducting Rorschach research were presented.
Content may be subject to copyright.
Original Article
A Scientific Critique of Rorschach Research
Revisiting ExnersIssues and Methods in Rorschach
Research (1995)
Jason M. Smith
1
, Carl B. Gacono
2
, Patrick Fontan
3
,
Enna E. Taylor
4
, Ted B. Cunliffe
5
, and Anne Andronikof
6
1
FCC Hazelton, Bruceton Mills, WV, USA
2
Private Practice, Asheville, NC, USA
3
Circonscription Saint Denis 1, Paris, France
4
Private Practice, San Francisco, CA, USA
5
Private Practice, Miami, FL, USA
6
Laboratoire IPSé, Université Paris Ouest, Nanterre, France
Abstract: Exners (1995a) Issues and Methods in Rorschach Research provided a standard of
care for conducting Rorschach research; however, the extent to which studies have followed
these guidelines has not been examined. Similarly, meta-analytic approaches have been
used to comment on the validity of Exners Comprehensive System (CS) variables without an
evaluation as to the extent that individual studies have conformed to the proposed
methodological criteria (Exner, 1995a; Gacono, Loving, & Bodholdt, 2001). In this article, 210
studies cited in recent meta-analyses by Mihura, Meyer, Dumitrascu, and Bombel (2013)
were examined. The studies were analyzed in terms of being research on the Rorschach
versus research with the Rorschach and whether they met the threshold of validity/gener-
alizability related to specific Rorschach criteria. Only 104 of the 210 (49.5%) studies were
research on the Rorschach and none met all five Rorschach criteria assessed. Trends and the
need for more stringent methods when conducting Rorschach research were presented.
Keywords: Rorschach, Exner Comprehensive System, psychological assessment
Exners(1995a) Issues and Methods in Rorschach Research set a standard of care for
conducting Rorschach research. At that time, he stated: A huge number of pub-
lished investigations are clearly marked by errors in design, implementation,
and/or analysis(Exner, 1995b, p. 3). Since Exners(1995a) cautions, no one
has examined the degree to which Rorschach research has conformed to the
guidelines offered by the chapter authors. Meta-analytic findings, in particular,
have been accepted at face value with little consideration for the degree to which
individual studies fell within the parameters outlined by Exner (1995a) and others
(e.g., Gacono, Loving, & Bodholdt, 2001).
Rorschachiana (2018), 39(2), 180203 ©2018 Hogrefe Publishing
https://doi.org/10.1027/1192-5604/a000102
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
The following is a brief overview of each chapter in Issues and Methods in
Rorschach Research. In addition to highlighting issues to be considered when con-
ducting Rorschach research, Exner (1995b) cautioned the researcher to be aware
of the complexity involved with this type of research. He presented many essential
ideas such as: (1) offering cautions about small sample size (at least 15 subjects are
needed in an experimental group for every dependent variable to be included in
the analysis); (2) presenting considerations related to personality/response style
(Lambda & Introversive/Extratensive); (3) suggesting how normative data can
be misused (any comparisons with normative data would find meaningless signif-
icant results; ingenuous conclusion that they have made a great discovery,
p. 17); (4) cautioning against overemphasizing or overgeneralizing results; and
(5)notusingtheshotgun study(p. 14) where there is no a priori model specified
before analyzing Rorschach data. This latter approach can increase the probability
of finding significant results by chance (Type I error; also see Viglione, 1995;Wei-
ner, 1995b). Further, and most importantly, Exner stressed that not everything
appearing in the literature was truth(p. 4).
Dies (1995a, 1995b) discussed issues with sample size (a sample size of less than
20 could lead to deceptive results due to deviant subjects), missing/inappropriate
control groups, and problems related to administration/scoring (also see Exner,
Kinder, & Curtis, 1995; Gacono, Evans, & Viglione, 2008;Ritzler&Exner,
1995). Dies also opined that a theoretical model needs to proceed a study and
it guides analyses (also Weiner, 1995b). Likewise, he identified certain Rorschach
studies that had methodological bias, some of which were included in recent
meta-analyses (e.g., Ball, Archer, Gordon, & French, 1991). Additionally, Dies (also
see Acklin & McDowell, 1995; Mcguire, Kinder, Curtiss, & Viglione, 1995)found
that researchers did not provide basic demographic data. This is necessary for
interpreting findings, especially when a study is focused on validating the corollar-
ies of a Rorschach variable, including sample parameters such as Lambda,
Responses, mean, standard deviations, and frequencies for the variables studied
(also see Gacono et al., 2001; Gacono & Gacono, 2008).
Viglione and Exner (1995) cautioned against inappropriate control groups (i.e.,
using an Exner normative sample as the comparison group; also see Shaffer, Erd-
berg, & Haroian, 1999 for other normative data) and stressed the importance of
critically evaluating previous Rorschach literature. Failure to critique the literature
can and has allowed studies with inaccurate and ambiguous results to infiltrate
published studies creating deceptive impressions (also see Cunliffe et al., 2012).
Proper statistical methods were also discussed, specifically the importance of
accurately applying parametric versus nonparametric statistical procedures (many
Rorschach variables form J-shaped curves that are not conducive to analysis with
©2018 Hogrefe Publishing Rorschachiana (2018), 39(2), 180203
A Scientific Critique of Rorschach Research 181
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
parametric procedures; Viglione, 1995). Further, Viglione (1995) identified the
variables appropriate for parametric and nonparametric statistics. Weiner
(1995b) discussed the differences of using the Rorschach as a dependent and inde-
pendent measure (doing research with the Rorschach or on the Rorschach), to pro-
vide adequate interrater reliabilities (80% agreement or better for Rorschach
variables) and that poorly designed research will contain Type I and II error (stat-
ing a relationship exists when it does not or failing to identify relationships that
exist). Ritzler and Exner (1995) discussed the limitations of clinical research and
its chance for confounding variables (i.e., R, EB, and Lambda). Mcguire and col-
leagues (1995) stated that Rorschach variables may need to be categorized (i.e.,
dichotomous variables). Finally, Zillmer and Vuz (1995) provided information to
perform factor analyses with Rorschach data.
Overall, many of the authors of Issues and Methods in Rorschach Research
stressed the importance of not using a small sample size, providing key variables
means/standard deviations (R, Lambda) as these can confound other variables,
and the need to critically evaluate Rorschach research that may contain problems
with methodology.
Gacono and colleagues (2001), Gacono and Gacono (2008), and Cunliffe et al.
(2012) have added to the necessary parameters for conducting Rorschach research
in order for a study to be generalizable and to allow reviewers to accurately inter-
pret findings (e.g., reporting mean, standard deviation, and frequencies for IQ,
Lambda, number of Responses). They offered five conceptual and four method-
ological criteria for evaluating the Rorschach/psychopathy research (Gacono
et al., 2001,p.32; also see Cunliffe et al., 2012;Gacono&Gacono,2008;Gacono
et al., 2008). Only Methodological Criteria 24(see below) are provided as they
apply to Rorschach research.
Methodological Issues in the Assessment of Rorschach/Psychopathy
Findings (Gacono et al., 2001, p. 32)
2. Studies need to account for (control or delineate) the limitations imposed by
factors such as gender, sexual deviance, concurrent Axis I psychosis, age, IQ,
testing setting, and legal status. These factors can influence the production of
certain Rorschach variables.
3. R (number of responses) must be considered. Increased R is found in certain
sex offender groups, (Bridges, Wilson, & Gacono, 1998; Gacono, Meloy, &
Bridges, 2000), whereas low R is typical among many criminal groups (Vig-
lione, 1999). Thus, R can act as a moderator influencing the relationship
Rorschachiana (2018), 39(2), 180203 ©2018 Hogrefe Publishing
182 J. M. Smith et al.
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
between Rorschach variables and criterion variables. Research should inves-
tigate this hypothesis by controlling for R and examining the relationship
between Rorschach variables and criterion constructs at different levels of
R (e.g., R = 1417, etc.).
4. Response style must be considered (Bannatyne, Gacono, & Greene, 1999).
Variables and styles such as R, Lambda, Extratensive, and Introversive can
impact the production of certain Rorschach variables (Exner, 1995b), con-
tributing to seemingly discrepant findings among studies.
While the wording of Methodological Criteria 2,3,and4is geared toward
Rorschach studies, similar issues (with modifications) are essential when conduct-
ing research with other psychological assessment instruments such as the Min-
nesota Multiphasic Personality Inventory-2(MMPI-2;Butcher,Dahlstrom,
Graham, Tellegen, & Kaemmer, 1989), Personality Assessment Inventory (PAI;
Morey, 1991), or Millon Clinical Multiaxial Inventory IV (MCMI-IV; Millon,
Grossman, & Millon, 2015). These criteria (2,3,and4) will be expanded because
they are important caveats for Rorschach research.
Potential confounds within the parameters of the instrument used should be
assessed. Rorschach responses and protocol constriction can be significantly
affected by mental illness, IQ, legal status, age, and testing environment (Method-
ological Criterion 2;Exner,1974,2003;Weiner,1966). IQ and/or Educational
level affect the production of certain research variables and must be accounted
for by researchers (Exner, 2003;Gaconoetal.,2001;Weiner,1966). Meyer, Giro-
mini, Viglione, Reese, and Mihura (2015) found years of adult education was cor-
related with different Rorschach variables related to complexity and cognitive
synthesis. Therefore, a lack of consideration of these factors may result in method-
ological bias. Demographic information must be provided in order for reviewers to
understand the meaning of any Rorschach data offered within the study.
As stated in Methodological Criterion 3, the number of Rorschach responses (R)
relates to the stability and reliability of Rorschach variables (Viglione & Meyer,
2008;Weiner,2003); protocols with less than 14 responses (R < 14)shouldbe
interpreted with caution as they typically do not have enough data for adequate
retest reliability (Weiner, 2003). Although it is acknowledged that R < 14 protocols
were considered acceptable if accompanied by a Lambda below 1.2(Exner, 1986)
and these protocols may have clinical significance (Gacono, 1997), the current
standard per the Exner Comprehensive System (CS; Exner, 1993,2003) requires
14 or more responses for interpretation. Further, the problem of R (Exner, 1992;
Meyer, 1992; Wood, Nezworski, & Stejskal, 1996) may or may not impact
Rorschach results; however, significant differences between experimental groups
on R could impact research findings (Weiner, 1995a). One can state with greater
©2018 Hogrefe Publishing Rorschachiana (2018), 39(2), 180203
A Scientific Critique of Rorschach Research 183
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
certainty that within a Rorschach protocol of over 30 responses having a T or Fr +
rF equal to zero, the absence of these variables was most likely due to either the
absence of the trait or the personality functioning of the subject. In a protocol with
less than 14 responses, additional hypotheses need to be considered. Similarly, in a
protocol with 12 responses and four reflections it is highly likely that the presence
of reflections are stating something about the personality of the person (Gacono &
Meloy, 1994).
Methodological Criterion 4includes participants with Lambda > .99 and/or
F% = .50 (Meyer, Viglione, & Exner, 2001). These response style variables
(Gacono & Gacono, 2008 Lambda and R) are essential to interpreting Rorschach
findings and are the two main validity criteria of a CS protocol (Exner, 2003). Fur-
ther, if attempting to validate the Rorschach with meta-analyses, valid protocols (R
and Lambda reported; R 14) are needed. Frequently seemingly discrepant find-
ings can be explained by a predominance of high Lambda protocols in their sam-
ples (Exner provided different normative samples for Introversive/Extratensive
patients for L > 0.99 &L<1.00). Further, Konishi (2003) found differences in
many Rorschach variables between high and low Lambda groups. While the con-
strictionof the protocols is frequently interpretable related to the sample charac-
teristics, comparing Rorschach variables between other studies (more normatively
distributed Lambdas) is impossible (Gacono & Gacono, 2008). It would be diffi-
cult to know if the presence or absence of a CS variable is due to a high Lambda
or the absence of the trait in a high Lambda sample. For example, if a study of
male offenders had an overall mean L > .99 (high Lambda) and did not produce
any reflections, it would be difficult to determine whether the lack of these vari-
ables was due to the protocols constriction or the absence of the trait represented
by these variables. Certainly, one cannot compare the high Lambda sample with a
sample of male offenders with a mean L = .75 (avg. Lambda range) where these
variables were present and then conclude that the negative findings in the high
sample negate the positive findings in the more normally distributed sample.
Assuming equality in the samples where one is a high Lambda sample is not jus-
tified. Generalizability is limited and only within-sample interpretive conclusions
are justified. This reinforces the rationale for including Lambda means, standard
deviations, etc. for all samples included in Rorschach research so the reviewer can
determine whether the lack of results was due to this potential confound, which
tends to create either a Type I (stating a relationship exists when it does not) or
Type II error (failing to identify a relationship that does in fact exist).
As noted by Gacono and Gacono (2008) when discussing the validity versus
generalizability of findings in their forensic outpatient groups:
Rorschachiana (2018), 39(2), 180203 ©2018 Hogrefe Publishing
184 J. M. Smith et al.
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
Atypical patterns of constriction are common and frequently represent accu-
rate portrayals of the patients psychology (referring to high Lambda sam-
ples) For example, the higher rate of positive SCZI and PTI and Level 2
Special Scores among Schizophrenic groupscompared to our outpatient
forensic groups does not indicate a lack of psychotic process among these
patients. It also does not, necessarily, suggest resistance to the testing pro-
cess. Rather, this finding must be interpreted in light of the patients chronic-
ity and their cognitive and emotional impoverishment. Rorschach
constriction or expansion must always be explained within the context of
the entire assessment battery and the patients psychosocial history. (p. 443)
Researchers have also indicated the importance of determining and reporting
interrater reliability for all Rorschach variables studied (Meyer, 1999; Viglione &
Meyer, 2008;Weiner,2003; Wood, Nezworski, & Stejskal, 1996). Therefore,
researchers should report interrater reliability (.60 [kappa and/or ICC] and
80% agreement is characterized as good to excellent; Meyer, 1999;Weiner,
1995a, 2003). Failure to report good/excellent interrater reliability is problematic
as without reliability one is not assured of finding valid studies (Borsboom, Mel-
lenbergh, & van Heerden, 2004).
Further, though there is nostandard that is considered an adequate sample size; it
appears that less than 20 is not acceptable using effect size, power, and alpha tables
for nonparametric tests (Cohen, 1992;Dies,1995a; Exner, 1995b). In neuroscience
research, having small samples negatively impacts reliability and has less power
(Button et al., 2013), and these points are also relevant for Rorschach research.
Therefore, when a study has low statistical power there is less chance of detecting
a true effect and low power also reduces the likelihood that a statistically signifi-
cant result reflects a true effect(p. 365; Type II error). Low-sample research can
also be viewed as unethical, inefficient, and wasteful (Button et al., 2013).
In summary, many different criteria need to be addressed when completing
Rorschach research based on previous research from prominent Rorschach
experts. However, five were selected that were mentioned most by many
researchers and that appear to be essential for the validity and generalizability
of research findings when conducting Rorschach research. These five are: (1)
IQ/Educational level; (2) Rorschach Responses; (3) Lambda/F%; (4) interrater
reliability; and (5)samplesize.
These criteria are paramount for Rorschach research as there has been an
increase in using meta-analyses with the Rorschach. Meta-analyses may overcome
some of these problems such as small sample size, low power, and small effect
sizes, unless the individual studies used in the meta-analyses contain problems
related to these five criteria.
©2018 Hogrefe Publishing Rorschachiana (2018), 39(2), 180203
A Scientific Critique of Rorschach Research 185
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
Meta-Analysis
Meta-analytic procedures are frequently utilized in assessing the validity of psy-
chological measures (Sánchez-Meca & Marín-Martínez, 2010) including the
Rorschach (Mihura et al., 2013;Woodetal.,2010). However, meta-analyses are
only as good as the individual studies that they include (Button et al., 2013;Cun-
liffe et al., 2012; Hunter & Schmidt, 2004). For example, recent meta-analyses by
Wood et al. (2010) examining the Rorschach and psychopathy were found to con-
tain methodological bias in the research studies suggesting that there were not
enough appropriate studies to perform the meta-analyses (see Cunliffe et al.,
2012). Out of the 22 studies utilized in the meta-analyses, only one study was valid
for inclusion by Rorschach/psychopathy standards (Cunliffe et al., 2012;Gacono
et al., 2001;Gaconoetal.,2008).
Recent meta-analyses by Mihura et al. (2013) questioned the validity of several
Rorschach Comprehensive System (RCS; Exner, 2003) variables (e.g., Egocentric-
ity Index and Isolation Index). Mihura et al. found mean validity coefficients of r=
.27 when using the Rorschach variables against externally assessed criteria (e.g.,
psychiatric diagnosis) and r=.08 for introspectively assessed criteria (e.g., self-
report measures). Further, effect sizes were calculated for the different Rorschach
individual variables and they used Hemphill (2003) criteria to interpret the effect
sizes. In total, 13 variables had excellent support (e.g., An + Xy; Perceptual Think-
ing Index; r.33), 17 had good support (e.g., Lambda, Affective Ratio; r.21), 10
had modest support (e.g., Vista, PHR; r=.15.21), 13 had little or no support (e.g.,
Pure Color, Food; r<.15), and 12 variables (e.g., Aspiration Ratio, Color Projec-
tion) did not have any validity studies.
Wood, Garb, Nezworski, Lilienfeld, and Duke (2015)critiquedtheMihuraetal.
(2013) meta-analyses and indicated that the research by Mihura et al. was biased,
since they failed to include certain articles/unpublished dissertations. Mihura,
Meyer, Bombel, and Dumitrascu (2015) responded to these criticisms and refuted
theclaimsofpublicationbias.TibonCzoppandZeligman(2016) also critiqued the
Mihura et al. (2013) meta-analyses. Tibon Czopp and Zeligman focused their cri-
tique on the 13 CS variables that Mihura et al. stated had little to no support (e.g.,
Pure Color, Food; r<.15). They argued that these variables should not be removed
from the CS system. They suggested there were discrepancies with the way
Mihura and colleagues defined the 13 CS variables in the meta-analyses, which
was not comparable to the customary CS interpretation for these variables. They
argued the 13 individual variables needed to be interpreted within the CS clusters
(e.g., self-perception, affective, etc.) due to the Rorschach being a multidimen-
sional method. Additionally, in order to validate the 13 CS variables, externally
assessed criteria (i.e., observer ratings, diagnosis) are better than introspectively
Rorschachiana (2018), 39(2), 180203 ©2018 Hogrefe Publishing
186 J. M. Smith et al.
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
assessed criteria (i.e., self-report measures). Tibon Czopp and Zeligman also pos-
tulated that there may have been studies on these 13 CS variables. These studies
that were not included in the meta-analyses may have supported the CS variables.
Therefore, findings that actually existed were not revealed in the Mihura et al.
meta-analyses (Type II error). Mihura, Meyer, Bombel, and Dumitrascu (2016)
responded, saying the statements used by Tibon Czopp and Zeligman were
biased, their arguments would have lowered validity coefficients, and they needed
to perform their own meta-analyses for these 13 variables.
There are different ways to critically evaluate studies within a meta-analysis.
One way to evaluate validity (Borsboom et al., 2004), especially relevant to
Rorschach studies, is to determine whether the articles used were studies with
the Rorschach (application studies) or studies on the Rorschach (validation studies;
Weiner, 1995b). Another way to conceptualize this would be using the Rorschach
as the dependent variable (application) or the independent variable (validation).
Typically, in application studies (studies with the Rorschach), Rorschach variables
are assumed to be valid measures of a specific psychological construct and they
can be replaced by another instrument measuring the same construct. A validation
study would be attempting to determine the Rorschach variable meaning and it
does not assume it is already valid. An example of an application study would
be a researcher studying cognitive remediation effects on thought disorder in
patients with schizophrenia. This researcher might choose to assess thought disor-
derwiththeRorschachCSWSum6variable. However, the WSum6score could be
replaced by the Thought and Language Disorder Scale (Kircher et al., 2014)with-
out modification to the experimental design (thought disorder could be opera-
tionalized differently). In this example, the researcher assumes that the WSum6
score is already a valid measure of thought disorder. Although this example would
seem to support concurrent validity of both psychological measures, the correla-
tion may not be a true estimate of validity (Borsboom et al., 2004). Therefore,
construct validity may be more important as without constructs and theory, psy-
chology would not be classified as a science (Dies, 1995a). By nature, application
studies (with the Rorschach) are not designed to validate specific Rorschach vari-
ables, rather they are used to study a clinical phenomenon. It would be problem-
atic to use application studies (with the Rorschach) which were not designed to
validate Rorschach variables in order to evaluate the validity of Rorschach scores.
Ideally, meta-analyses on the validity of individual Rorschach variables should be
based on validation studies (i.e., studies on the Rorschach; also see Borsboom
et al., 2004).
Another method for assessing individual studies in a meta-analysis is to evaluate
specific criteria relevant to the instrument or procedure studied (Cunliffe et al.,
2012; Gacono et al., 2001). Hunter and Schmidt (2004) outlined the necessary
©2018 Hogrefe Publishing Rorschachiana (2018), 39(2), 180203
A Scientific Critique of Rorschach Research 187
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
components of a meta-analytic study: author, date, sample size, standardized
effect score, subject characteristics, diagnostic conditions (scope, duration, and
severity), strength of study design, and individual study methodological concerns.
Validity forms the basis of Hunter and Schmidts(2004) comments concerning
the importance of ensuring that reliable, valid, and methodologically sound stud-
ies are selected for inclusion in a meta-analysis. Further, although researchers
appear to use mathematically sound methods (equations used as intended), the
issue lies in the validity (i.e., meaning; Borsboom et al., 2004)oftheapplication
of these techniques to methodologically biased studies included in a meta-
analysis. Therefore, it is important to analyze the Rorschach studies included in
a specific meta-analysis and to evaluate them with specific criteria (e.g., sample
size, interrater reliability).
Present Study
In this article, the studies from the Mihura et al. (2013) meta-analyses were ana-
lyzed examining the quality of the study in light of Exner (1995a) and others (Cun-
liffe et al., 2012;Gaconoetal.,2001). The 210 studies (similar to Dies, 1995b)
from the Mihura et al. (2013) meta-analyses were examined focusing on whether
the articles were application or validation studies (studies with the Rorschach vs.
studies on the Rorschach) as well as the five methodological Rorschach criteria
related to the validity and generalizability discussed by many Rorschach research-
ers. The five criteria were: (1) IQ/Education level; (2) Responses; (3) Lambda/F%;
(4) interrater reliability; and (5)samplesize.
Method
All articles used in the Mihura et al. (2013) meta-analyses were obtained and
reviewed. Then, all 210 articles were examined for the following five main areas:
1. IQ/Education level,
2. Responses,
3. Lambda/F%,
4. Interrater reliability, and
5. Sample size.
Within each five criteria, different questions were examined. The issues were tal-
lied up and a sum total for all 210 articles was obtained when the answer to the
questions (see below) was no. Therefore, the examinations were:
Rorschachiana (2018), 39(2), 180203 ©2018 Hogrefe Publishing
188 J. M. Smith et al.
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
1. IQ/Education level
a. Did the article have statistics related to IQ/Educational level (Mand
range)?
b. If either mean or range was provided, did the article include both
statistics?
c. If IQ range was reported, did all participants have an IQ > 80?
2. Responses (R)
a. Did the article have statistics related to R (Mand range)?
b. If mean was provided, did the article include range?
c. If range was provided, did the article include mean?
d. If range was reported, did all protocols have R 14?
3. Lambda/F%
a. Did the article have statistics related to Lambda/F% (Mand range)?
b. If mean was provided, were the means for Lambda/F% < 0.99/50%?
d. If mean was provided, did the article report range?
d. If the mean of Lambda/F% > .99/.50, did the article report IQ/Education
Level? (L/F% > .99/.50 and IQ/Education level were examined in combi-
nation as this affects generalizability.)
4. Interrater reliability
a. Did the article have interrater reliability statistics?
b. If interrater reliability was reported, were there values for ICC/κ.60 or
80% agreement?
5. Sample size
a. Did the article have comparison groups with 20 participants?
An overall analysis of the 210 articles was conducted after the aforementioned
analyses. This was to determine how many problems an article had with the five
main criteria. For example, if an article did not report Lambda and interrater reli-
ability statistics, it was calculated that the study had two methodological issues.
Therefore, the five criteria were tallied up to determine how many problems an
article had (i.e., the article contained zero, one, two, etc. problems).
After assessing issues in the 210 articles, the articles were then classified as
either validation or application studies. See the previous section to understand
how validation and application studies were operationalized. After identifying
©2018 Hogrefe Publishing Rorschachiana (2018), 39(2), 180203
A Scientific Critique of Rorschach Research 189
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
the validation studies, and since they are preferred for meta-analyses, these arti-
cles were reviewed with the same five criteria as all 210 articles.
Due to the importance of validation studies in a meta-analysis, only these arti-
cles were examined to determine whether the methodological bias stated earlier
had an impact on findings. The validation studies were reviewed to determine
whether there were any counterintuitive findings. A counterintuitive study that
we propose can be operationalized as a study that had findings that were incon-
sistent with theory or did not replicate previous findings (potential Type II error;
where biased designed studies fail to identify relationships that in fact exist). Fur-
ther, a study was classified as counterintuitive if there was mixed support for the
Rorschach variables studied. For example, many research studies in the Mihura
et al. meta-analyses examined multiple Rorschach variables (i.e., Hart, 1991). Hart
did not find results consistent for the Egocentricity Index (EGOI) but support was
found for the other variables examined; however, it was classified as counterintu-
itive owing to the inconsistent finding for EGOI. The importance of this distinction
is, simply stated, if counterintuitive findings occur within a methodologically
biased design there is no way to tell if the findings are true findings independent
of confounds introduced by the methodology (Type II error; Weiner, 1995b).
Specifically, these type of studies (including those with confounds that are not
addressed) rarely identify relationships that do not exist (Type I error); rather,
counterintuitive findings in Rorschach research often result in a failure to identify
relationships that do in fact exist (Type II error). These counterintuitive studies
within the validation studies were tallied up and an overall number was calculated.
Results
All 210 articles had some issues in the criteria assessed. When looking at the five
criteria globally (IQ/Education level, R, Lambda/F%, interrater reliability, sample
size), every article had methodological issues (see Table 1). Four articles only had
one methodological issue (1.9%), 17 had two (8.1%), 85 had three (40.5%), 87 had
four (41.4%), and 17 had five (8.1%). An example of an article with two problems
was the one by Dao and Prevatt (2006). They provided statistics for R, provided
an adequate interrater reliability, and had comparison groups greater than 20;
however, there was no mention of IQ/Education level or Lambda/F%. An exam-
ple of an article with five issues was that by Abraham, Mann, Lewis, Coontz, and
Lehman (1990). They had three comparison groups of 15 participants, did not pro-
vide statistics for R, Lambda/F%, or interrater reliability, and did not provide a
rangeforIQ.Thespecificquestionsforthefivecriteriawerealsocalculated
Rorschachiana (2018), 39(2), 180203 ©2018 Hogrefe Publishing
190 J. M. Smith et al.
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
(see Table 2). The total calculations show if the answer was no to the question
asked.
Two independent raters then used the aforementioned operationalizations
about application and validation studies and analyzed each of the 210 studies.
In total, 106 studies were classified as validation studies; however, there was a
Table 1.Methodological issues analysis (N= 210)
Methodological issues (n) Articles (n) Percentage
000%
1 4 1.9%
2 17 8.1%
3 85 40.5%
4 87 41.4%
5 17 8.1%
Table 2.Five methodological criteria analysis (N= 210)
Criterion Articles (n) Percentage
IQ/Education level
No Mand range 56 26.7%
Missing Mor range 118 56.2%
IQ M< 80 20 9.5%
Responses (R)
No Mand range 108 51.4%
Mreported but no range 65 31.0%
Range reported but no M2 1.0%
Responses < 14 22 10.5%
Lambda/F%
No Mand range 141 67.1%
M= L > .99/F% > .50 40 19.0%
No range for L/F% 45 21.4%
M= L > .99/F% > .50 & No IQ/Ed. level reported 28 13.3%
Interrater reliability
None reported 72 34.2%
< .60 (κ/ICC) and/or 80% agreement 16 7.6%
Sample size
< 20 participants 56 26.7%
Note.M= mean; L = Lambda; F% = percentage of F responses.
©2018 Hogrefe Publishing Rorschachiana (2018), 39(2), 180203
A Scientific Critique of Rorschach Research 191
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
disagreement on three articles. The interrater reliability was measured by kappa
(κ=.97). A third rater resolved the disagreement, and 104 of the 210 studies
(49.5%) were classified as validation studies.
Since validation studies are more appropriate for meta-analyses, the 104 studies
were analyzed with the same criteria used for all 210 articles (see Table 3and
Table 4). After these analyses, the counterintuitive finding operationalization
was used with the 104 validation studies. It was found that 45 studies (43.3%)
had counterintuitive findings.
Discussion
Exner and colleagues (1995a; Cunliffe et al., 2012;Gaconoetal.,2001; Gacono
et al., 2008) provided pertinent caveats for conducting Rorschach research. At this
time, there are no set rules for conducting Rorschach research and few have cri-
tiqued the points provided by the authors of Issues and Methods in Rorschach
Research and Gacono et al. (2001). Other researchers have proposed important
concepts to consider when conducting Rorschach research (i.e., not using a nor-
mative sample as a comparison group); however, the criteria used in this article
appear to be the most pertinent to validity and generalizability. Those who
research the Rorschach must be aware of these complexities and pitfalls (Exner,
1995a). Failure to heed these cautions produced biased Rorschach studies that
led to inaccurate claims about the test (many resulting from Type II error). Most
of the researchers did not provide descriptive data for Rorschach samples (Vig-
lione & Exner, 1995). For example, 51.4% did not provide any statistics for R
and 67.1% did not provide these data for Lambda/F%. Further, of the 210 articles,
90% had three or more issues, which indicated these articles manifested gaps in
their methodology.
A little more than half the articles included in these meta-analyses were
application studies (with the Rorschach; 50.5%) and this is problematic when
Table 3.Methodological issues in validation studies only (N= 104)
Methodological issues (n) Articles (n) Percentage
000%
100%
2 12 11.5%
3 34 32.7%
4 50 48.1%
5 8 7.7%
Rorschachiana (2018), 39(2), 180203 ©2018 Hogrefe Publishing
192 J. M. Smith et al.
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
evaluating the validity of Rorschach variables since this type of study is not meant
to validate Rorschach variables. Even more concerning is the fact that these two
very different types of research (studies with the Rorschach vs. studies on the
Rorschach) were combined indiscriminately in the Mihura et al. (2013) meta-ana-
lyses. Indeed, in application studies, Rorschach variables are typically dependent
variables of the research design (e.g., the effects of cognitive remediation on
thought disorder in patients with schizophrenia as assessed by the CS WSum6).
Treating application studies (with the Rorschach) as validation studies (on the
Rorschach) is the equivalent of confusing dependent and independent variables
of research designs.
Since validation studies are more pertinent for a meta-analysis looking to vali-
date Rorschach scores, many of the Rorschach validation studies included in
Mihura et al. (2013) meta-analyses did not meet the methodological criteria
assessed (90% had three or more problems with the criteria). Failure to meet
the five criteria assessed limits the validity and generalizability of the study. If
there are problems in a study it may not be advisable to use it in a meta-analysis.
Finding a relationship despite poor methodology is more impressive and comes
Table 4.Five methodological criteria analysis for validation studies only (N= 104)
Criterion Articles (n) Percentage
IQ/Education level
No Mand range 27 26.0%
Missing Mor range 61 58.7%
IQ M< 80 9 8.7%
Responses (R)
No Mand range 58 55.8%
Mreported but no range 26 25.0%
Range reported but no M2 1.9%
Responses < 14 9 8.7%
Lambda/F%
No Mand Range 80 76.9%
M= L > .99/F% > .50 16 15.4%
No range for L/F% 14 13.5%
M= L > .99/F% > .50 & No IQ/Ed. level reported 12 11.5%
Interrater reliability
None reported 33 31.7%
< .60 (κ/ICC) and/or 80% agreement 7 6.7%
Sample size
< 20 participants 25 24.0%
Note.M= mean; L = Lambda; F% = percentage of F responses.
©2018 Hogrefe Publishing Rorschachiana (2018), 39(2), 180203
A Scientific Critique of Rorschach Research 193
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
with more weight than failing to find a relationship because of poor methodology.
This leaves open the possibility that the relationship exists and can be found with
better methods (overcoming Type II error) as poor methodological studies rarely
yield Type I error (identifying relationships that do not exist). A more refined anal-
ysis of the validation studies was undertaken to determine whether the studies
had counterintuitive findings (i.e., were inconsistent with theory and/or previous
research). In all, 45 studies had counterintuitive findings (43.3%). Owing to the
methodological issues it is unclear if these findings are due to Type II error, the
findings are real, an artifact of the methodology, a quirk, or an atypical sample.
Moreover, negative results could be attributable to Lambda or EB. That is, a vari-
able could be valid for some purpose among L < 1.00 records; however, it could be
invalid among L > 0.99 records, and similarly a variable could be valid for some
purpose among introversive but not extratensive records (EB style).
For example, Simon (1989) did not find support of the Isolation Index (one of
the variables Mihura et al. suggested had no support) compared with an MMPI
scale. However, Simon did not report L/F% and IQ/Education level and included
protocols with R < 14 (R = 10). Without even considering the problems when using
MMPI scales to validate Rorschach indices, the failure to take into account these
criteria, it makes it impossible for the reader to determine whether the lack of cor-
relation between the MMPI scale and the Rorschach index was nothing more than
the result of an atypical sample. Hart (1991) looked at the Egocentricity Index
(EGOI; another variable identified by Mihura et al. as having poor support) and
the results did not support previous findings. However, Hart did not have IQ
scores for all the participants, some had IQ < 80, and L/F% and interrater relia-
bility were not reported. Another study examining the EGOI (Brems & Johnson,
1990) also did not find support for the index. However, the participants (inpatient
psychiatric patients) had a high Lambda (M=1.20) and IQ/Education level and R
statistics were not provided. Ball and colleagues (1991; a study found to contain
bias [Dies, 1995a]) found inconsistent findings with the DEPI; however, percent
agreement in terms of some Rorschach variables was 60%, much less than the
acceptable 80%. Again, these issues from all these studies makes it difficult to
generalize the findings.
Although the counterintuitive findings pertain to Type II error, there are prob-
lems with Rorschach studies that can lead to Type I error. This would include
using a shotgun study (where many statistical analyses are performed), normative
samples being used as a comparison group (i.e., Exner, 1995b; Shaffer et al., 1999),
or inappropriate statistical comparisons being used (i.e., using parametric statistics
like ANOVA rather than a chi-square analysis). Exner (1995b) cautioned that using
a normative sample would result in some statistically significant findings by
chance and should not be used. This would also be found when using too many
Rorschachiana (2018), 39(2), 180203 ©2018 Hogrefe Publishing
194 J. M. Smith et al.
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
comparisons (shotgun studies). Viglione (1995) provided a table of which CS vari-
ables are appropriate for parametric analyses (i.e., ttest) and which are appropri-
ate for non-parametric (i.e., chi-square) analyses. For example, any content
variable is more appropriate for chi-square analyses, while X-% is generally appro-
priate for parametric analyses. It is often more appropriate with variables such as
texture to evaluate this in terms of T=0,T=1,andT>1.Thisisthecasewith
many Rorschach variables and conducting the correct analysis is essential when
examining studies to be included within meta-analyses. Examining the 210 articles
in the Mihura et al. meta-analyses, 17 (8.1%) studies met criteria for a shotgun
study, 12 (5.7%) had compared their sample with an Exner normative sample,
and 70 (33.3%) used inappropriate statistical comparisons (i.e., using parametric
statistics like ANOVA rather than a chi-square analysis). Therefore, many studies
included in the Mihura et al. (2013) meta-analyses may be problematic because of
both Type I and Type II error.
In 1995, Exner stated: A huge number of published investigations are clearly
marked by errors in design, implementation, and/or analysis(Exner, 1995b, p. 3).
Further, Viglione and Exner (1995) stated: A substantial proportion [of Rorschach
research] have been marked by flawsand all literature cannot be afforded equal
weight(p. 55). Though there have been improvements, this continues to be the
case as identified in the analysis. Many of the caveats provided by researchers
have not been followed (Dies, 1995a, 1995b; Exner, 1995; Gacono et al., 2001;Rit-
zler & Exner, 1995;Viglione,1995; Viglione & Exner, 1995;Weiner,1995b).
Implications
These results can factor into future Rorschach meta-analytic research. Using more
appropriate and methodologically sound studies, the effect sizes may be larger and
more power can be given to the findings (overcoming Type I and II errors). Cur-
rently, two systems are being used to administer, score, and interpret the
Rorschach. The Exner CS and the Rorschach Performance Assessment System
(R-PAS; Meyer, Viglione, Mihura, Erard, & Erdberg, 2011). The CS and the
R-PAS are two entirely different systems; however, both use the same research
foundation. Specifically, R-PAS variable selection draws heavily on the meta-ana-
lyses by Mihura et al. (2013)(Erard, Meyer, & Viglione, 2014,p.172)andthe
Mihura et al. (2013) results form the foundation for the statements made
throughout Chapter 15 [of the R-PAS manual] on Variable Selection and Validity
(p. 172). Therefore, the results would also apply to the R-PAS.
However, the results of this paper should not dampen the spirts of those who
use the CS and/or R-PAS or inspire those who have criticized the Rorschach
(i.e., Wood and colleagues). It is imperative that researchers who use the
©2018 Hogrefe Publishing Rorschachiana (2018), 39(2), 180203
A Scientific Critique of Rorschach Research 195
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
Rorschach produce methodologically sound studies so that any subsequent meta-
analytic studies will have more power. Mihura et al. (2013) in a single article took
on the challenge to address the validity of each Exner variable. Although critics
(Tibon Czopp & Zeligman, 2016; Wood et al., 2015) and to a certain extent this
article have commented on it, the Mihura et al. meta-analyses began to address
the validity of each Rorschach variable.
Limitations and Future Directions
The five criteria used in the main analyses were gathered from different published
Rorschach research. However, there might be other criteria researchers may find
important to include in this type of analysis. The five criteria included were
expressed by multiple authors; therefore, there appears to be consensus about
their importance related to generalizability and validity. Additionally, the concepts
expressed in this article may be novel to the reader (application vs. validation stud-
ies; counterintuitive findings). Nevertheless, these concepts are important when
examining individual studies included in a meta-analysis. Another limitation is
that this analysis only focused on articles referenced in the Mihura et al. (2013)
meta-analyses. Therefore, other Rorschach articles not cited may have contained
less methodological bias. The Mihura et al. (2013) meta-analytic articles were used
because they form the current Rorschach validity research base. Additionally, all
articlescitedinMihuraetal.werepublishedbefore2012. Therefore, current arti-
cles may be more methodologically sound. Future studies may examine articles
from 2012 to the present with the same analyses (five criteria).
Conclusion
The following can be gleaned from this analysis:
1. Although it was published over 20 years ago, the Exner (1995a) edited book
Issues and Methods in Rorschach Research is essential reading for any
researcher using the Rorschach. The information provided should be fol-
lowed, which will result in a methodologically sound study (also see Cunliffe
et al., 2012; Gacono et al., 2001; Gacono & Gacono, 2008). Improved
research designed studies will enhance the validity of the Rorschach.
2. The articles cited in the Mihura et al. (2013) meta-analyses suffered from
methodological problems. Therefore, when referring to the meta-analytic
Rorschachiana (2018), 39(2), 180203 ©2018 Hogrefe Publishing
196 J. M. Smith et al.
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
findings, some caution is advised. Further, Mihura et al. (2016) continued a
trend to respond to criticism of the meta-analyses from a statistical perspec-
tive, rather than acknowledging and addressing issues raised by including
poorly designed studies in the meta-analyses.
3. When researchers use the Rorschach in future studies, it is imperative that
they understand what influences Rorschach research (i.e., IQ/Educational
level), increase interrater reliability, report sample parameters (M,SD, and
frequency for Lambda, R and any variables studied) and include appropriate
sample sizes. Without this information the consumer cannot determine
whether the statistical procedures in the studies were appropriate and
whether the findings can be generalized beyond the sample studied (Gacono
& Gacono, 2008; Gacono et al., 2008).
4. There is value in both application and validation studies; however, more val-
idation studies are needed to perform appropriate meta-analyses.
5. It should be noted that some of these methodological issues are not unique to
the Rorschach and they appear to be relevant in other psychological measure
research (i.e., PAI, MMPI-2).
6. Only articles published before 2012 were reviewed owing to the focus on
Mihura et al. (2013). Future research will investigate whether these issues
are present in current research (2012 and beyond).
Acknowledgments
Thank you to Dr. Eric Lugo for the Spanish translation of the summary. Opinions
expressed in this article are those of the author and do not necessarily represent
the opinions of the Federal Bureau of Prisons or the Department of Justice.
References
Abraham, P. P., Mann, T., Lewis, M. G., Coontz, B., & Lehman, M. (1990). The Cognitive
Synthesis Test and the Rorschach indicators of internal and external reality in
schizophrenic, borderline, and organic/psychotic adolescents. British Journal of Projec-
tive Psychology, 34,3444.
Acklin, M. W., & McDowell, C. I. (1995). Statistical power in Rorschach research. In J. E.
Exner (Ed.), Issues and methods in Rorschach research (pp. 181193). Hillsdale, NJ:
Erlbaum.
Ball, J. D., Archer, R. P., Gordon, R. A., & French, J. (1991). Rorschach Depression indices
with children and adolescents: Concurrent validity findings. Journal of Personality
Assessment, 57, 465476. https://doi.org/10.1207/s15327752jpa5703_6
©2018 Hogrefe Publishing Rorschachiana (2018), 39(2), 180203
A Scientific Critique of Rorschach Research 197
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
Bannatyne, L., Gacono, C., & Greene, R. (1999). Differential patterns of responding among
three groups of chronic, psychotic and forensic outpatients. Journal of Clinical
Psychology, 55(12), 15531565.
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity.
Psychological Review, 111(4), 10611071.
Brems, C., & Johnson, M. E. (1990). Further exploration of the Egocentricity Index in an
inpatient psychiatric population. Journal of Clinical Psychology, 46, 675679. https://doi.
org/10.1002/1097-4679(199009)46:5<675::AID-JCLP2270460521 3.0.CO;2-6
Bridges, M., Wilson, J., & Gacono, C. B. (1998). A Rorschach investigation of defensiveness,
self-perception, interpersonal relations, and affective states in incarcerated pedophiles.
Journal of Personality Assessment, 70, 365385.
Butcher, J. N., Dahlstrom, W. G., Graham, J. R., Tellegen, A., & Kaemmer, B. (1989).
Minnesota Multiphasic Personality Inventory2 (MMPI2): Manual for administration.
scoring, and interpretation. Minneapolis, MN: University of Minnesota Press.
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., &
Munafo, M. R. (2013). Power failure: Why small sample size undermines the reliability of
neuroscience. Nature Reviews Neuroscience, 14, 365376.
Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155159.
Cunliffe, T. B., Gacono, C. B., Meloy, J. R., Smith, J. M., Taylor, E. E., & Landry, D. (2012).
Psychopathy and the Rorschach: A response to Wood et al. (2010). Archives of
Assessment Psychology, 2(1), 131.
Dao, T. K., & Prevatt, F. (2006). A psychometric evaluation of the Rorschach Comprehensive
Systems Perceptual Thinking Index. Journal of Personality Assessment, 86, 180189.
https://doi.org/10.1207/s15327752jpa8602_07
Dies, R. (1995a). Conceptual issues in Rorschach research. In J. E. Exner (Ed.), Issues and
methods in Rorschach research (pp. 2551). Hillsdale, NJ: Erlbaum.
Dies, R. (1995b). Subject variables in Rorschach research. In J. E. Exner (Ed.), Issues and
methods in Rorschach research (pp. 99121). Hillsdale, NJ: Erlbaum.
Erard, R. E., Meyer, G. J., & Viglione, D. J. (2014). Setting the record straight: Comment on
Gurley, Piechowski, Sheehan, and Gray (2014) on the admissibility of the Rorschach
Performance Assessment System (R-PAS) in court. Psychological Injury and Law, 7, 165
177. https://doi.org/10.1007/s12207-014-9195-x
Exner, J. E. (1974). The Rorschach: A comprehensive system: Vol. 1. Basic foundations
(1st ed.). New York, NY: Wiley.
Exner, J. E. (1986). The Rorschach: A comprehensive system: Vol. 1. Basic foundations (2nd
ed.). New York, NY: Wiley.
Exner, J. E. (1992). R in Rorschach research: A ghost revisited. Journal of Personality
Assessment, 58, 245251.
Exner, J. E. (1993). The Rorschach: A comprehensive system: Vol. 1. Basic Foundations
(3rd ed.). New York, NY: Wiley.
Exner J. E. (Ed.). (1995a). Issues and methods in Rorschach research. Mahwah, NJ:
Erlbaum.
Exner, J. E. (1995b). Introduction. In J. E. Exner (Ed.), Issues and methods in Rorschach
research (pp. 124). Mahwah, NJ: Erlbaum.
Exner, J. E. (2003). The Rorschach: A comprehensive system: Vol. 1. Basic foundations and
principles of interpretation (4th ed.). Hoboken, NJ: Wiley.
Exner, J. E., Kinder, B. N., & Curtis, G. (1995). Reviewing basic design features. In J. E.
Exner (Ed.), Issues and methods in Rorschach research (pp. 145158). Mahwah, NJ:
Erlbaum.
Rorschachiana (2018), 39(2), 180203 ©2018 Hogrefe Publishing
198 J. M. Smith et al.
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
Gacono, C. B. (1997). Is the Rorschach Aggressive Movement Response enough? British
Journal of Projective Psychology, 42(2), 511.
Gacono, C. B., Evans, F. B., & Viglione, D. J. (2008). Essential issues in the forensic use of
the Rorschach. In C. B. Gacono, F. B. Evans, N. Kaser-Boyd, & L. A. Gacono (Eds.), The
handbook of forensic Rorschach assessment (pp. 321). New York, NY: Routledge.
Gacono, C. B., & Meloy, J. R. (1994). The Rorschach assessment of aggressive and
psychopathic personalities. Hillsdale, NJ: Erlbaum.
Gacono, C. B., Meloy, J. R., & Bridges, M. R. (2000). A Rorschach comparison of
psychopaths, sexual homicide perpetrators, and nonviolent pedophiles: Where angels
fear to tread. Journal of Clinical Psychology, 56, 757777. https://doi.org/10.1002/(SICI)
1097-4679(200006)56:6<757::AID-JCLP6>3.0.CO;2-I
Gacono, L. A., & Gacono, C. B. (2008). Some considerations for the Rorschach assessment
of forensic psychiatric outpatients. In C. B. Gacono, F. B. Evans, N. Kaser-Boyd, & L. A.
Gacono (Eds.), The handbook of forensic Rorschach assessment (pp. 421444). New
York, NY: Routledge/Taylor & Francis Group.
Gacono, C. B., Loving, J. L., & Bodholdt, R. H. (2001). The Rorschach and psychopathy:
Toward a more accurate understanding of the research findings. Journal of Personality
Assessment, 77,1638.
Hart, L. R. (1991). The Egocentricity Index as a measure of self-esteem and egocentric
personality style for inpatient adolescents. Perceptual and Motor Skills, 73, 907914.
https://doi.org/10.2466/PMS.73.7.907-914
Hemphill, J. F. (2003). Interpreting the magnitudes of correlation coefficients. American
Psychologist, 58,7879. https://doi.org/10.1037/0003-066X.58.1.78
Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in
research findings (2nd ed.). Thousand Oaks, CA: Sage.
Kircher, T., Krug, A., Stratmann, M., Ghazi, S., Schales, C., Frauenheim, M., & Grosvald, M.
(2014). A rating scale for the assessment of objective and subjective formal Thought and
Language Disorder (TALD). Schizophrenia research, 160(1), 216221. https://doi.org/
10.1016/j.schres.2014.10.024
Konishi, H. (2003). The lambda in the Rorschach comprehensive system [Original in
Japanese]. Shinrigaku Kenkyu, 73(6), 502505.
Mcguire, H., Kinder, B. N., Curtiss, G., & Viglione, D. J. (1995). Some special issues in data
analysis. In J. E. Exner (Ed.), Issues and methods in Rorschach research (pp. 227250).
Hillsdale, NJ: Erlbaum.
Meyer, G. J. (1992). Response frequency problems in the Rorschach: Clinical and research
implications with suggestions for the future. Journal of Personality Assessment, 58,
231244. https://doi.org/10.1207/s15327752jpa5802_2
Meyer, G. J. (1999). Simple procedures to estimate chance agreement and kappa for the
interrater reliability of response segments using the Rorschach Comprehensive System.
Journal of Personality Assessment, 72, 230255.
Meyer, G. J., Giromini, L., Viglione, D. J., Reese, J. B., & Mihura, J. L. (2015). The association
of gender, ethnicity, age, and education with Rorschach scores. Assessment, 22,4664.
https://doi.org/10.1177/1073191114544358
Meyer, G. J., Viglione, D. J., & Exner, J. E. (2001). Superiority of Form% over Lambda for
research on the Rorschach Comprehensive System. Journal of Personality Assessment,
76,6875. https://doi.org/10.1207/S15327752JPA7601_4
Meyer, G. J., Viglione, D. J., Mihura, J. L., Erard, R. E., & Erdberg, P. (2011). Rorschach
performance assessment system: Administration, coding, interpretation, and technical
manual. Toledo, OH: Rorschach Performance Assessment System.
©2018 Hogrefe Publishing Rorschachiana (2018), 39(2), 180203
A Scientific Critique of Rorschach Research 199
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
Mihura, J. L., Meyer, G. J., Dumitrascu, N., & Bombel, G. (2013). The validity of individual
Rorschach variables: Systematic reviews and meta-analyses of the comprehensive
system. Psychological Bulletin, 139, 548605. https://doi.org/10.1037/a0029406
Mihura, J. L., Meyer, G. J., Bombel, G., & Dumitrascu, N. (2015). Standards, accuracy, and
questions of bias in Rorschach meta-analyses: Reply to Wood, Garb, Nezworski,
Lilienfeld, and Duke (2015). Psychological Bulletin, 141, 250260. https://doi.org/
10.1037/a0038445
Mihura, J. L., Meyer, G. J., Bombel, G., & Dumitrascu, N. (2016). On conducting construct
validity meta-analyses for the Rorschach: A reply to Tibon Czopp and Zeligman (2016).
Journal of Personality Assessment, 98(4), 343350. https://doi.org/10.1080/
00223891.2016.1158182
Millon, T., Grossman, S., & Millon, C. (2015). Millon Clinical Multiaxial Inventory-IV manual.
Minneapolis, MN: Pearson.
Morey, L. C. (1991). Personality Assessment Inventory manual. Odessa, FL: Psychological
Assessment Resources.
Ritzler, B. A., & Exner, J. E. (1995). Special issues in subject selection and design. In J. E.
Exner (Ed.), Issues and methods in Rorschach research (pp. 123143). Hillsdale, NJ:
Erlbaum.
Sánchez-Meca, J., & Marín-Martínez, F. (2010). Meta-analysis in psychological research.
International Journal of Psychological Research, 3(1), 151163.
Shaffer, T. W., Erdberg, P., & Haroian, J. (1999). Current nonpatient data for the Rorschach,
WAIS-R, and MMPI-2. Journal of Personality Assessment, 73(2), 305316.
Simon, M. J. (1989). Comparison of the Rorschach Comprehensive Systems Isolation Index
and MMPI Social Introversion score. Psychological Reports, 65, 499502. https://doi.org/
10.2466/pr0.1989.65.2.499
Tibon Czopp, S., & Zeligman, R. (2016). The Rorschach Comprehensive System (CS)
psychometric validity of individual variables. Journal of Personality Assessment, 98(4),
335342. https://doi.org/ 10.1080/00223891.2015.1131162
Viglione, D. J. (1995). Basic considerations regarding data analysis. In J. E. Exner (Ed.),
Issues and methods in Rorschach research (pp. 195226). Hillsdale, NJ: Erlbaum.
Viglione, D. J. (1999). A review of recent research addressing the utility of the Rorschach.
Psychological Assessment, 11(3), 251265.
Viglione, D. J., & Exner, J. E. (1995). Formulating issues in Rorschach research. In J. E.
Exner (Ed.), Issues and methods in Rorschach research (pp. 5371). Hillsdale, NJ:
Erlbaum.
Viglione, D. J., & Meyer, G. J. (2008). An overview of psychometrics for forensic practice. In
C. B. Gacono, F. B. Evans, N. Kaser-Boyd, & L. A. Gacono (Eds.), The handbook of
forensic Rorschach assessment (pp. 2154). New York, NY: Routledge.
Weiner, I. B. (1966). Psychodiagnostics in schizophrenia. New York, NY: Wiley.
Weiner, I. B. (1995a). Methodological considerations in Rorschach research. Psychological
Assessment, 7(3), 330337.
Weiner, I. B. (1995b). Variable selection in Rorschach research. In J. E. Exner (Ed.), Issues
and methods in Rorschach research (pp. 7397). Hillsdale, NJ: Erlbaum.
Weiner, I. B. (2003). Principles of Rorschach interpretation (2nd ed.). Mahwah, NJ: Erlbaum.
Wood, J. M., Garb, H. N., Nezworski, M. T., Lilienfeld, S. O., & Duke, M. C. (2015). A second
look at the validity of widely used Rorschach indices: Comment on Mihura, Meyer,
Dumitrascu, and Bombel (2013). Psychological Bulletin, 141, 236249. https://doi.org/
10.1037/a0036005
Rorschachiana (2018), 39(2), 180203 ©2018 Hogrefe Publishing
200 J. M. Smith et al.
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
Wood, J. M., Lilienfeld, S. O., Nezworski, M. T., Garb, H. N., Holloway Allen, K., &
Wildermuth, J. L. (2010). Validity of Rorschach inkblot scores for discriminating
psychopaths from nonpsychopaths in forensic populations: A meta-analysis. Psycho-
logical Assessment, 22(2), 336349. https://doi.org/10.1037/a0018998
Wood, J. M., Nezworski, M. T., & Stejskal, W. J. (1996). The Comprehensive System for the
Rorschach: A critical examination. Psychological Science, 7,310. https://doi.org/
10.1111/j.1467-9280.1996.tb00658.x
Zillmer, E. A., & Vuz, J. K. (1995). Factor analysis with Rorschach data. In J. E. Exner (Ed.),
Issues and methods in Rorschach research (pp. 251306). Hillsdale, NJ: Erlbaum.
Received December 3, 2016
Revision received October 22, 2017
Accepted February 5, 2018
Published online November 9, 2018
Jason M. Smith
FCC Hazelton
Bruceton Mills, WV
USA
jmsmithpsyd@gmail.com
Summary
ExnersIssues and Methods in Rorschach Research (1995a) provided a standard of care for conduct-
ing Rorschach research; however, the extent to which studies have followed these guidelines has
not been examined. Similarly, meta-analytic approaches have been used to comment on the valid-
ity of Comprehensive System (CS) variables without an evaluation as to the extent the individual
studies have conformed to proposed methodological criteria (Exner, 1995a; Gacono, Loving, &
Bodholdt, 2001). In this article, the 210 studies cited in recent Mihura, Meyer, Dumitrascu, and
Bombel (2013) meta-analyses were examined. Individual studies were analyzed for: research on
the Rorschach versus research with the Rorschach and whether they met the threshold of valid-
ity/generalizability related to specific Rorschach criteria. These criteria were: (1)IQ/EducationLe-
vel; (2) Responses; (3) Lambda; (4) interrater reliability; and (5) sample size. Out of 210,104
(49.5%) studies focused on research on the Rorschach and none met all five Rorschach criteria as-
sessed. Further, 90%ofthestudiesexaminedhadthreeormore issues related to the above cri-
teria. Therefore, the Exner (1995a) edited book Issues and Methods in Rorschach Research is
essential reading for any researcher using the Rorschach. When researchers use the Rorschach
in future studies, it is imperative that they understand what influences Rorschach research (i.e.,
IQ/Education level), increase interrater reliability, use appropriate statistics, report sample param-
eters (M,SD, and frequency for Lambda/R and any variables studied), and include appropriate
sample sizes. Without this information the consumer cannot determine whether the statistical pro-
cedures in the studies were appropriate and whether the findings can be generalized beyond the
sample studied. Further, there is value in both application and validation studies; however, more
validation studies are needed to perform appropriate meta-analyses. The results of this paper
should not dampen the spirts of those who use the CS and/or R-PAS or inspire those who have
criticized the Rorschach (i.e., Wood and colleagues).Itisimperativethatresearcherswhouse
the Rorschach produce methodologically sound studies so that any subsequent meta-analytic stud-
ies will have more power.
©2018 Hogrefe Publishing Rorschachiana (2018), 39(2), 180203
A Scientific Critique of Rorschach Research 201
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
Résumé
Louvrage Issues and Methods in Rorschach Research (Exner, 1995a) présente des recommandations
concernant les standards de la recherche Rorschach; cependant, la mesure dans laquelle les re-
cherches publiées ont suivi ces lignes directrices na pas été examinée. De même, des approches
méta-analytiques ont permis dévaluer la validité des variables du Système Intégré, mais sans prise
en compte de la qualité méthodologique des études examinées au regard des standards proposés
par Exner, 1995a; puis Gacono, Loving & Bodholdt (2001). Dans cet article, les 210 études citées
dans les dernières méta-analyses de Mihura, Meyer, Dumitrascu et Bombel (2013)ontéexam-
inéesselonlescritèressuivants:recherchedevalidation(surleRorschach)vsrecherchedappli-
cation (avec Rorschach) dunepartetdautre part les critères suivants : (1) IQ / niveau
déducation; (2) Nombre de Réponses; (3) Lambda; (4) fiabilité inter-évaluateur; et (5)Taillede
léchantillon. Sur 210,104 (49,5%) des études étaient des études de validation à proprement parler
et aucune ne remplissait lensemble des critères méthodologiques évalués. En outre, 90%des
études examinées présentait au moins trois problèmes méthodologiques. Par conséquent, le livre
Issues and Methods in Rorschach Research (Exner, 1995a) est une lecture essentielle pour tout
chercheur utilisant le Rorschach. A lavenir, il est impératif que les chercheurs comprennent les
facteurs susceptibles dinfluencer la recherche sur le Rorschach : QI, fiabilité inter-évaluateur, util-
isation des statistiques appropriées, présentation détaillée des paramètres de léchantillon (M,SD
et fréquence pour Lambda / R et toutes les variables étudiées) et tailles déchantillon appropriées.
Sans ces informations, le consommateur ne peut pas déterminer si les procédures statistiques em-
ployées dans une étude étaient appropriées et si les résultats peuvent être généralisés au-delà de
léchantillon étudié. En outre, si les études dapplication et de validation présentent chacune leut
intérêt; il semble nécessaire dentreprendre de nouvelles études de validation pour effectuer des
méta-analyses appropriées. Les résultats de cet article ne devraient pas décourager ceux qui utili-
sent le CS et / ou le R-PAS ou encourager ceux qui ont critiqué le Rorschach (cest-à-dire Wood et
ses collègues). Il est impératif que les chercheurs qui utilisent le Rorschach produisent des études
solides sur le plan méthodologique ce qui permettra daugmenter la puissance de méta-analyses
ultérieures.
Resumen
Exner (1995a) Issues and Methods in Rorschach Research proporcionó un estándar de precaución
paralarealizacióndelainvestigacióndeRorschach.Sinembargo,nosehaexaminadohasta
qué punto los estudios han seguido estas pautas. Del mismo modo, los enfoques meta-analíticos
se han utilizado para comentar la validez de las variables de CS sin una evaluación que incluya
los estudios individuales ajustado a los criterios metodológicos propuestos (Exner, 1995a; Gacono,
Loving, & Bodholdt, 2001). En este artículo se examinaron los 210 estudios citados en los recien-
tes meta-análisis de Mihura, Meyer, Dumitrascu y Bombel (2013). Se analizaron los estudios indi-
viduales para: investigación sobre el Rorschach versus investigación con el Rorschach ysicumplíanel
umbral de validez / generalización relacionado con los criterios específicos de Rorschach. Estos
criterios fueron 1) CI / Nivel de educación; 2) Respuestas; 3) Lambda; 4) Confiabilidad entre eval-
uadores; y 5)Tamañodemuestra.De210,104 (49.5%) estudios se enfocaron en la investigación
del Rorschach y ninguno cumplió con los cinco criterios de Rorschach evaluados. Además, el 90%
delosestudiosexaminadosteníantresomásproblemasrelacionadosconloscriteriosmenciona-
dos anteriores. Por lo tanto, el libro editado de Exner (1995a) Issues and Methods in Rorschach Re-
search es una lectura esencial para cualquier investigador que use el Rorschach. Cuando los
investigadores utilicen el Rorschach en estudios futuros, es imperativo que entiendan qué
Rorschachiana (2018), 39(2), 180203 ©2018 Hogrefe Publishing
202 J. M. Smith et al.
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
influencia tiene la investigación de Rorschach (e.g. CI), aumento de la confiabilidad entre evalu-
adores, uso de estadísticas apropiadas, información de los parámetros de la muestra (M, DE, y fre-
cuencia para Lambda / R y cualquier variable estudiada) e incluye tamaños de muestra
apropiados. Sin esta información, el consumidor no puede determinar si los procedimientos
estadísticos de los estudios fueron apropiados y si los resultados pueden generalizarse más allá
de la muestra estudiada. Además, hay validez en los estudios de aplicación y validación; sin em-
bargo, se necesitan más estudios de validez para realizar los meta-análisis apropiados. Los resul-
tados de este documento no deben desalentar a los espíritus de quienes usan el CS y / o R-PAS o
inspirar a aquellos que han criticado el Rorschach (e.g. Wood y sus colegas). Es imperativo que los
investigadores que usan el Rorschach produzcan estudios metodológicamente sólidos para que
cualquier estudio meta-analítico posterior tenga más poder.
Exner (1995a) のロールシャッハ
Exner1995aロールシャッハはロールシャッハをおこなうすべき
したが、どのがこのガイドラインにってされたのかについてはされてきて
いない。に、ロールシャッハシステムのについてコメントするのにメタアプローチ
いられてきているが、々のされたExner, 1995; Gacono, Bodholdt,
&Loving,2001してどのしているかについてのはされていない。では
Mihura, Meyer, Dumitrascu, and Bombel (2013)された210のメタをおこなった。
々ののようにされたロールシャッハの  ロールシャッハをいての、これら
のロールシャッハのするしているかどうか。これらの
のようなものである。1IQレベル、23ラムダ、4
5サンプルサイズ。210のうち、10449.5%)はロールシャッハのてたものであ
り、これらのにロールシャッハの5つのをすべてたしたものはなかった。さらに90がこれら
する3つかそれ以上していることがらかになった。ゆえに、Exner (1995)
したロールシャッハはロールシャッハをいるにとってのであると
る。でロールシャッハをもちいようとするがロールシャッハの
すればIQめるかをし、使い、のパラメー
MSD、ラムダRや、されているし、なサンプルサイズをむこと
になる。これらのなしではは、こきがかどうか、され
たことはのサンプルをえてできるかどうかをめることはできない。さらに、
であるが、さらになメタするがある。
システムR-PAS使するもののさせ、ロールシャッハをするものWoodやそ
づけるものではない。ロールシャッハをいる
ことがであり、にメタになるであろう。
©2018 Hogrefe Publishing Rorschachiana (2018), 39(2), 180203
A Scientific Critique of Rorschach Research 203
${protocol}://econtent.hogrefe.com/doi/pdf/10.1027/1192-5604/a000102 - Jason M. Smith <jmsmithpsyd@gmail.com> - Friday, November 09, 2018 3:19:11 AM - IP Address:98.236.88.255
... Assessing the current psychopathy literature makes one realize that a plethora of largely unchallenged biases, unsound methodology, and faulty conclusions predominate the landscape Gacono, 2016;Smith et al., 2021). These poorly designed studies become part of meta-analytic ones, which add conflicting findings to an already confusing literature (Cunliffe et al., 2012;Gacono, 2019;Smith, Gacono, Fontan, et al., 2018. Taken together, disparate findings-artifacts of poor research-create "apparent controversies," which lead to "pseudo-debates," and add to the "armchair quality" of the Psychopathy Checklist-Revised (PCL-R; Hare, 2003) literature (Gacono, 2016(Gacono, , 2019(Gacono, , 2021Hare, 1998). ...
... This is especially critical for meta-analytic studies that hide the internal validity issues of individual studies within a group of studies where the flawed designs are glossed over with the glitter of statistical analysis (Gacono, 2019). As we have done for two Rorschach meta-analyses (Mihura et al., 2013;Wood et al., 2010), individual studies must be examined related to potential internal validity issues (Cunliffe et al., 2012;Smith, Gacono, Fontan, et al., 2018. The journal review process is also impacted by a partial understanding of the issues and misinformation created by these studies. ...
... Criterion 3: Were the appropriate statistics used? Though there is no accepted value for sample size, it has been suggested that less than 20 is problematic (Smith, Gacono, Fontan et al., 2018. This study had 16 male offenders with PCL: SV ≥13 (n = 5; 13-17), with only 11 scoring 18 or above (possible psychopaths with PCL-R ≥ 30), compared to a control sample of 35. ...
Article
Full-text available
Erratum--please note on page 116, the total score of 34 should be 37. Also, the text should read "might receive" instead of receive, accounting for the potential individual items scoring.
... Assessing the current psychopathy literature makes one realize that a plethora of largely unchallenged biases, unsound methodology, and faulty conclusions predominate the landscape Gacono, 2016;Smith et al., 2021). These poorly designed studies become part of meta-analytic ones, which add conflicting findings to an already confusing literature (Cunliffe et al., 2012;Gacono, 2019;Smith, Gacono, Fontan, et al., 2018. Taken together, disparate findings-artifacts of poor research-create "apparent controversies," which lead to "pseudo-debates," and add to the "armchair quality" of the Psychopathy Checklist-Revised (PCL-R; Hare, 2003) literature (Gacono, 2016(Gacono, , 2019(Gacono, , 2021Hare, 1998). ...
... This is especially critical for meta-analytic studies that hide the internal validity issues of individual studies within a group of studies where the flawed designs are glossed over with the glitter of statistical analysis (Gacono, 2019). As we have done for two Rorschach meta-analyses (Mihura et al., 2013;Wood et al., 2010), individual studies must be examined related to potential internal validity issues (Cunliffe et al., 2012;Smith, Gacono, Fontan, et al., 2018. The journal review process is also impacted by a partial understanding of the issues and misinformation created by these studies. ...
... Criterion 3: Were the appropriate statistics used? Though there is no accepted value for sample size, it has been suggested that less than 20 is problematic (Smith, Gacono, Fontan et al., 2018. This study had 16 male offenders with PCL: SV ≥13 (n = 5; 13-17), with only 11 scoring 18 or above (possible psychopaths with PCL-R ≥ 30), compared to a control sample of 35. ...
Article
Full-text available
Recently in Psychoanalytic Psychology, Gullhaugen et al. (2021) proposed a Dynamic Model of Psychopathy (DMP) to better understand psychopathic traits. Several issues with the authors' methodology, including the use of the Psychopathy Checklist: Screening Version (PCL: SV) as an independent measure and a small sample size (N = 16) relative to their conceptual approach and the number of statistical analyses conducted, limit the conclusions that can be drawn from their data. Additionally, the authors discuss their findings as if the data from this study with all males could apply to women. In this article, we use the methodological issues presented in the Gullhaugen study to discuss problems with the broader psychopathy literature. We also provide a psychodynamic model of psychopathy consistent with theory and empirical data.
... Therefore, it is not recommended the findings be compared to males in medium-security prisons. 8 Protocols were excluded for PAI scores of INF > 74, ICN > 73, and/or NIM > 76 (Morey, 1991) or on the Rorschach, low IQ (< 80) and/or less than 14 responses (Exner, 2003;Smith, Gacono, Fontan et al., 2018;Smith et al., 2020). One was removed for an extreme number of Rorschach responses (R ¼ 90). ...
Article
Full-text available
Managing the incarcerated population is the primary task within correctional settings. Using psychological assessment to predict institutional behavior, the psychologist has a unique set of skills essential to the management of prisoners. PCL-R, PAI, and Rorschach data were compared with institutional infractions (total, physical, verbal, non-aggressive) among 126 incarcerated women. Multiple binary logistic regression analyses were used which found significant correlations between PCL-R total score, PAI scales (BOR, ANT, VPI), and Rorschach variables (ROD, EGOI, TCI, AgPot, AgPast, SumV, SumC’, MOR) with total, verbal, physical, and nonviolent incident reports. Each of these measures adds incrementally to the assessment and understanding of institutional misbehavior for incarcerated women. Clinical implications of the findings were presented.
... To overcome these biases and to broaden the somewhat narrowed symptom-focused view of mental disorders, psychological scientists have been trying for decades to assess hidden psychological constructs and personality patterns through indirect projective psycho-diagnostic tools, such as the Rorschach test or the Thematic Apperception Test (TAT) (Groth-Marnat, 2009). However, these tools are often being criticized as having poor inter-rater reliability and poor criterion validity, and their usage in clinical/research settings is controversial (Groth-Marnat, 2009;Lilienfeld et al., 2001;Smith et al., 2018). Moreover, such indirect tools are applied only sparingly, among a small portion of the population because they require large investments of time and money. ...
Article
Full-text available
Suicide, a leading cause of death, is a complex and a hard-to-predict human tragedy. In this article, we introduce a comprehensive outlook on the emerging movement to integrate computational linguistics (CL) in suicide prevention research and practice. Focusing mainly on the state-of-the-art deep neural network models, in this “travel guide” article, we describe, in a relatively plain language, how CL methodologies could facilitate early detection of suicide risk. Major potential contributions of CL methodologies (e.g., word embeddings, interpretational frameworks) for deepening that theoretical understanding of suicide behaviors and promoting the personalized approach in psychological assessment are presented as well. We also discuss principal ethical and methodological obstacles in CL suicide prevention, such as the difficulty to maintain people’s privacy/safety or interpret the “black box” of prediction algorithms. Ethical guidelines and practical methodological recommendations addressing these obstacles are provided for future researchers and clinicians.
... To overcome these biases and to broaden the somewhat narrowed symptom-focused view of mental disorders, psychological scientists have been trying for decades to assess hidden psychological constructs and personality patterns through indirect projective psycho-diagnostic tools, such as the Rorschach test or the Thematic Apperception Test (Groth-Marnat, 2009). However, these tools are often criticized as having poor interrater reliability and poor criterion validity, and their usage in clinical/research settings is controversial (Groth-Marnat, 2009;Lilienfeld et al., 2001;Smith et al., 2018). Moreover, such indirect tools are applied only sparingly, among a small portion of the population, because they require large investments of time and money. ...
Preprint
Suicide, a leading cause of death, is a complex and a hard-to-predict human tragedy. This article introduces a comprehensive outlook on the emerging movement to integrate Computational Linguistics (CL) in suicide prevention research and practice. Focusing mainly on the state-of-the-art Deep Neural Network models, this "travel guide" article describes, in a relatively plain language, how CL methodologies could facilitate early detection of suicide risk (section 1). Major potential contributions of CL methodologies (e.g., word embeddings, interpretational frameworks) for deepening our theoretical understanding of suicide behaviors (section 2) and promoting the personalized approach in psychological assessment (section 3), are presented as well. Importantly, the article also discusses principal ethical (section 4) and methodological (section 5) obstacles in CL-suicide prevention, such as the difficulty to maintain peoples' privacy/safety or interpret the "black box" of prediction algorithms. Ethical guidelines and practical methodological recommendations addressing these obstacles, are provided for future researchers and clinicians.
... The Rorschach Comprehensive System (CS; Exner, 1974Exner, , 2003 can be controversial but is still a frequently used evaluation method in clinical practice (Wright et al., 2017). It also continues to be refined in the research context (Smith et al., 2018). ...
Article
The administration process of the Rorschach test is of utmost importance as it influences both the coding and the interpretative procedures. Performing it appropriately requires complex skills, knowledge, and solid training. The aim of the study is to describe students’ interests in and difficulties with administering the Rorschach (Comprehensive System) for the first time. A two-phase methodology, including an analysis of questionnaire responses followed by a study of students’ written narratives, using Iramuteq textual analysis software, was implemented with two different samples of third-year undergraduates (including 63 and 253 participants, respectively), recruited from a French psychology school. Our results show that students have a strong interest in understanding the test and wish to use it in their future practice. When administering the Rorschach for the first time, students find it difficult to cope with the complexity of the procedures at a technical, emotional, and relational level.
Article
This manuscript presents a single case study of a psychotically disturbed adult male (whom we call “Peter”), focusing on similarities and differences in Rorschach interpretation based on three different Rorschach approaches. Specific questions were raised as to whether the client suffered from a paranoid psychosis (paranoia) or paranoid schizophrenia. Three distinct models of psychopathology and Rorschach interpretation are initially presented. We then address Peter’s psychotic symptoms, according to the Parisian approach (specifically the Nancy French subgroup), the Lausanne Rorschach approach, and the American Rorschach approach (Comprehensive System and R-PAS). Analysis shows many convergences between the three approaches on the client’s nature of conflicts and links to reality, object relations, self-representation and anxiety, defense mechanisms, and disordered thinking, but interpretation of these variables differed somewhat despite agreement on a diagnosis within the psychotic spectrum. Concluding remarks discuss the divergences and point out the limitations of a case study method. Future research is suggested.
Chapter
Long before psychology, bias has existed in science. From the beginning, concerns have been raised about the reliability, validity, and accuracy of social science research (Meehl, 1954). In this chapter, we define and discuss the origins of bias and how it can erode the scientific method. We focus specifically on bias in psychological research, theory, assessment, and treatment. We discuss the range of common misconceptions and misinformation that permeates the female offender literature. Finally, we conclude with ten myths about female offenders and offer guidelines for identifying bias and how to avoid it.
Chapter
Historically, the cornerstone of the psychologist’s identity rested on providing competent in-depth psychological assessment (Rapaport, Gill, & Schafer, 1946). The ability to utilize a battery of assessment methods to elucidate complex issues makes the psychologist unique among other mental health professionals. Recent trends, however, have tarnished that cornerstone. Not surprisingly, the movement away from proficiency in psychological assessment has led to a decline in the need for psychologists. In this chapter, we discuss these harmful trends, define psychological assessment, offer a model for assessing female offenders, and provide examples of how record review, clinical interview, the Hare Psychopathy Checklist-Revised (PCL-R), Personality Assessment Inventory (PAI), and Rorschach Inkblot test can be useful with female offenders. We discuss the interpersonal aspects of the assessment process, evaluate gender specific patterns for several PCL-R criteria (also see Appendices A & B), and provide caveats for assessing female offenders. We conclude with a case study.
Chapter
In this chapter, we provide a theoretical and empirically based understanding of antisocial and psychopathic women. We begin by clarifying the differences between psychopathy, sociopathy, and ASPD, and then provide a historical perspective of hysteria. While the underlying personality of the female psychopath is paranoid, malignant hysteria is their predominant personality style (Gacono & Meloy, 1994). Overviews of the Hare Psychopathy Checklist-Revised (PCL-R), Personality Assessment Inventory (PAI), and Rorschach are offered as a refresher for those experienced clinicians and as a resource for those that are not. Finally, we present group PAI and Rorschach data (also Trauma Symptom Inventory-2 [TSI-2]) for 337 female offenders including subsets of psychopathic (N = 124) and non-psychopathic (N = 57) females. We make note of the differences between female and male psychopaths.
Article
Full-text available
Wood et al. (2010) published a meta-analysis in which the authors challenged the utility of the Rorschach Inkblot Test in delineating key differences between psychopathic and non-psychopathic individuals identified by the Psychopathy Checklist-Revised (PCL-R; Hare, 1991/2003). In this article, Gacono et al.’s (2001) five conceptual and four methodological criteria for the evaluation and interpretation of psychopathy/Rorschach literature were employed to provide a detailed review of the approach and procedure used by Wood et al. (2010). We identify and discuss a number of conceptual and methodological problems with the meta-analysis including confusion of the related but distinct terms of diagnosis and assessment, selection of studies, categorical versus dimensional interpretations of data, characterization of PCL-R and Rorschach findings, and meta-analytic methodology. Finally, recommendations for the essential components of well designed and implemented PCL-R and Rorschach studies are provided.
Article
The Personality Assessment Inventory (PAI) is an objective self‐report measure of adult personality with clinical, forensic, and personnel selection applications. Comprised of 22 non‐overlapping scales, the PAI was developed using a construct validation framework. Concurrent and construct validity, as well as diagnostic utility, are well‐established. Internal consistency and test‐retest reliability are similarly strong. Strengths of the PAI include its low reading level, multiple large sample norms, readily understood constructs, and quick scoring. Limitations include the absence of some constructs that may be of clinical interest and its use with individuals not fluent in English or Spanish.
Article
We respond to Tibon Czopp and Zeligman's (2016) critique of our systematic reviews and meta-analyses of 65 Rorschach Comprehensive System (CS) variables published in Psychological Bulletin (2013). The authors endorsed our supportive findings but critiqued the same methodology when used for the 13 unsupported variables. Unfortunately, their commentary was based on significant misunderstandings of our meta-analytic method and results, such as thinking we used introspectively assessed criteria in classifying levels of support and reporting only a subset of our externally assessed criteria. We systematically address their arguments that our construct label and criterion variable choices were inaccurate and, therefore, meta-analytic validity for these 13 CS variables was artificially low. For example, the authors created new construct labels for these variables that they called "the customary CS interpretation," but did not describe their methodology nor provide evidence that their labels would result in better validity than ours. They cite studies they believe we should have included; we explain how these studies did not fit our inclusion criteria and that including them would have actually reduced the relevant CS variables' meta-analytic validity. Ultimately, criticisms alone cannot change meta-analytic support from negative to positive; Tibon Czopp and Zeligman would need to conduct their own construct validity meta-analyses.
Article
Elements of response style were examined among three groups of chronic, psychotic, forensic patients: paranoid schizophrenics(N = 89); undifferentiated-disorganized schizophrenics ( N = 38), and schizoaffective patients(N = 53). Forensic patients with elevated MMPI-2 L Scales produced increased percentages of Pure Form (F%) on the Rorschach. A similar relationship occurred when the Rorschach was used as the independent measure. Schizoaffective patients reported more psychotic symptoms on the MMPI-2 and lower F% (Rorschach) than both schizophrenic groups. Although undifferentiated schizophrenics evidenced the most psychopathology on the Rorschach (impaired reality testing and perceptual accuracy disturbance), all three groups produced lower than expected frequencies for Rorschach variables commonly associated with thought disorder and poor reality testing (Exner, 1995b). The clinical importance of using the MMPI-2 and Rorschach in tandem with forensic psychiatric patients is discussed. Our empirical findings suggest the need for forensic evaluators to consider the important relationship between psychiatric diagnosis and response style (defensiveness, denial, illness chronicity, medications, and concurrent Axis II psychopathology) when interpreting often constricted psychological testing protocols in chronic forensic patient populations. (C) 1999 John Wiley & Sons, Inc.
Article
Since the publication of the Rorschach Inkblot Method (Rorschach, 1921/1942 ), theorists, researchers, and practitioners have been debating the nature of the task, its conceptual foundation, and most important its psychometric properties. The validity of the Rorschach Comprehensive System (CS; Exner, 1974 , 2003; Exner & Weiner, 1995 ) has been supported by several meta-analyses that used different types of nontest external criterion for validating individual variables. In a recent meta-analysis, Mihura, Meyer, Dumitrascu, and Bombel ( 2013 ) found coefficients ranging from modest to excellent for most of the selected CS variables, with 13 of them reported as showing "little to no support." This article focuses on these variables. Although endorsing Mihura et al.'s mainly validating findings, we also suggest that the evidence presented for the little or no validity of these 13 variables is not quite compelling enough to warrant changing their definition or coding, or removing them from the system. We point to some issues concerning the description and interpretation of these variables and the appropriateness of the external criteria used for exploring their validity, and suggest considering these issues in further CS research. Implications of Mihura et al.'s meta-analysis for clinical and forensic practice are discussed.
Article
This second edition of Irving Weiner's classic comprehensive, clinician-friendly guide to utilizing the Rorschach for personality description has been revised to reflect both recent modifications in the Rorschach Comprehensive System and new evidence concerning the soundness and utility of Rorschach assessment. It integrates the basic ingredients of structural, thematic, behavioral, and sequence analysis strategies into systematic guidelines for describing personality functioning. It is divided into three parts. Part I concerns basic considerations in Rorschach testing and deals with conceptual and empirical foundations of the inkblot method and with critical issues in formulating and justifying Rorschach inferences. Part II is concerned with elements of interpretation that contribute to thorough utilization of data in a Rorschach protocol: the Comprehensive System search strategy; the complementary roles of projection and card pull in determining response characteristics; and the interpretive significance of structural variables, content themes, test behaviors, and the sequence in which various response characteristics occur. Each of the chapters presents and illustrates detailed guidelines for translating Rorschach findings into descriptions of structural and dynamic aspects of personality functioning. The discussion throughout emphasizes the implications of Rorschach data for personality assets and liabilities, with specific respect to adaptive and maladaptive features of the manner in which people attend to their experience, use ideation, modulate affect, manage stress, view themselves, and relate to others. Part III presents 10 case illustrations of how the interpretive principles delineated in Part II can be used to identify assets and liabilities in personality functioning and apply this information in clinical practice. These cases represent persons from diverse demographic backgrounds and demonstrate a broad range of personality styles and clinical issues. Discussion of these cases touches on numerous critical concerns in arriving at different diagnoses, formulating treatment plans, and elucidating structural and dynamic determinants of behavior. © 2003 by Lawrence Erlbaum Associates, Inc. All rights reserved.