Understanding Rorschach Research: Using the Mihura (2019) Commentary as a Reference

Abstract

Mihura’s (2019) recent commentary on Smith et al.’s (2018) article, “A Scientific Critique of Rorschach Research: Revisiting Exner’s Issues and Methods in Rorschach Research (1995)” raised several issues surrounding our internal validity criteria and our approach to validating Rorschach research. Mihura also conducted additional statistical analyses that failed to address important, critical issues. In this article, we further clarify the importance of refining internal validity criteria for Rorschach research and Rorschach meta-analytic studies (our 2018 article). We offer this information and analysis to guide the Rorschach consumer toward a better understanding of how to assess the validity of Rorschach data and empirical findings. Criticisms of our criteria for evaluating Rorschach research (inter-rater reliability, IQ/Education level, Rorschach Responses, Lambda/F%, sample size) and our recommendation to include descriptive data for critical variables in all published Rorschach studies contradict accepted standards for Rorschach research. Mihura stated that “the associations between the Rorschach and self-report measures were not used to determine the validity of Rorschach variables” (p. 171) in her meta-analyses. Consequently, we applied our methodological criteria to only Mihura’s externally assessed criteria studies (removing self-report studies; 67%) and found that 91% had three or more problems related to internal validity (13 had counter-intuitive findings). In addition, other central issues related to meta-analyses, application/validation studies, and counter-intuitive findings regarding the Rorschach were discussed.

Full text

SIS J. Proj. Psy. & Ment. Health (2020) 27: 71-82
1We have used the Mihura commentary to provide further clarity to the above issues. Also see our online supplemental points at
https://mfr.osf.io/render?url=https%3A%2F%2Fosf.io%2Fg9thn%2Fdownload
Jason M. Smith, PsyD, ABPP, Pierpont Community & Technical College, Fairmont, WV jmsmithpsyd@gmail.com,
Carl B. Gacono, PhD, ABAP Private Practice, Asheville, NC, Patrick Fontan, PhD, School Psychologist, Cachan,
France Ted B. Cunliffe, PhD, Private Practice, Miami, FL, Anne Andronikof, PhD, Emeritus Professor at the
University Paris Nanterre, France
Key Words: Rorschach; CS; R-PAS; meta-analyses
Understanding Rorschach Research: Using the Mihura (2019) Commentary as a Reference
Jason M. Smith, Carl B. Gacono, Patrick Fontan, Ted B. Cunliffe, & Anne Andronikof
0LKXUD¶V  UHFHQW FRPPHQWDU\ RQ 6PLWK HW DO¶V  DUWLFOH ³$ 6FLHQWLILF Critique of Rorschach
5HVHDUFK 5HYLVLWLQJ ([QHU¶V ,VVXHV DQG 0HWKRGV LQ 5RUVFKDFK 5HVHDUFK ´ UDLVHG VHYHUDO LVVXHV
surrounding our internal validity criteria and our approach to validating Rorschach research. Mihura also
conducted additional statistical analyses that failed to address important, critical issues. In this article, we
further clarify the importance of refining internal validity criteria for Rorschach research and Rorschach meta-
analytic studies (our 2018 article). We offer this information and analysis to guide the Rorschach consumer
toward a better understanding of how to assess the validity of Rorschach data and empirical findings. Criticisms
of our criteria for evaluating Rorschach research (inter-rater reliability, IQ/Education level, Rorschach
Responses, Lambda/F%, sample size) and our recommendation to include descriptive data for critical variables
in all published Rorschach studies contradict accepted standards for Rorschach research. Mihura stated that
³WKHDVVRFLDWLRQV EHWZHHQ Whe Rorschach and self-report measures were not used to determine the validity of
5RUVFKDFK YDULDEOHV´ S  LQKHU PHWD-analyses. Consequently, we applied our methodological criteria to
RQO\0LKXUD¶V H[WHUQDOO\DVVHVVHG FULWHULDVWXGLHVUHPRYLQJVHOI-report studies; 67%) and found that 91% had
three or more problems related to internal validity (13 had counter-intuitive findings). In addition, other central
issues related to meta-analyses, application/validation studies, and counter-intuitive findings regarding the
Rorschach were discussed.
Introduction:
7KH JRDOV RI WKH 6PLWK HW DO  DUWLFOH ³$
Scientific Critique of Rorschach Research:
5HYLVLWLQJ([QHU¶V,VVXHVDQG0HWKRGVLQ
5RUVFKDFK 5HVHDUFK ´ ZHUH WR  EHWWHr
understand the quality of Rorschach research
studies--XVLQJ FULWHULD SUHVHQWHG LQ ([QHU¶V
(1995a) edited Rorschach research book and more
recent evaluation criteria (Cunliffe et al., 2012;
Gacono, Loving, & Bodholdt, 2001); 2) to offer
understandable criteria for the Rorschach
practitioner, manuscript reviewer, and researcher
to assess the validity of Rorschach research; 3) to
challenge, support, or offer cautions on the use of
meta-analytic articles that significantly impact the
applied usage and acceptance of the Rorschach and
other psychological measures (also see
Andronikof, 2019; Fontan, 2019; Fontan & Smith,
2018; Gacono, 2019); and 4) ultimately to help
reduce poorly designed studies that contribute to
apparent controversies in the Rorschach
literature²at a time when this body of Rorschach
research mostly contribute to the devaluation of
the instrument.
Primarily using the meta-analyses of Mihura,
Meyer, Dumitrascu, and Bombel (2013) as our
³VXEMHFW´ DV LW ZDV D UHFHQW VWXG\ ZH VRXJKW WR
examine the methodology used in individual
studies (application versus validation) and to
determine if these studies accounted for inter-rater
reliability, presented descriptive statistics for
IQ/Education, Lambda/F%, number of Rorschach
Responses, and had an appropriate sample size
(these would be found in a method section, hence
methodology). The importance of these
descriptions has been discussed in depth
previously, and in the case of Lambda more
recently (see Gacono, 2019).
Our article resulted in a critique from Mihura
(2019) published in Rorschachiana. As we were
not offered the opportunity to provide a comment
to further clarify any unresolved issues, one is
offered here1. It was suggested that we attempted
to evaluate the impact of Exner (1995a) before and

72: Smith et al
DIWHULWVSXEOLFDWLRQVSHFLILFDOO\VWDWLQJDQ³DUWLFOH
[that] is framed as a formal evaluation of whether
UHVHDUFKHUV KDYH IROORZHG ([QHU¶V D
recommendations in his Rorschach methodology
book± EXW WKLV UHVHDUFK TXHVWLRQ LV QRW WHVWHG´
(Mihura, p.  +RZHYHURXU SXUSRVH ZDV ³,Q
this article, the studies from the Mihura et al.
(2013) meta-analyses were analyzed examining
their quality in light of Exner (1995a) and others
&XQOLIIHHWDO*DFRQRHWDO´6PLWK
et al., 2018, p. 188).
2XU REMHFWLYHV ZHUH QHLWKHU WR DVVHVV UHVHDUFKHUV¶
FRPSOLDQFH WR ([QHU¶V DQG KLV FR-contributors
(1995a) research criteria, nor to determine if
researchers had been applying these criteria after
1995. Our point was and is that these well
accepted criteria, should have been considered
when selecting studies for inclusion in Rorschach
meta-analyses, because retrospectively, these
criteria, when not followed, are known to lower
internal validity. Mihura et al. (2013) included
these studies in their meta-analyses on the external
validity of Rorschach scores without examining
the quality of the studies nor determine if the
studies had any internal validity issues.
We have not been the only ones to offer concerns
about the Mihura et al. (2013) meta-analyses.
Tibon Czopp and Zeligman (2016) stated there
were concerns with the descriptions and external
criteria of some of the Rorschach variables used in
the meta-analyses. As a significant support for the
Rorschach Performance Assessment System (R-
PAS; Meyer, Viglione, Mihura, Erard, & Erdberg,
2011) comes from these meta-analyses, it is
imperative that the data from this different
Rorschach system be open to scientific scrutiny.
Considering scientific considerations and ethical
guidelines, we provided our data for the Mihura
(2019) commentary. In preparation of this current
commentary, we asked for the Mihura et al. (2013)
meta-analytic data in order to review meta-analytic
decisions and judgments; unfortunately, this
request was rejected.
Background:
Scientific collaboration: Mihura (2019) stated
WKDW ³>V@FLHQFH PXVW EH DEOH WR LQFRUSRUDWH
corrective feedback and to criticize itself.
However, when proposing a scientific criticism,
one must take great care to ensure that the critique
itself is scientific and accurate. Perhaps the best
ZD\WR DGGUHVV FRQFHUQV DERXW RWKHUV¶UHVHDUFK LV
WR WDON GLUHFWO\ ZLWK WKH UHVHDUFKHUV LQYROYHG´ S
182). Fontan did this when he organized a series of
meetings during the 2017 International Society of
the Rorschach and Projective Measures
Conference in Paris between R-PAS authors
(Meyer, Mihura, Erdberg) and CSIRA members
(Andronikof, Fontan, Smith). The intention was to
directly discuss the points raised in the Smith et al.
(2018) paper and other concerns with the R-PAS
method. Further objectives were to find ways to
collaborate and to move toward a third system
based on the Exner Comprehensive System (CS;
Exner, 2003) and R-PAS. We were unable to
receive continued support for these discussions.
Consequently, the focus of this article was an
HYDOXDWLRQ RI WKH ILUVW SRLQW ³ZKHQ SURSRVLQJ D
scientific criticism, one must take great care to
ensure that the critique itself is scientific and
DFFXUDWH´S
Meta-analytic research: A current trend in
psychology, is accepting the findings of meta-
analyses without scrutiny of the individual studies-
-a pattern where statistical gloss is used to cover
methodological flaws (Gacono 2019). A second
pattern--conducting additional statistical analyses
that do not address conceptual gaps raised in
critiques--is not unique to Mihura. It contributes
VLJQLILFDQWO\WR WKH ³DUPFKDLU TXDOLW\´RIUHVHDUFK
DQG WKH FUHDWLRQ RI ³SVHXGR-GHEDWHV´ DQG
³DSSDUHQW FRQWURYHUVLHV´ *DFRQR 
Piotrowski, 2017). A better scientific practice
would be increasing the quality of the articles used
in a meta-analysis rather than expecting that the
meta-analysis would correct for poor
methodological quality (Ioannidis, 2016).
Due to our commitment to the ethical and
competent use of the Rorschach, our examination
of the quality of research articles included in meta-
analyses has been an ongoing project. In 2012, we
examined the Wood et al. (2010) meta-analyses
examining psychopathy and the Rorschach
(Cunliffe et al.) and found multiple conceptual and
methodological concerns related to the studies they
used. This led us to evaluate further these issues

Understanding Rorschach Research: 73
using the Mihura et al. (2013) meta-analytic
studies as our primary subject matter.
In the process of our reviews, we became aware of
the degree to which meta-analyses influenced
critical decisions such as acceptance or rejection of
manuscripts submitted for publication and
SURIHVVLRQDOV¶ YLHZV RI SV\FKRORJLFDO WHVWV
Though meta-analyses are widely used in science,
there are many concerns with how it is applied
(Ioannidis, 2016). One of the first critics of meta-
analyses in the field of Psychology was Eysenck
(1978). In his criticisms of the Smith and Glass
(1977) meta-analysis about psychotherapy, he was
perplexed by the low standards of inclusion studies
reported by Smith and Glass. Specifically, he
VWDWHG ³D PDVV RI UHSRUWV ± good, bad, and
indifferent- are fed into a computer in the hope that
people will cease caring about the quality of the
PDWHULDORQ ZKLFK WKH FRQFOXVLRQV DUH EDVHG´ DQG
³WKH QRWLRQ WKDW RQH FDQ GLVWLOl scientific
knowledge from a compilation of studies mostly of
poor design, relying on subjective, unvalidated,
and certainly unreliable clinical judgments, and
dissimilar with respect to nearly all the vital
SDUDPHWHUVGLHVKDUG´S
Eysenck has not been the only one to criticize
meta-analyses (also see Slavin, 1986). Greco,
Zangrillo, Biondi-Zoccai, and Landoi (2013) also
cautioned against meta-analyses especially related
to the quality of research included. They wrote,
³WKH FRQFOXVLRQV RI D PHWD-analysis depend
strongly on the quality of the studies identified to
estimate the pooled effect. The internal validity
may be affected by errors and incorrect
HYDOXDWLRQV´ S  7KH\ DOVR VWDWHG ³LW LV
strongly recommended that reviewers use a set of
specific rules to assign a quality category, aiming
IRU WUDQVSDUHQF\ DQG UHSURGXFLELOLW\´ S 
Schmidt and Hunter (2014) drew special attention
to the importance of making sure that the data on
which a meta-analysis is based were reliable and
valid. FXUWKHUWKH\VWDWHGWKDW³RQHUHTXLUHPHQWRI
an effective epistemology in empirical research is
proper correction for the artifacts that distort the
HPSLULFDOGDWD´S
Applying statistics without putting them into
context is never justified (i.e., Simpson paradox).
It may lead to large effect sizes and statistical
significance, but the results can still be
meaningless. This over-evaluation of statistical
prowess at the expense of conceptual
understanding has resulted in the tail (statistics)
wagging the dog (sound conceptual driven
methodology; Gacono 2019). As noted by Gacono
 ³RQH VWUDWHJ\ IRU GHDOLQJ ZLWK FRQIXVLQJ
findings on Rorschach research is to narrow the
stimulus field; to focus on the Ds and Dds rather
than getting lost in the Ws FUHDWHGE\VWDWLVWLFV´S
3HUKDSV6LJPXQG)UHXGVDLGLW EHVW³7KRVH
critics who limit their studies to methodological
investigations remind me of people who are
always polishing their glasses instead of putting
WKHPRQDQG VHHLQJZLWKWKHP´LQ5Hik, 1956, p.
54).
Points of Clarification related to Mihura (2019)
Methodological bias/internal validity: Mihura
FULWLTXHG RXU XVH RI WKH WHUP ³PHWKRGRORJLFDO
ELDV´DQGVWDWHGWKDWLQWHUQDOYDOLGLW\LVVXHVZHUHD
EHWWHU GHVFULSWLRQ RI WKLV QRWLRQ DQG WKDW ³ELDV´
UHIHUV WR D SHUVRQ¶V judgment and we agree. Two
of our authors (Fontan & Andronikof) are French,
aQGWKH\SURSRVHGWKHWHUP³PHWKRGRORJLFDOELDV´
7R WKH )UHQFK ³ELDV´ UHIHUV PRUH GLUHFWO\ WR
PHWKRGVELDLVPpWKRGRORJLTXHWKDQWRDSHUVRQ¶V
MXGJPHQWV µ,QWHUQDO YDOLGLW\ LVVXHV¶ LV D EHWWHU
description of the concerns we raise in our paper,
and it also helps to substantially explain our
purpose.
0LKXUD  VWDWHG WKDW ³>L@I D VWXG\ GRHV QRW
KDYH LQWHUQDO YDOLGLW\ WKHQ WKH VWXG\¶V UHVXOWV DUH
not meaningful in the first place even to the
sample and setting in which the study was
FRQGXFWHG´ RQOLQH Vupplemental). We concur, as
this point was the very purpose of our paper,
specifically, if the results are real or an artifact of
the methodology. This means a Rorschach study
with poor internal validity (e.g., poor inter-rater
reliability, many Rorschach protocols with R < 14)
would be considered unsound and meaningless and
would not be appropriate for a meta-analysis.
Although she did not agree with our terminology
and suggested a different one, she dismissed our
assertion that internal validity issues in her meta-
analyses were present, despite our findings that

74: Smith et al
91% of the studies in the Mihura et al. (2013)
meta-analyses had poor internal validity. The
definition Mihura provided and what we noted in
Smith et al. (2018) has the same meaning.
([QHU¶V 5b) internal validity criteria:
Mihura (2019) suggested that we should have
examined how the studies included in her meta-
analyses (Mihura et al., 2013) have responded to
([QHU¶V E 5RUVFKDFK UHVHDUFK
recommendations (studies before and after 1995;
are they making the same methodological issues?).
In her commentary (Mihura, 2019), she implied
that studies prior to 1995 had significantly more
internal validity issues than those published after
1995. Yet, these older studies with questionable
internal validity were still included in Mihura et
DO¶V PHWD-analyses, making it necessary to
evaluate their internal validity.
We stated in our paper that Exner (1995a/b) was
not the only source in providing criteria for
evaluating the quality of Rorschach research
(Cunliffe et al., 2012; Gacono et al., 2001).
Specifically, we used the following criteria2 where
there is a large consensus in current Rorschach
research (R, Lambda, & interrater reliability;
Smith et al., 2018, p. 189):
1. IQ/Education level
i. Did the article have statistics related to
IQ/Educational level (M and range)?
ii. If either mean or range was provided,
did the article include both statistics?
iii. If IQ range was reported, did all
participants have an IQ  80?
2. Responses
a. Did the article have statistics related to
R (M and range)?
b. If mean was provided, did the article
include range?
c. If range was provided, did the article
include mean?
d. If range was reported, did all protocols
KDYH5"
3. Lambda/F%
a. Did the article have statistics related to
Lambda/F% (M and range)?
b. If mean was provided, were the means
for Lambda/F% < 0.99/50%?
c. If mean was provided, did the article
report range?
d. If the mean of Lambda/F% > .99/.50,
did the article report IQ/Education
Level? (L/F% > .99/.50 and
IQ/Education level were examined in
combination as this affects
generalizability.)
4. Interrater reliability
a. Did the article have interrater
reliability statistics?
b. If interrater reliability was reported,
ZHUHWKHUHYDOXHVIRU,&&țRU
80% agreement?
5. Sample size
a. Did the article have comparison
JURXSVZLWKSDUWLFLSDQWV"
0LKXUD  VWDWHG WKDW RXU FULWHULD ³ZHUH QRW
VRXQG´SDQGWKDW ([QHU¶VDQGVRPHRI WKH
6PLWKHW DO DXWKRUV¶ SXEOLVKHG VWXGLHVGLG QRWXVH
these criteria. However, in the Mihura table (2019,
p. 172), the data were inaccurate. There were only
seven articles from Exner as Primary Investigator.
Exner, Colligian, Boll, Stischer, and Hillman
(1996) had only one issue, while one had three
issues, five had four issues, and none had five
issues (M = 3.42). The same can be said for the
Smith et al. authors. One had two issues, five had
three issues, three had four issues, and none had
five issues (M = 3.22). The mean issues for Exner,
Smith et al., and R-3$6 DXWKRUV¶ VWXGLHV ZHUH
similar and not statistically significant (Exner = M
= 3.42; Smith et al. = M = 3.22; R-PAS = M =
3.06; F[2,30] = 0.43, p = 0.65; see Table 13 ).
Inter-rater reliability: Mihura (2019) stated that
³:HLQHU ZDVWKH ILUVWWRHVWDEOLVKWKHQHHG
WR UHSRUW 5RUVFKDFK LQWHUUDWHU UHOLDELOLW\´ S -
173). However, only 78.4% of the studies reported
inter-rater reliability after 1991 (12% before 1991;
Mihura, 2019). All studies (pre/post 1991) were
still included in the Mihura et al. (2013) meta-
analyses. Without an inter-rater reliability statistic,
2See https://mfr.osf.io/render?url=https%3A%2F%2Fosf.io%2Fxqhy8%2Fdownload for how we classified all 210 studies
3See online supplement data: https://mfr.osf.io/render?url=https%3A%2F%2Fosf.io%2Fxqhy8%2Fdownload

Understanding Rorschach Research: 75
Table 1. Methodological Issues in Mihura et al. (2013) meta-analyses studies by Exner, Smith et al.
(2018), & R-PAS Authors as Primary Investigators
Methodological Issues
Exner (n = 7)
Smith et al. (n = 9)
Mihura, Meyer, Viglione, & Erdberg (R-
PAS) (n = 17)
0
0 (0%)
0 (0%)
0 (0%)
1
1 (14%)
0 (0%)
1 (6%)
2
0 (0%)
1 (11%)
3 (18%)
3
1 (14%)
5 (56%)
7 (41%)
4
5 (71%)
3 (33%)
6 (35%)
5
0 (0%)
0 (0%)
0 (0%)
M = 3.42
M = 3.22
M = 3.06
Note. M = mean; n = number of studies. For the exact study count see the online supplemental at
https://mfr.osf.io/render?url=https%3A%2F%2Fosf.io%2Fw8ms9%2Fdownloa
it is not possible to determine the reliability and
validity of the study and makes the generalizability
of the findings impossible when counter-intuitive
findings are produced (Gacono, 2019). It should
be noted that many of the research studies
presented at the Society of Personality
Assessment (SPA) FRQIHUHQFHV SULRU WR :LHQHU¶V
 UHFRPPHQGDWLRQ URXWLQHO\ SUHVHQWHG VRPH
IRUPRILQWHUUDWHUDJUHHPHQWDOWKRXJKSHUKDSVQRW
HQRXJKRIWKHP&%*
Sample size: Mihura (2019) stated we should have
used E[QHU¶V VXJJHVWLRQ RI  IRU VDPSOH VL]H
rather than the 20 we recommended. We utilized
'LHV¶UHFRPPHQGDWLRQLQ([QHUD ZKHUH KH
examined studies with 20 or fewer participants
(1995, p. 106). Mihura (2019) in her
reply UHFRPPHQGHG XVLQJ D ³FRPELQHG´ Vample
size of N = 105 for validation purposes (p.
182). While small sample size research is
prone to raise statistical errors, Mihura (2019)
mentioned,
A key strength of meta-analytic procedures is that
they combine small-N studies to produce a more
powerful result. Imagine 10 studies with N = 15 and r
= .40. In an individual study with such a small N, that
correlation is not significant (p = 0.14). However, if
you combine all 10 studies in a meta-analysis, you
now have a sample size of 150, and r = .40 is
significant at p = 0.00000040 (p. 173).
We agree if studies can reasonably be combined
(i.e., that they seem to assess the same construct).
However, this determination was not always
present in Mihura et al. (2013, p. 562) and is
frequently difficult to achieve in Rorschach
research where conceptual issues are often
overlooked. For example, Mihura et al. (2013)
VWDWHG WKDW WKH ³,VRODWLRQ ,QGH[ ZDV QRW
significantly related to its criteria that indicate
Social Isolation, Either the Behavior or the
Psychological Experience (r = .12, S = 4;
$VSHUJHU¶V FKLOG PDOWUHDWPHQW >LQFOXGLQJ 
neglect], and parent ratings of
ZLWKGUDZQGHSUHVVHG FKLOGUHQ´ S  ,Q WKH
four studies on the Isolation Index included in
Mihura et al. (2013) meta-analyses, one focused
on the effectiveness of the CS Depression Index
(DEPI) between 30 depressed (ICD criteria)
individuals and 30 participants from a normative
sample in which the authors proposed a revision to
the DEPI (George & Kumar, 2008). Another
studied the psychological functioning of children
and adolescents with severe burns using the
Rorschach and the Child Behavior Checklist
(Holaday & Blakeney, 1994). Holaday, Moak, and
Shipley (2001) compared 24 boys (children and
adolescents) with Asperger's to 24 boys with
emotional and behavioral disorders (not on the
autism spectrum). Finally, the last study examined
Rorschach indices and MMPI-A scales of 157
children and adolescents to determine the severity
of abuse inflicted on them (Perfect, Tharinger,
Keith, & Lyle-Lahroud, 2011).
Clearly, these studies provided no direct
information on the CS Isolation Index (i.e., social
isolation/sociometric status). The conceptual gaps
in these studies were ignored when applying them
to a meta-analytic study of the Isolation Index.
Simply stated, the lack of evidence for the
Isolation Index speaks more directly to the lack of
relevant research rather than a lack of construct
validity concluded in the Mihura et al. (2013)
meta-analyses.

76: Smith et al
With sample size, idealism and realism need to be
differentiated. Ideally, quality Rorschach research
with a sample size of 15 or 20 could, and at times,
should be published; however, this appears to be
unrealistic with the current academic journal
publishing criteria. For example, we recently
submitted an article to a prominent journal
(Journal of Personality Assessment) that publishes
R-PAS/CS articles. The editor did not send out the
5RUVFKDFKDUWLFOHIRU UHYLHZ EHFDXVH WKH³VDPSOH
size is much too limited to draw confident
FRQFOXVLRQV DERXW WKHVH GDWD´ (Personal
Communication with Editor, January 30, 2019).
This judgment was rendered despite the
comparison groups having more than 30 in each
group with a limited number of statistical
comparisons which made the sample sizes
adequate for the study. Therefore, our criterion
was probably less stringent based on the current
research climate and this would suggest meta-
analyses in the future may include less appropriate
studies and the meta-analyses may not be able to
correct for methodological deficits (also see
Piotrowski, 2017; Thomas, 2019).
Additionally, when reviewing a manuscript for
publication and considering an appropriate sample
size for a given study, one must always consider
the uniqueness of the sample and the number of
variables (relative to the N) that are being
statistically analyzed. For example, in a pilot study
examining 10 subjects where only 25 subjects
exist in the known universe, it would not be in the
interest of science to discard the data because the
sample was less than 20. Certainly, a small sample
size limits the number of statistical comparisons
that can be conducted. In one review years ago,
CBG was told the sample size was too low for a
sample of sexual homicide perpetrators (N = 38;
despite a limited number of comparative analysis),
an opinion totally out of context in the real world
where this had been the largest sample of this type
of subject in the literature at the time. A very
difficult sample to obtain as, fortunately, sexual
KRPLFLGHSHUSHWUDWRUVGRQRWµJURZRQWUHHV¶
Gacono and colleagues (1994) have repeatedly
provided descriptive data for samples studied, but
have been more selective, based on sample size, in
choosing small numbers of variables for statistical
comparison (thereby not increasing the chance of a
statically significant finding that occurs as an
artifact of comparing a plethora of variables).
Small sample sizes can be scientifically useful,
when statistical analyses are limited, when the
conclusions are offered as preliminary, and when
the participants studied were difficult if not
impossible to obtain (e.g., valid protocols of sexual
homicide perpetrators). It seems we have reached a
stage were many reviewers lack an awareness of
the reason for assigning a specific number to
sample size and fail to incorporate these principles
into their decision making, rather they base their
decisions on a set number isolated from any
meaningful context.
Number of Responses: Along with Lambda, the
number of responses (R) is used to assess validity.
A 20-subject sample with a mean R of 15 (which
is below the expectation of normative data; Exner,
2003) would likely not be appropriate for a
validation study related to specific variables,
ratios, or indexes. It may be a valid indicator of the
sample studied. The best use of the data would be
to explain the constriction of R in the specific
sample as related to the personality functioning of
the sample (Gacono & Gacono, 2008). Is this a
sample of low IQ or brain injured subjects? Why is
the R well below the mean for most comparison
samples? Hence, the findings may be valid for the
individual study, but including it in meta-analyses
to assess validity of individual Rorschach
variables, however, would be inappropriate.
Smith et al. (2018) mentioned that 22 studies
included in the Mihura et al. (2013) meta-analyses
had less than 14 responses. Mihura stated this was
because those studies were conducted prior to
Exner (1990) changing the Comprehensive System
administration procedure and should not be
considered an error. The goal of Smith et al.
(2018) was to show the quality of the study
relative to being able to accept the findings²
particularly when those findings were atypical.
The minimum number of responses was increased
WR5WRKHOSLQFUHDVHWKHYDOLGLW\RIDQ
individual protocol. The need to consider R when
HYDOXDWLQJ D VWXG\¶V ILQGLQJV UHPDLQV HVVHQWLDO
regardless of when the study was conducted.
Interestingly, however, older studies with this
methodological issue (R < 14) were still included
in the Mihura et al. meta-analyses.

Understanding Rorschach Research: 77
We have frequently stated that many lower R
protocols are clinically valid (Gacono & Meloy,
1994). We distinguish between clinical usage
which must be evaluated on an individual case by
case basis, and the inclusion of an unspecified
number of low R protocols for analyzing group
data.
Lambda or Form%: Providing descriptive
statistics for Lambda or F% is essential for
determining the appropriateness of a study for
comparative analysis (i.e., generalizability;
Gacono, 2019). Lambda is a straightforward
computation, L = F\R-F. By definition, the more
Pure F in the record relative to R, the higher the
Lambda. Therefore, there are fewer other
determinants (i.e., C, T, Y, V) in a record
dominated by Pure F (Gacono, 2019). High
Lambda (i.e., constricted protocols), most
frequently lack a normative distribution of other
determinants and variables. For example, a
protocol with a Lambda of 1.50 and 15 responses
means that of the 15 possible responses, 9 had pure
form only and only 6 responses had an opportunity
for other determinants. By contrast in a protocol
with 15 responses and a Lambda of 0.50, there are
5 pure form and 10 other responses that may have
determinants. Beyond the math, one must consider
WKH UHDVRQ IRU WKH SURWRFRO¶V FRQVWULFWLRQ KLJK
Lambda) which has an additional generalized
effect of constricting the number of determinants
produced on any response that were not pure form
(Gacono, 2019; Gacono & Gacono, 2008).
Equally important is the inclusion of basic
descriptive data (M, SD, Frequency) for the
Rorschach variables studied. The inclusion of
descriptive data for Lambda and the variables
studied, make it possible to determine if a finding
RI ³QR UHODWLRQVKLS´ LV DQ DUWLIDFW RI WKHVH
confounds or a true finding. For example, are the
lack of findings the result of the protocol
constriction (Lambda) or the fact that there were
not enough of the variable studied. Validation
studies that attempt to assess the relationships
among variables and indices, where elevated
Lambdas exist, are mostly inappropriate (Gacono,
2019). Additionally, presenting this basic
demographic information is essential for
determining the distribution of the variables. As
Viglione (1995) noted many variables form J-
shaped curves not amenable to analysis with
parametric procedures. Like the need to include
descriptive data for Lambda, including mean,
standard deviation, and frequency for variables
studied is essential related to determining validity
as well the generalizability of a study.
The additional statistical analyses that were
conducted by Mihura (2019) did not address this
conceptual problem with Lambda. She interpreted
our findings as the need for controlling Lambda, a
position we have never advocated for. A statistical
analysis by Giromini, Viglione4, and McCullaugh
DFWXDOO\ IRXQG ³DV FRPSDUHG WR WKH >5RUVFKDFK@
international norms5, the CS 600 norms provided
an even better fit for our San Diego sample, when
WKH/DPEGDYDULDEOHZDVWDNHQLQWRFRQVLGHUDWLRQ´
(2014, p. 361)²highlighting the importance of
Lambda when interpreting Rorschach results.
IQ and Education: Other factors, such as
IQ/Education level, may also impact Rorschach
production. While factors that impact Rorschach
production do not necessarily need to be controlled
(e.g., in descriptive studies where the constriction
is an accurate reflection of the personality
functioning being examined; see Gacono &
Gacono, 2008), these variables should always be
considered and reported in validation studies. This
allows the reader to determine the appropriateness
of the study when it is compared to other studies
with more normal distributions²and frequently
explains the presence of counter-intuitive findings
(Gacono, 2019). If IQ is not assessed/available, at
least education should be provided as it is a basic
demographic characteristic (our criterion was IQ
and/or education level). Actually, Meyer,
Giromini, Viglione, Reese, and Mihura (2015)
VWDWHG ZLWK 5RUVFKDFK YDULDEOHV ³WKH RQO\
demographic variable that produced several
VLJQLILFDQW FRUUHODWLRQV LV HGXFDWLRQ´ S ²
highlighting its importance in Rorschach research.
4 A co-author of R-PAS
5 The international norms are used in R-PAS

78: Smith et al
Counterintuitive findings and negative results:
Mihura (2019) contended that our counter-intuitive
conceptualization was problematic. Recently,
discussing counterintuitive findings, Gacono stated
³WKLV ERG\RI SRRUO\ GHVLJQHG UHVHDUFK IUHTXHQWO\
results in contradictory and/or counterintuitive
findings that are best understood by their
PHWKRGRORJLFDO HUURUV UDWKHU WKDQ WUXH ILQGLQJV´
(2019, p. 104). Counterintuitive findings (those
that occur counter to theory and other well-
designed studies) may be true findings or these
may be an artifact of methodological errors
(Gacono, 2019; Gacono et al., 2001). It is essential
for the inclusion of descriptive data (M, SD,
frequencies) for Lambda, R, and the variables
VWXGLHG LQ RUGHU WR DVVHVV D VWXG\¶V LQWHUQDO
validity.
:HGLGQRWLPSO\WKDW³HYHU\>5RUVFKDFK@ ILQGLQJ
>ZDV@WREHVXSSRUWLYH´0LKXUDSRU
it is counterintuitive. Instead, we discussed the
value of negative findings within the studies with
methodological issues. Specifically, we stated,
³>L@Q DOO  VWXGLHV KDG FRXQWHULQWXLWLYH ILQGLQJV
(43.3%). Owing to the methodological issues it is
unclear if these findings are due to Type II error,
the findings are real, an artifact of the
meWKRGRORJ\ D TXLUN RU DQ DW\SLFDO VDPSOH´
(Smith et al., 2018, p. 194). From this, it should be
clear that we consider that some negative findings
might be real, it is just impossible to determine
which ones represent true findings, considering the
internal validity of the study (using our above-
mentioned criteria).
In order to refine our operationalization of
counterintuitive findings, it may be more apt to
state these findings are simply negative results, to
include non-significant results and those
inconsistent with theory in poorly designed
studies. Examining only the validation studies (N
= 101), 32 of the self-report (introspective) studies
and 13 of the externally assessed studies had
counter-intuitive findings.
Application and validation studies: Mihura
(2019) stated that our distinction between
validation and application studies is not significant
DQG WKH UHDVRQ ³ZK\ UHVHDUFKHUV DUH FRPSDULQJ
JURXSV GRHV QRW PDWWHU´ S  )XUWKHU VKH
stated,
,I WZR VWXGLHV¶ PHWKRGRORJLHV DUH WKH VDPH LW
does not matter why the researchers state they are
comparing the groups. For instance, one
researcher might test the validity of X-% by
comparing patients with schizophrenia to those
with schizotypal personality. Another researcher
might use X-% to see if perceptual accuracy
differs between these patient groups. The first is a
validation study; the second is an application
study. However, each study might use the same
methodology and study design, and both speak
equally to the validity of X-% (p. 178).
That research objectives and hypotheses do not
PDWWHU QHHGV WR EH DGGUHVVHG ³LW GRHV QRW PDWWHU
why the researchers state they are comparing the
JURXSV´S
Concerning the above example between
schizophrenics and schizotypals, such a study
would not be very appropriate if a researcher was
to assess the validity of X-%. Indeed, psychiatric
groups here are a proxy, or an indirect measure of
the construct which is assessed (some
schizophrenics may be stable; some schizotypals
may be in crisis; some might be misdiagnosed).
Instead a researcher trying to assess the validity of
X-%, should seek a direct measure of reality
testing failures, such as form perception tests (e.g.,
Benton Face Recognition; Hooper Visual
Organization Test). Therefore, research objectives
matter, as there should be a close correspondence
between research design and its objectives. This is
an important issue. In an application study, a
Rorschach score relies on the assumption that this
score is valid. If this study is considered, without
hesitation, as equivalent to a validation study, it
leads to a tautological problem (i.e., proving
something, which is assumed to be true, or more
simply proving that truth is true). Therefore,
logically, a Rorschach score is either valid or the
Rorschach score is not valid. Indeed, in the
application study, the score is assumed to be valid.
In a validation study, the objective is to assess the
validity of a score. These are scientifically quite
different.
However, and in contradiction with her reply to
our paper, Mihura et al. (2013) stated elsewhere,

Table 2. Self-Report/Externally Assessed Classifications noted in the Mihura et al. (2013) Studies
Type of Study
Self-Report/External
Application/Validation
E
E/SR
SR
Total
Application
91
15
106
Validation
50
3
51
104
Total
141
3
66
210
Note. E = externally assessed criteria study; SR = self-report study.
Table 3. Methodological Issues with the Different Types of Studies cited in Mihura et al. (2013)
External
Total E
Self-Report
Total SR
Total
# issues
A
V
A
V
1
2
1
3
1
1
4
2
4
6
10
2
4
6
16
3
38
22
60
7
16
23
83
4
36
18
54
6
27
33
87
5
11
3
14
3
3
17
Total
91
50
141
15
51
66
207
Note. E = externally assessed criteria study; SR = self-report study; A = application; V = validation. Three studies were excluded that
had both Self-Report/Externally assessed criteria to simplify results.
there are two types of studies that contain potentially
relevant validity coefficients. The first type is
specifically designed to validate a psychological test
variable, and in this type of study an external
criterion measure is purposefully chosen to match
WKH WHVW YDULDEOH¶V FRQVWUXFW 7KH VHFRQG W\SH LV
designed to understand a condition and uses the
psychological test variable as the external validity
criterion. Often, the goal of this type of study is to
describe psychological characteristics that differ
between groups. For example, the research question
PLJKW EH ³+RZ GR SDWLHQWV ZLWK ERUGHUOLQH
personality differ from patients with schizotypal
SHUVRQDOLW\"´RU ³+RZ GR VH[RIIHQGHUVGLIIHUIURP
RWKHU W\SHV RI RIIHQGHUV"´ ,Q WKHVH FDVHV D
multiscale test like the Rorschach is used as an
assessment tool to understand the target conditions
with the assumption that its scores provide valid
measures of many different psychological
constructs. This second type of study can contain a
substantial number of coefficients that are not
intended to be a probative evaluation of the test
YDULDEOH¶V FRQVWUXFW YDOLGLW\ DV ZHOO DV VRPH
coefficients that may qualify as core validation
criteria (p. 558).
Results presented in Tables 2 and 3 show that
most validity coefficients, based on externally
assessed criterion in Mihura HW DO¶V PHWD-
analyses, were based on application studies
(~65%), which is in contrast with the position
related to their last sentence in the above quote.
Assessing internal validity: Mihura (2019)
VWDWHG ³, DSSOLHG 6PLWK HW DO¶V   WR 
study methodological ratings to the 770 meta-
analytic findings in Mihura et al. (2013). A
higher score indicates that Smith et al. thought
the study had more methodological problems.
The relationship between their ratings and the
YDOLGLW\HIIHFWVL]HVZDVU ´ (p. 179). Based
on this result, Mihura concluded that the criteria
ZH XVHG ³ZHUH QRW VRXQG´ S  GXH WR WKH
low validity effect size. She suggested that the
criteria we chose did not affect the meta-analyses
and the studies should not have been removed
because of them. However, these analyses did
not address our contentions (Smith et al., 2018).
Mihura (2019) also insisted on the point that the
criteria we used referred to internal validity and
that her meta-analyses assessed the Rorschach
YDULDEOHV¶ Hxternal validity. Basically, Mihura
managed to demonstrate that internal and external
validity are not related, or more simply that
methods and results are two different things. The
very idea that internal validity can be assessed by
external validity, or that the quality of the methods
is related to the effect sizes of the results is very
Understanding Rorschach Research: 79

80: Smith et al.
perplexing and speaks to the conceptual gaps
mentioned above. Reliability and validity are two
different concepts. We never expected studies with
better methodology to present higher effect sizes,
but simply to have higher reliability coefficients
and be conceptually sound.
Externally assessed validity criterion: Mihura
(2019) reported that not all the Mihura et al.
(2013) meta-analyses articles/studies were used to
test the validity of the Rorschach scores.
6SHFLILFDOO\ VKH VWDWHG ³DVVRFLDWLRQV EHWZHHQ WKH
Rorschach and self-report measures were not used
to detHUPLQH WKH YDOLGLW\ RI 5RUVFKDFK YDULDEOHV´
EHFDXVH ³5RUVFKDFK VWXGLHV XVLQJ VHOI-report
measures for validity criteria are typically
SXEOLVKHG LQ OHVV ULJRURXV MRXUQDOV´ LH MRXUQDO
impact factor, p. 171). She disagreed with our
inclusion of those studLHVUHSRUWHGLQ6PLWKHWDO¶V
(2018) analyses. This means that the Mihura et al.
(2013) validity results were derived from a smaller
set of the 210 articles.
Consequently, we examined each of the articles
included in our analyses to determine if these
investigations were a self-report study and/or an
externally assessed study (see Tables 2 & 3) using
the criteria from Mihura et al. (2013) with most of
the studies being externally assessed (67%).
However, 91 (65%) of the articles that were
considered externally assessed criteria articles
were application studies while only 35% (50) were
validation studies with externally assessed criteria.
Concerning external criteria studies, 91% of the
studies presented with three or more internal
validity issues (86% for validation studies; 93%
for application studies). Therefore, the internal
validity issues were still present even when only
examining the externally assessed studies.
Conclusion:
The Mihura et al. (2013) meta-analyses and
subsequent commentary (Mihura, 2019) provided
XVHIXO ³VXEMHFWV´ IRU VWXG\LQJ FXUUHQW WUHQGV LQ
Rorschach research. It is essential to understand
the issues, in the context of our article (Smith et
al., 2018), as these findings, discussions, and
conclusions are utilized to accept or deny
manuscripts for publication, and to promote the R-
PAS. Reviewers and editors must understand that
the R-PAS is still early in its development, has
unaddressed problems and issues with its validity,
the normative data, and variables used (Fontan &
Smith, 2018). For manuscript reviewers, a more
in-depth understanding of both conceptual and
statistical issues is needed to ensure sound
scholarship is not spuriously rejected and inferior
research published. Understanding the conceptual
basis of the Rorschach and its variables is essential
for providing a fair review (Gacono, 2019). We
encourage reviewers to not become enamored with
the statistical gloss at the expense of verifiable
conceptual foundations (Gacono 2019).
7KH LVVXHV ZH UDLVH DUH QRW D µFRQWURYHUV\¶ but
rather based on a scientific approach to the data.
We strongly recommend that R-PAS not be used
in any context, clinical or forensic, without further
validation of its utility and validity, based on R-
PAS administered protocols and validation studies
with R-PAS variables. While the CS has an
extensive history of clinical and forensic usage
meeting the standards of admissibility and being
accepted in the courtroom (Gacono & Evans,
2008; Gacono, Evans, & Viglione, 2008; McCann
& Evans, 2008; Meloy, 2008), R-PAS does not
(extensive applied usage, meeting various critical
standards) and, in our view, the rush to use it will
ultimately be to the detriment of the Rorschach. In
research terms, it is premature to consider data
generated by R-PAS as appropriate for application
studies, but rather needs to be considered only in
validation designs.
We encourage and welcome scientific debate
where the goal is to discover the truth and to
improve the psychometric credibility of research
ILQGLQJV 0LKXUD¶V UHVSRQVHV in her commentary
were like previous responses to the criticisms of
her meta-analyses (see Tibon Czopp & Zeligman,
2016). A pattern in which issues raised are
responded to by conducting additional statistical
analyses that fail to address these concerns.
However, the commentary was helpful to us in
highlighting areas in need of further clarification.
As Mihura noted in her commentary (2019, p.
179), in the spirit of ethical guidelines and
scientific cooperation, we provided her with our
data for review and analysis6 ; however, she
refused (Personal communication, March 17,
2019) to provide her meta-analytic data to us

despite multiple requests, stating the data were no
ORQJHU DYDLODEOH LQ ³WKHLU RULJLQDO IRUPDW´ LW ZDV
more than five years after publication, and that the
data we requested would be explained in her reply
(it was not). This last point is of significant
concern--that data used in a significant way to
support and promote R-PAS is no longer available
LQ³WKHLURULJLQDOIRUPDW´.
6To continue our openness, we have released online all our
decisions for Smith et al. (2018) and this article at https://
mfr.osf.io/render?url=https%3A%2F%2Fosf.io%2Fxqhy
8%2Fdownload
References:
Andronikof, A. (2019). What future for the
Comprehensive System? Paper presented at the
3rd Congress of the Comprehensive System
International Rorschach Association
(CSIRA/ARISI). https://www.researchgate.net/
publication/3377414 39
Cunliffe, T.B., Gacono, C.B., Meloy, J.R., Smith, J.M.,
Taylor, E.E., & Landry, D. (2012). Psychopathy
and the Rorschach: A response to Wood et al.
(2010). Archives of Assessment Psychology, 2(1),
1-31.
Dies, R. (1995). Conceptual issues in Rorschach
research. In J. E. Exner (Ed.), Issues and methods
in Rorschach research (pp. 25-51). Hillsdale, NJ:
Lawrence Erlbaum Associates, Inc.
Exner, J. E. (1990). A Rorschach workbook for the
Comprehensive System. (3rd ed.). Asheville, NC:
Rorschach Workshops.
Exner, J. E. (Ed.). (1995a). Issues and methods in
Rorschach research. Mahwah, NJ: Erlbaum.
Exner, J. E. (1995b). Introduction. In J. E. Exner (Ed.),
Issues and methods in Rorschach research (pp.
1±24). Mahwah, NJ: Erlbaum.
Exner, J. E. (2003). The Rorschach: A comprehensive
system (4th ed.). Hoboken, NJ: Wiley.
Exner, J. E., Colligan, S. C., Boll, T. J., Stischer, B., &
Hillman, L. (1996). Rorschach findings
concerning closed head injury patients.
Assessment, 3, 317±326.
Eysenck, H. J. (1978). An exercise in mega-silliness.
American Psychologist, 33(5), 517.
Fontan, P. (2019). The Future of the Rorschach test.
Journal of Japan Rorschach Society for the
Comprehensive System, 23(1), 2-8.
Fontan, P., & Smith, J.M. (2018). A Critical Review of
R-PAS, Symposium conducted at the Society for
Personality Assessment at the 2018 Society of
Personality Assessment Annual Convention,
Washington DC, USA. https://www.research
gate.net/publication/328743530
Gacono, C.B. (2019). The importance of Lambda to the
generalizability of Rorschach findings reported in
the literature. SIS Journal of Projective
Psychology & Mental Health, 26, 104-106.
Gacono, C. B., & Evans, F. B. (Eds.). (2008). The LEA
series in personality and clinical psychology. The
handbook of forensic Rorschach assessment (N.
Kaser-Boyd & L. A. Gacono, Collaborators).
Routledge/Taylor & Francis Group.
Gacono, C. B., Evans, F. B., & Viglione, D. J. (2008).
Essential issues in the forensic use of the
Rorschach. In C. B. Gacono, F. B. Evans (Eds.) &
N. Kaser-Boyd, L. A. Gacono (Collaborators),
The LEA series in personality and clinical
psychology. The handbook of forensic Rorschach
assessment (p. 3±20). Routledge/Taylor & Francis
Group.
Gacono, L.A., & Gacono, C.B. (2008). Some
considerations for the Rorschach assessment of
forensic psychiatric outpatients. In C.B. Gacono,
F.B. Evans, N. Kaser-Boyd, & L.A. Gacono
(Eds.), The handbook of forensic Rorschach
assessment (pp. 421-444). New York: Lawrence
Erlbaum Associates.
Gacono, C. B., Loving, J. L., & Bodholdt, R. H. (2001).
The Rorschach and psychopathy: Toward a more
accurate understanding of the research findings.
Journal of Personality Assessment, 77, 16-38.
Gacono, C.B., & Meloy, J.R. (1994). The Rorschach
assessment of aggressive and psychopathic
personalities. Hillsdale, NJ: Erlbaum.
George, L., & Kumar, R. (2008). Diagnostic efficiency
of new Rorschach Depression Index (DEPI). SIS
Journal of Projective Psychology & Mental
Health, 15, 118±127.
Giromini, L., Viglione, D. J., & McCullaugh, J. (2015).
Introducing a Bayesian approach to determining
degree of fit with existing Rorschach norms.
Journal of Personality Assessment, 97, 354±363.
doi:10.1080/00223891.2014.959127.
Greco, T., Zangrillo. A.,Biondi-Zoccai,G., & Landoni,
G., (2013)Meta-analysis: pitfalls and hints. 
Heart Lung Vessel5(4)219±225.
Holaday, M., & Blakeney, P. (1994). A comparison of
psychologic functioning in children and
adolescents with severe burns on the Rorschach
and the Child Behavior Checklist. Journal of Burn
Care & Rehabilitation, 15, 412±415.
Holaday, M., Moak, J., & Shipley, M. A. (2001).
Rorschach protocols from children and
adolescents wiWK $VSHUJHU¶V GLVRUGHU -RXUQDO RI
Personality Assessment, 76, 482±495.
Understanding Rorschach Research: 81

82: Smith et al.
Ioannidis, J. P. A. (2016). The mass production of
redundant, misleading, and conflicted systematic
reviews and meta-analyses. Milbank Quarterly,
94(3), 485±514.
McCann, J. T., & Evans, F. B. (2008). Admissibility of
the Rorschach. In C. B. Gacono, F. B. Evans
(Eds.) & N. Kaser-Boyd, L. A. Gacono
(Collaborators), The LEA series in personality and
clinical psychology. The handbook of forensic
Rorschach assessment (p. 55±78).
Routledge/Taylor & Francis Group.
Meloy, J. R. (2008). The authority of the Rorschach: An
update. In C. B. Gacono, F. B. Evans (Eds.) & N.
Kaser-Boyd, L. A. Gacono (Collaborators), The
LEA series in personality and clinical psychology.
The handbook of forensic Rorschach assessment
(p. 79-88). Routledge/Taylor & Francis Group.
Meyer, G. J., Giromini, L., Viglione, D. J., Reese, J. B.,
& Mihura, J. L. (2015). The association of gender,
ethnicity, age, and education with Rorschach
scores. Assessment, 22, 46±64.
Meyer, G. J., Viglione, D. J., Mihura, J. L., Erard, R. E.,
& Erdberg, P. (2011). Rorschach Performance
Assessment System: Administration, coding,
interpretation, and technical manual. Toledo, OH:
Rorschach Performance Assessment System.
Mihura, J. L. (2019). CorrHFWLQJ 6PLWK HW DO¶V 
criticisms of all Rorschach studies in Mihura,
Meyer, Dumitrascu, and Bombel (2013).
Rorschachiana, 40(2), 169-186.
Mihura, J. L., Meyer, G. J., Dumitrascu, N., & Bombel,
G. (2013). The validity of individual Rorschach
variables: Systematic reviews and meta-analyses
of the comprehensive system. Psychological
Bulletin, 139, 548±605. doi:10.1037/a0029406
Perfect, M. M., Tharinger, D. J., Keith, T. Z., & Lyle-
Lahroud, T. (2011). Relations between Minnesota
Multiphasic Personality Inventory±A scales and
Rorschach variables with the scope and severity of
maltreatment among adolescents. Journal of
Personality Assessment, 93, 582±591.
Piotrowski, C. (2017). Rorschach research through the
lens of bibliometric analysis: Mapping
investigatory domain. Journal of Projective
Psychology & Mental Health, 24(1), 34-38.
Reik, T. (1956). The search within; the inner
experiences of a psychoanalyst. New York, NY:
Farrar, Strauss and Cudahy.
Schmidt, F. L., & Hunter, J. E. (2014). Methods of
meta-analysis: Correcting error and bias in
research findings (3rd ed.). Thousand Oaks, CA:
Sage.
Slavin, R. E. (1986). Best-evidence synthesis: An
alternative to meta-analytic and traditional
reviews. Educational Researcher, 15(9), 5±11.
https://doi.org/10.3102/0013189X015009005
Smith, J. M., Gacono, C. B., Fontan, P., Taylor, E. E.,
Cunliffe, T. B., & Andronikof, A. (2018). A
scientific critique of Rorschach research:
5HYLVLWLQJ ([QHU¶V ,VVXHV DQG 0HWKRGV LQ
Rorschach Research (1995). Rorschachiana, 39,
180±203. doi:10.1027/1192-5604/a000102
Smith, M. L., & Glass, G. V. (1977). Meta-analysis of
psychotherapy outcome studies. American
Psychologist, 32,752-760.
Thomas, E. (2019). Peer review is broken. https://multi-
biome.com/science/2019/12/27/peer-review-is-
broken
Tibon Czopp, S., & Zeligman, R. (2016). The Rorschach
Comprehensive System (CS) psychometric
validity of individual variables. Journal of
Personality Assessment, 98, 335±342.
Viglione, D. J. (1995). Basic considerations regarding
data analysis. In J. E. Exner (Ed.), Issues and
methods in Rorschach research (pp. 195-226).
Hillsdale, NJ, US: Lawrence Erlbaum Associates,
Inc.
:HLQHU , %  (GLWRU¶V QRWH ,QWHUVFRUHU
agreement in Rorschach research. Journal of
Personality Assessment, 56, 1.
Wood, J. M., Lilienfeld, S. O., Nezworski, M. T., Garb,
H. N., Holloway Allen, K. & Wildermuth, J. L.
(2010). Validity of Rorschach inkblot scores for
discriminating psychopaths from nonpsychopaths
in forensic populations: A meta-analysis.
Psychological Assessment, 22(2), 336-349.