Correcting Smith et al.’s (2018) Criticisms of All Rorschach Studies in Mihura, Meyer, Dumitrascu, and Bombel’s (2013) Meta-Analyses

Abstract

Smith et al. (2018) describe their article as “an evaluation as to the extent that individual studies have conformed to [ Exner’s (1995a) ] proposed methodological criteria” (Abstract). However, the authors did not conduct analyses to compare research before and after Exner (1995a) in order to assess its impact nor were the set of criteria they used Exner’s. Instead, they critiqued the individual studies in Mihura and colleagues’ (2013) meta-analyses, declaring all methodologically unsound (including Exner’s). They conjectured that Mihura et al. omitted studies with less “methodological bias” that would have provided more support for Rorschach validity. I explain why most of the criteria they use to criticize the studies’ methodology are not sound. But to directly test their hypotheses, I requested their ratings of study methodology. Findings from studies they rated as having more methodological “issues” (e.g., not reporting IQ or Lambda range) or as being “application studies” – which they said should be excluded – were not less supportive of Rorschach validity as they assumed would be the case. The small effect size associations ( r < |.10|) were also in the opposite direction of which Smith et al. argued to be true, indicating that the criteria by which they evaluated other researchers’ studies were not sound. Our findings do indicate that researchers are responding to the one criterion that is clearly stated in Exner (1995a) , which is Weiner’s (1991) recommendation to report interrater reliability; before 1991, 12% of studies reported interrater reliability, which afterward jumped to 78.4%. Other claims in the article by Smith et al. are also addressed.

Full text

Commentary
Correcting Smith et al.’s(2018)
Criticisms of All Rorschach Studies
in Mihura, Meyer, Dumitrascu, and
Bombel’s(2013) Meta-Analyses
Joni L. Mihura
Department of Psychology, University of Toledo, OH, USA
Abstract: Smith et al. (2018) describe their article as “an evaluation as to the extent that
individual studies have conformed to [Exner’s (1995a)] proposed methodological criteria”
(Abstract). However, the authors did not conduct analyses to compare research before and
after Exner (1995a) in order to assess its impact nor were the set of criteria they used Exner’s.
Instead, they critiqued the individual studies in Mihura and colleagues’(2013) meta-analyses,
declaring all methodologically unsound (including Exner’s). They conjectured that Mihura
et al. omitted studies with less “methodological bias”that would have provided more support
for Rorschach validity. I explain why most of the criteria they use to criticize the studies’
methodology are not sound. But to directly test their hypotheses, I requested their ratings of
study methodology. Findings from studies they rated as having more methodological “issues”
(e.g., not reporting IQ or Lambda range) or as being “application studies”–which they said
should be excluded –were not less supportive of Rorschach validity as they assumed would
be the case. The small effect size associations (r< |.10|) were also in the opposite direction of
which Smith et al. argued to be true, indicating that the criteria by which they evaluated other
researchers’studies were not sound. Our findings do indicate that researchers are
responding to the one criterion that is clearly stated in Exner (1995a), which is Weiner’s
(1991) recommendation to report interrater reliability; before 1991, 12% of studies reported
interrater reliability, which afterward jumped to 78.4%. Other claims in the article by Smith
et al. are also addressed.
Keywords: Rorschach, meta-analysis, methodology
Smith et al. (2018;inRorschachiana) published a critique of the 215 studies included
in the 210 articles in Mihura, Meyer, Dumitrascu, and Bombel’s(2013) meta-
analyses of 65 Rorschach variables –meta-analyses that had resulted in the critics
lifting their call for an all-out moratorium on the use of the Rorschach (Wood, Garb,
Nezworski, Lilienfeld, & Duke, 2015; see also Mihura, Meyer, Bombel, &
Dumitrascu, 2015)–and declared that the work of hundreds of Rorschach
©2019 Hogrefe Publishing Rorschachiana (2019), 40(2), 169–186
https://doi.org/10.1027/1192-5604/a000118
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.