John Ruscio’s research while affiliated with College of New Jersey and other places

When new data are collected to check the findings of an original study, it can be challenging to evaluate replication results. The small-telescopes method is designed to assess not only whether the effect observed in the replication study is statistically significant but also whether this effect is large enough to have been detected in the original study. Unless both criteria are met, the replication either fails to support the original findings or the results are mixed. When implemented in the conventional manner, this small-telescopes method can be impractical or impossible to conduct, and doing so often requires parametric assumptions that may not be satisfied. We present an empirical approach that can be used for a variety of study designs and data-analytic techniques. The empirical approach to the small-telescopes method is intended to extend its reach as a tool for addressing the replication crisis by evaluating findings in psychological science and beyond. In the present tutorial, we demonstrate this approach using a Shiny app and R code and included an analysis of most studies (95%) replicated as part of the Open Science Collaboration’s Reproducibility Project in Psychology. In addition to its versatility, simulations demonstrate the accuracy and precision of the empirical approach to implementing small-telescopes analysis.

Developing psychological assessment instruments often involves exploratory factor analyses, during which one must determine the number of factors to retain. Several factor-retention criteria have emerged that can infer this number from empirical data. Most recently, simulation-based procedures like the comparison data approach have shown the most accurate estimation of dimensionality. The factor forest, an approach combining extensive data simulation and machine learning modeling, showed even higher accuracy across various common data conditions. Because this approach is very computationally costly, we combine the factor forest and the comparison data approach to present the comparison data forest. In an evaluation study, we compared this new method with the common comparison data approach and identified optimal parameter settings for both methods given various data conditions. The new comparison data forest approach achieved slightly higher overall accuracy, though there were some important differences under certain data conditions. The CD approach tended to underfactor and the CDF tended to overfactor, and their results were also complementary in that for the 81.7% of instances when they identified the same number of factors, these results were correct 96.6% of the time.

A central question in psychological science concerns whether psychological constructs are best conceptualized as dimensional or consist of one or more categories. The present study uses contemporary taxometric procedures to examine the latent structure of suicidal thoughts, with implications for how suicidal thoughts and behavior (STB) ought to be conceptualized, assessed, measured, and managed. Three nonredundant taxometric procedures (MAMBAC, MAXEIG, and L-Mode) were performed on various sets of indicators, and analyses were replicated across two large samples that included large numbers of individuals reporting current and recent STB. Results provide further evidence that the latent structure of suicidal thoughts is best understood as dimensional. However, inconsistent findings across studies and the relatively small number of taxometric studies conducted to date both suggest that it is premature to draw clear or definitive conclusions about the latent structure of STB being dimensional or categorical based on taxometric evidence. We report a meta-analysis of the current literature which evidences this ambiguity. We provide a detailed, critical discussion of the STB taxometric literature and outline key directions for future taxometric studies in this area, particularly how taxometric analysis relates to testing ‘ideation to action’ theoretical models, which hypothesize that the development of suicidal ideation and the progression from suicide desire to attempting suicide are distinct processes with distinct explanations/mechanisms. It remains entirely possible that qualitatively distinct types of STB (e.g., representing ideation vs. action) or populations have different latent structures indicating different levels of risk.

Fernbach et al. (2013) found that political extremism and partisan in-group favoritism can be reduced by asking people to provide mechanistic explanations for complex policies, thus making their lack of procedural-policy knowledge salient. Given the practical importance of these findings, we conducted two preregistered close replications of Fernbach et al.’s Experiment 2 (Replication 1a: N = 306; Replication 1b: N = 405) and preregistered close and conceptual replications of Fernbach et al.’s Experiment 3 (Replication 2: N = 343). None of the key effects were statistically significant, and only one survived a small-telescopes analysis. Although participants reported less policy understanding after providing mechanistic policy explanations, policy-position extremity and in-group favoritism were unaffected. That said, well-established findings that providing justifications for prior beliefs strengthens those beliefs, and well-established findings of in-group favoritism, were replicated. These findings suggest that providing mechanistic explanations increases people’s recognition of their ignorance but is unlikely to increase their political moderation, at least under these conditions.

A central question in psychological science concerns whether psychological constructs are best conceptualized as dimensional or consist of one or more categories. The present study uses contemporary taxometric procedures to examine the latent structure of suicidal thoughts, with implications for how suicidal thoughts and behavior (STB) ought to be conceptualized, assessed, measured, and managed. Three nonredundant taxometric procedures (MAMBAC, MAXEIG, and L-Mode) were performed on various sets of indicators, and analyses were replicated across two large samples that included large numbers of individuals reporting current and recent STB. Results provide further evidence that the latent structure of suicidal thoughts is best understood as dimensional. However, inconsistent findings across studies and the relatively small number of taxometric studies conducted to date both suggest that it is premature to draw clear or definitive conclusions about the latent structure of STB being dimensional or categorical based on taxometric evidence. We report a meta-analysis of the current literature which evidences this ambiguity. We provide a detailed, critical discussion of the STB taxometric literature and outline key directions for future taxometric studies in this area, particularly how taxometric analysis relates to testing "ideation to action" theoretical models, which hypothesize that the development of suicidal ideation and the progression from suicide desire to attempting suicide are distinct processes with distinct explanations/mechanisms. It remains entirely possible that qualitatively distinct types of STB (e.g., representing ideation vs. action) or populations have different latent structures indicating different levels of risk. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Taxometric analysis is often used in clinical psychological research to determine whether a construct of interest is categorical or dimensional in nature. This article reviews the method and provides empirical guidelines for performing and interpreting results of taxometric analysis. Doing so can be quite subjective, and we describe recent advances for reducing this subjectivity. We describe a software package (RTaxometrics) for taxometric analysis and demonstrate its use with illustrative categorical and dimensional data sets. These analyses show how to determine whether data sets are appropriate for taxometric analysis, how to perform various taxometric procedures, and how to interpret the results.

Research training in psychological science emphasizes common threats to internal validity, with no comparably systematic or rigorous treatment of external validity. Trade-offs between internal and external validity are well known in some areas (e.g., efficacy vs. effectiveness studies in clinical psychology), less so in others (e.g., forensic research on eyewitness identification, false memories, or confessions). We present a framework for examining external validity grounded in four domains—populations, settings, outcomes, and timeframes—that can be used to enhance the generalizability of findings. We discuss this framework and then illustrate its use by reviewing mindless eating interventions intended to help people lose weight. Research in this published literature seldom samples from appropriate populations (e.g., overweight or obese individuals) or measures appropriate outcomes (e.g., weight change) in appropriate settings (e.g., the home) over appropriate timeframes (e.g., sustained interventions with follow-up) to determine whether practical advice is empirically supported. In their applied work, we encourage psychological scientists to design studies, analyze data, and report findings with greater attention to external validity to demonstrate, rather than assume, the generalizability of findings to the intended populations, settings, outcomes, and timeframes. Editors and reviewers can hold investigators accountable for doing so.

In three studies, Wilkins and Kaiser found that both chronic and experimental salience of racial progress in the United States increased the perceptions of anti-White bias only among people high in status-legitimizing beliefs (SLBs). We conducted four preregistered high-powered replications of this research. Studies 1, 2, and 3a were close replications of studies 1–3, respectively. Study 3b was a close replication that included an additional experimental condition. Contrary to the original findings, none of the four expected interaction effects tested were statistically significant in the predicted direction, and only one of the four survived a “small telescopes” analysis. We provide additional tests addressing whether changing social contexts explain our failures to replicate, with mixed conclusions. Whereas it is possible that changing social contexts may have eliminated a once true effect, it is also possible that the original results were false positives.

Findings that money priming increases socioeconomic system support have proven difficult to replicate. Schuler and Wänke found that subjective socioeconomic status (SES) moderates money priming effects on system justification and belief in a just world. We conducted three preregistered replications of this research, with sample sizes 3 times those of the original studies. Replication 1 was a conceptual replication that combined elements from the original two studies, and Replications 2 and 3 were close replications of Studies 1 and 2, respectively. None of the four subjective SES × Money Prime interaction effects tested were statistically significant, and only one of the four survived a “small telescopes” analysis. We discuss reasons for our general failure to replicate the original findings and implications for money priming effects.