Replicability, Confidence, and Priors

Arizona State University, Tempe, AZ 85287-1104, USA.
Psychological Science (Impact Factor: 4.43). 01/2006; 16(12):1009-12. DOI: 10.1111/j.1467-9280.2005.01653.x
Source: PubMed
Download full-text


Available from: Peter R Killeen
  • Source
    • "Most importantly, in the long run, this would involve a shift toward a Bayesian view of probability (Trafimow, 2003). Such a change in graduate training would direct attention to the explicit acknowledgment of researchers' subjective beliefs and prior probabilities in theory validation (Cumming, 2005; Killeen, 2005b; Krueger, 2001). In contrast to the traditional view, which brackets researcher beliefs and, thus, leads to a false sense of objectivity (Gatsonis et al., 2001; Kline, 2004), Bayesian estimation explicitly specifies researchers' expectations or degrees of belief (Chater, Tenenbaum, & Yuille, 2006). "
    [Show abstract] [Hide abstract]
    ABSTRACT: The purpose of this paper is to propose possible solutions to the methodological problem of null hypothesis significance testing (NHST), which is framed as deeply embedded in the institutional structure of the social and organizational sciences. The core argument is that, for the deinstitutionalization of statistical significance tests, minor methodological changes within an unreformed epistemology will be as unhelpful as emotive exaggerations of the ill effects of NHST. Instead, several institutional-epistemological reforms affecting cultural-cognitive, normative, and regulative processes and structures in the social sciences are necessary and proposed in this paper. In the conclusion, the suggested research reforms, ranging from greater emphasis on inductive/abductive reasoning to statistical modeling and Bayesian epistemology, are classified according to their practical importance and the time horizon expected for their implementation. Individual-level change in researchers' use of NHST is unlikely if it is not facilitated by these broader epistemological changes.
    Full-text · Article · Dec 2011 · Organizational Research Methods
  • Source
    • "1006). In response, Killeen (2005b) argued that p rep and the p value, " although informationally equivalent , are distinguished by the inferences they warrant; p rep is a valid posterior predictive probability, p is not " (p. 1011). "
    [Show abstract] [Hide abstract]
    ABSTRACT: The purpose of the recently proposed prep statistic is to estimate the probability of concurrence, that is, the probability that a replicate experiment yields an effect of the same sign (Killeen, 2005a). The influential journal Psychological Science endorses prep and recommends its use over that of traditional methods. Here we show that prep overestimates the probability of concurrence. This is because prep was derived under the assumption that all effect sizes in the population are equally likely a priori. In many situations, however, it is advisable also to entertain a null hypothesis of no or approximately no effect. We show how the posterior probability of the null hypothesis is sensitive to a priori considerations and to the evidence provided by the data; and the higher the posterior probability of the null hypothesis, the smaller the probability of concurrence. When the null hypothesis and the alternative hypothesis are equally likely a priori, prep may overestimate the probability of concurrence by 30% and more. We conclude that prep provides an upper bound on the probability of concurrence, a bound that brings with it the danger of having researchers believe that their experimental effects are much more reliable than they actually are.
    Full-text · Article · Jun 2010 · Psychological Methods
  • Source
    • "probability (p rep ) shows the predictive probability of finding an effect of the same sign in a replica of experience (Killeen, 2005). The convergence of different indicators is to be observed, since the significant correlations reflect effects at least intermediate whose replication probability equals or exceeds 90%. "
    [Show abstract] [Hide abstract]
    ABSTRACT: In this study we are interested in the predictive incremental validity of a Q-sort on self image at work; the items are derived from the five factor model (FFM) used in addition to a test of general mental ability, Domino70. The grades obtained by a sample of 120 students were the criterion. The results confirm that the dimension of Conscientiousness shows good predictive position, as well as, more specifically the dimension of Agreeableness. We also show that the personological trait does not intervene directly on the level of academic success. It is mediated by a causal chain extending from the normative proximity or distance of self image to a prototype model created by professors, while the second mediating variable is the coherence of choices as compared to the FFM dimensions. KEYWORDS: predictive validity, personality questionnaire, five factor model,
    Full-text · Article · Jan 2009
Show more