Sample size planning for the standardized mean difference: Accuracy in parameter estimation via narrow confidence intervals

Indiana University Bloomington, Bloomington, Indiana, United States
Psychological Methods (Impact Factor: 4.45). 01/2007; 11(4):363-85. DOI: 10.1037/1082-989X.11.4.363
Source: PubMed


Methods for planning sample size (SS) for the standardized mean difference so that a narrow confidence interval (CI) can be obtained via the accuracy in parameter estimation (AIPE) approach are developed. One method plans SS so that the expected width of the CI is sufficiently narrow. A modification adjusts the SS so that the obtained CI is no wider than desired with some specified degree of certainty (e.g., 99% certain the 95% CI will be no wider than omega). The rationale of the AIPE approach to SS planning is given, as is a discussion of the analytic approach to CI formation for the population standardized mean difference. Tables with values of necessary SS are provided. The freely available Methods for the Behavioral, Educational, and Social Sciences (K. Kelley, 2006a) R (R Development Core Team, 2006) software package easily implements the methods discussed.

Download full-text


Available from: Ken Kelley, Oct 02, 2015
55 Reads
  • Source
    • "Planning a study by focusing on its power is not equivalent to focusing on its accuracy and can lead to different results and decisions (Kelley & Rausch, 2006). For example, for regression coefficients, precision of a parameter estimate depends on sample size, but it is mostly unaffected by effect size, whereas power is affected by both (Kelley and Maxwell, 2003; Figure 2). "
    [Show abstract] [Hide abstract]
    ABSTRACT: Replicability of findings is at the heart of any empirical science. The aim of this article is to move the current replicability debate in psychology towards concrete recommendations for improvement. We focus on research practices but also offer guidelines for reviewers, editors, journal management, teachers, granting institutions, and university promotion committees, highlighting some of the emerging and existing practical solutions that can facilitate implementation of these recommendations. The challenges for improving replicability in psychological science are systemic. Improvement can occur only if changes are made at many levels of practice, evaluation, and reward.
    European Journal of Personality 03/2013; 27(2):108-119. DOI:10.1002/per.1919 · 3.35 Impact Factor
  • Source
    • "When planning new research, previously observed effect sizes can be used to calculate power and thereby estimate appropriate sample sizes. Cohen (1988), Keppel and Wickens (2004), and most statistical textbooks provide guidance on calculating power; a very brief, elementary guide appears in the Appendix along with mention of planning sample sizes based on accuracy in parameter estimation (i.e., planning the size of the CIs; Cumming, 2012; Kelley & Rausch, 2006; Maxwell, Kelley, & Raush, 2008). A brief note on the terminology used in this article may be helpful. "
    [Show abstract] [Hide abstract]
    ABSTRACT: The Publication Manual of the American Psychological Association (American Psychological Association, 2001, American Psychological Association, 2010) calls for the reporting of effect sizes and their confidence intervals. Estimates of effect size are useful for determining the practical or theoretical importance of an effect, the relative contributions of factors, and the power of an analysis. We surveyed articles published in 2009 and 2010 in the Journal of Experimental Psychology: General, noting the statistical analyses reported and the associated reporting of effect size estimates. Effect sizes were reported for fewer than half of the analyses; no article reported a confidence interval for an effect size. The most often reported analysis was analysis of variance, and almost half of these reports were not accompanied by effect sizes. Partial η2 was the most commonly reported effect size estimate for analysis of variance. For t tests, 2/3 of the articles did not report an associated effect size estimate; Cohen's d was the most often reported. We provide a straightforward guide to understanding, selecting, calculating, and interpreting effect sizes for many types of data and to methods for calculating effect size confidence intervals and power analysis.
    Journal of Experimental Psychology General 08/2011; 141(1):2-18. DOI:10.1037/a0024338 · 5.50 Impact Factor
  • Source
    • "However, obtaining an accurate estimate of the parameter does not change at all based on the effect size or tends to do so rather slowly. The relationship between the necessary sample size for statistical power and AIPE relationship has been illustrated for regression coefficients (Kelley & Maxwell, 2003), mean differences (Kelley, Maxwell, & Rausch, 2003), and the standardized mean difference (Kelley & Rausch, 2006), among other effect sizes. Although the two sample size planning approaches can suggest largely discrepant sample sizes at times, such a situation is reasonable since they have very different goals. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Longitudinal studies are necessary to examine individual change over time, with group status often being an important variable in explaining some individual differences in change. Although sample size planning for longitudinal studies has focused on statistical power, recent calls for effect sizes and their corresponding confidence intervals underscore the importance of obtaining sufficiently accurate estimates of group differences in change. We derived expressions that allow researchers to plan sample size to achieve the desired confidence interval width for group differences in change for orthogonal polynomial change parameters. The approaches developed provide the expected confidence interval width to be sufficiently narrow, with an extension that allows some specified degree of assurance (e.g., 99%) that the confidence interval will be sufficiently narrow. We make computer routines freely available, so that the methods developed can be used by researchers immediately.
    Psychological Methods 07/2011; 16(4):391-405. DOI:10.1037/a0023352 · 4.45 Impact Factor
Show more