Psychological Methods

Publications in this journal

• Article: Reliability Estimation in a Multilevel Confirmatory Factor Analysis Framework.

  G John Geldhof, Kristopher J Preacher, Michael J Zyphur
  [show abstract] [hide abstract]
  ABSTRACT: Scales with varying degrees of measurement reliability are often used in the context of multistage sampling, where variance exists at multiple levels of analysis (e.g., individual and group). Because methodological guidance on assessing and reporting reliability at multiple levels of analysis is currently lacking, we discuss the importance of examining level-specific reliability. We present a simulation study and an applied example showing different methods for estimating multilevel reliability using multilevel confirmatory factor analysis and provide supporting Mplus program code. We conclude that (a) single-level estimates will not reflect a scale's actual reliability unless reliability is identical at each level of analysis, (b) 2-level alpha and composite reliability (omega) perform relatively well in most settings, (c) estimates of maximal reliability (H) were more biased when estimated using multilevel data than either alpha or omega, and (d) small cluster size can lead to overestimates of reliability at the between level of analysis. We also show that Monte Carlo confidence intervals and Bayesian credible intervals closely reflect the sampling distribution of reliability estimates under most conditions. We discuss the estimation of credible intervals using Mplus and provide R code for computing Monte Carlo confidence intervals. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
  Psychological Methods 05/2013;
• Article: Reversed Item Bias: An Integrative Model.

  Bert Weijters, Hans Baumgartner, Niels Schillewaert
  [show abstract] [hide abstract]
  ABSTRACT: In the recent methodological literature, various models have been proposed to account for the phenomenon that reversed items (defined as items for which respondents' scores have to be recoded in order to make the direction of keying consistent across all items) tend to lead to problematic responses. In this article we propose an integrative conceptualization of three important sources of reversed item method bias (acquiescence, careless responding, and confirmation bias) and specify a multisample confirmatory factor analysis model with 2 method factors to empirically test the hypothesized mechanisms, using explicit measures of acquiescence and carelessness and experimentally manipulated versions of a questionnaire that varies 3 item arrangements and the keying direction of the first item measuring the focal construct. We explain the mechanisms, review prior attempts to model reversed item bias, present our new model, and apply it to responses to a 4-item self-esteem scale (N = 306) and the 6-item Revised Life Orientation Test (N = 595). Based on the literature review and the empirical results, we formulate recommendations on how to use reversed items in questionnaires. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
  Psychological Methods 05/2013;
• Article: Coupled Latent Differential Equation With Moderators: Simulation and Application.

  Yueqin Hu, Steve Boker, Michael Neale, Kelly L Klump
  [show abstract] [hide abstract]
  ABSTRACT: Latent differential equations (LDE) use differential equations to analyze time series data. Because of the recent development of this technique, some issues critical to running an LDE model remain. In this article, the authors provide solutions to some of these issues and recommend a step-by-step procedure demonstrated on a set of empirical data, which models the interaction between ovarian hormone cycles and emotional eating. Results indicated that emotional eating is self-regulated. For instance, when people do more emotional eating than normal, they will subsequently tend to decrease their emotional eating behavior. In addition, a sudden increase will produce a stronger tendency to decrease than will a slow increase. We also found that emotional eating is coupled with the cycle of the ovarian hormone estradiol, and the peak of emotional eating occurs after the peak of estradiol. The self-reported average level of negative affect moderates the frequency of eating regulation and the coupling strength between eating and estradiol. Thus, people with a higher average level of negative affect tend to fluctuate faster in emotional eating, and their eating behavior is more strongly coupled with the hormone estradiol. Permutation tests on these empirical data supported the reliability of using LDE models to detect self-regulation and a coupling effect between two regulatory behaviors. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
  Psychological Methods 05/2013;
• Article: Error, Power, and Cluster Separation Rates of Pairwise Multiple Testing Procedures.

  Juliet Popper Shaffer, Rhonda K Kowalchuk, H J Keselman
  [show abstract] [hide abstract]
  ABSTRACT: In comparing multiple treatments, 2 error rates that have been studied extensively are the familywise and false discovery rates. Different methods are used to control each of these rates. Yet, it is rare to find studies that compare the same methods on both of these rates, and also on the per-family error rate, the expected number of false rejections. Although the per-family error rate and the familywise error rate are similar in most applications when the latter is controlled at a conventional low level (e.g., .05), the 2 measures can diverge considerably with methods that control the false discovery rate at that same level. Furthermore, we shall consider both rejections of true hypotheses (Type I errors) and rejections of false hypotheses where the observed outcomes are in the incorrect direction (Type III errors). We point out that power estimates based on the number of correct rejections do not consider the pattern of those rejections, which is important in interpreting the total outcome. The present study introduces measures of interpretability based on the pattern of separation of treatments into nonoverlapping sets and compares methods on these measures. In general, range-based (configural) methods are more likely to obtain interpretable patterns based on treatment separation than individual p-value-based measures. Recommendations for practice based on these results are given in the article. Although the article is complex, these recommendations can be understood without the necessity for detailed perusal of the supporting material. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
  Psychological Methods 05/2013;
• Article: Posterior Predictive Checks of Tetrad Subsets for Covariance Structures of Measurement Models.

  Timothy R Johnson, Todd E Bodner
  [show abstract] [hide abstract]
  ABSTRACT: Confirmatory tetrad analysis is based on tetrads that are assumed to be 0 or to "vanish" under an assumed model to test the goodness of fit of a model or the relative fit of nested models. A strength of a tetrad-based approach to model fit assessment over standard methods is that researchers can evaluate the plausibility of a theoretical model structure when estimation or identification is problematic. Previous research has developed confirmatory tetrad analysis for covariance structures using asymptotic and bootstrap sampling distributions for omnibus test statistics for all vanishing tetrads. We extend confirmatory tetrad analysis in 2 ways. First, we develop an approach to confirmatory tetrad analysis based on posterior predictive checks for Bayesian probability models using tetrads to construct discrepancy measures. Second, we show how the discrepancy measure for all vanishing tetrads can be additively decomposed into measures for subsets of tetrads using a typology of tetrads proposed by Kenny (1979). We show that posterior predictive checks using a hierarchy of subsets of tetrads from this typology can be effectively used to identify potential and specific sources for lack of fit. We demonstrate the utility of this approach with classic data sets from the literature and with simulated data. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
  Psychological Methods 03/2013;
• Article: The Reservoir Model: A Differential Equation Model of Psychological Regulation.

  Pascal R Deboeck, C S Bergeman
  [show abstract] [hide abstract]
  ABSTRACT: Differential equation models can be used to describe the relationships between the current state of a system of constructs (e.g., stress) and how those constructs are changing (e.g., based on variable-like experiences). The following article describes a differential equation model based on the concept of a reservoir. With a physical reservoir, such as one for water, the level of the liquid in the reservoir at any time depends on the contributions to the reservoir (inputs) and the amount of liquid removed from the reservoir (outputs). This reservoir model might be useful for constructs such as stress, where events might "add up" over time (e.g., life stressors, inputs), but individuals simultaneously take action to "blow off steam" (e.g., engage coping resources, outputs). The reservoir model can provide descriptive statistics of the inputs that contribute to the "height" (level) of a construct and a parameter that describes a person's ability to dissipate the construct. After discussing the model, we describe a method of fitting the model as a structural equation model using latent differential equation modeling and latent distribution modeling. A simulation study is presented to examine recovery of the input distribution and output parameter. The model is then applied to the daily self-reports of negative affect and stress from a sample of older adults from the Notre Dame Longitudinal Study on Aging. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
  Psychological Methods 03/2013;
• Article: Robustness of Fit Indices to Outliers and Leverage Observations in Structural Equation Modeling.

  Ke-Hai Yuan, Xiaoling Zhong
  [show abstract] [hide abstract]
  ABSTRACT: Normal-distribution-based maximum likelihood (NML) is the most widely used method in structural equation modeling (SEM), although practical data tend to be nonnormally distributed. The effect of nonnormally distributed data or data contamination on the normal-distribution-based likelihood ratio (LR) statistic is well understood due to many analytical and empirical studies. In SEM, fit indices are used as widely as the LR statistic. In addition to NML, robust procedures have been developed for more efficient and less biased parameter estimates with practical data. This article studies the effect of outliers and leverage observations on fit indices following NML and two robust methods. Analysis and empirical results indicate that good leverage observations following NML and one of the robust methods lead most fit indices to give more support to the substantive model. While outliers tend to make a good model superficially bad according to many fit indices following NML, they have little effect on those following the two robust procedures. Implications of the results to data analysis are discussed, and recommendations are provided regarding the use of estimation methods and interpretation of fit indices. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
  Psychological Methods 03/2013;
• Article: Mixture Class Recovery in GMM Under Varying Degrees of Class Separation: Frequentist Versus Bayesian Estimation.

  Sarah Depaoli
  [show abstract] [hide abstract]
  ABSTRACT: Growth mixture modeling (GMM) represents a technique that is designed to capture change over time for unobserved subgroups (or latent classes) that exhibit qualitatively different patterns of growth. The aim of the current article was to explore the impact of latent class separation (i.e., how similar growth trajectories are across latent classes) on GMM performance. Several estimation conditions were compared: maximum likelihood via the expectation maximization (EM) algorithm and the Bayesian framework implementing diffuse priors, "accurate" informative priors, weakly informative priors, data-driven informative priors, priors reflecting partial-knowledge of parameters, and "inaccurate" (but informative) priors. The main goal was to provide insight about the optimal estimation condition under different degrees of latent class separation for GMM. Results indicated that optimal parameter recovery was obtained though the Bayesian approach using "accurate" informative priors, and partial-knowledge priors showed promise for the recovery of the growth trajectory parameters. Maximum likelihood and the remaining Bayesian estimation conditions yielded poor parameter recovery for the latent class proportions and the growth trajectories. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
  Psychological Methods 03/2013;
• Article: Modeling Actor and Partner Effects in Dyadic Data When Outcomes Are Categorical.

  Tom Loeys, Geert Molenberghs
  [show abstract] [hide abstract]
  ABSTRACT: When 2 people interact in a relationship, the outcome of each person can be affected by both his or her own inputs and his or her partner's inputs. For Gaussian dyadic outcomes, linear mixed models taking into account the correlation within dyads are frequently used to estimate actor's and partner's effects based on the actor-partner interdependence model. In this article, we explore the potential of generalized linear mixed models (GLMMs) for the analysis of non- Gaussian dyadic outcomes. Several approximation techniques that are available in standard software packages for these GLMMs are investigated. Despite the different modeling options related to these different techniques, none of these have an overall satisfactory performance in estimating actor and partner effects and the within-dyad correlation, especially when the latter is negative and/or the number of dyads is small. An approach based on generalized estimating equations for the analysis of non-Gaussian dyadic data turns out to be an interesting alternative. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
  Psychological Methods 03/2013;
• Article: Information utility: Quantifying the total psychometric information provided by a measure.

  Kristian E Markon
  [show abstract] [hide abstract]
  ABSTRACT: Although advances have improved our ability to describe the measurement precision of a test, it often remains challenging to summarize how well a test is performing overall. Reliability, for example, provides an overall summary of measurement precision, but it is sample-specific and might not reflect the potential usefulness of a test if the sample is poorly suited for the test's purposes. The test information function, conversely, provides detailed sample-independent information about measurement precision, but it does not provide an overall summary of test performance. Here, the concept of information utility is introduced. Information utility provides an index of how much psychometric information a measure (e.g., item, test) provides about a trait overall. Information utility has a number of important applied implications, including test selection, trait estimation, computerized adaptive testing, and hypothesis testing. Information utility may have particular utility in situations where the accuracy of prior information about trait level is vague or unclear. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
  Psychological Methods 03/2013; 18(1):15-35.
• Article: Correction to doebler, holling, & böhning (2012).

  Philipp Doebler, Heinz Holling, Dankmar Böhning
  [show abstract] [hide abstract]
  ABSTRACT: Reports an error in "A mixed model approach to meta-analysis of diagnostic studies with binary test outcome" by Philipp Doebler, Heinz Holling and Dankmar Böhning (Psychological Methods, 2012[Sep], Vol 17[3], 418-436). For the article, Drs. Daming Lin of the Dalla Lana School of Public Health and George Tomlinson of Toronto General Hospital and the Dalla Lana School of Public Health noted an error in the final version of Equations 6 and 7 on page 423. Dr. Doebler, in a conversation with the Interim Editor, acknowledged the error in the printing of the equations. Dr. Doebler also checked and assured the Interim Editor that the R-code that generated the substantive results for the paper were correctly coded and are identical to the R-code that would result from the derivations suggested by Drs. Lin and Tomlinson and is provided. (The following abstract of the original article appeared in record 2012-12662-001.) We propose 2 related models for the meta-analysis of diagnostic tests. Both models are based on the bivariate normal distribution for transformed sensitivities and false-positive rates. Instead of using the logit as a transformation for these proportions, we employ the tα family of transformations that contains the log, logit, and (approximately) the complementary log. A likelihood ratio test for the cutoff value problem is developed, and summary receiver operating characteristic (SROC) curves are discussed. Worked examples showcase the methodology. We compare the models to the hierarchical SROC model, which in contrast employs a logit transformation. Data from various meta-analyses are reanalyzed, and the reanalysis indicates a better performance of the models based on the tα transformation. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
  Psychological Methods 03/2013; 18(1):120.
• Article: Examination of the equivalence of self-report survey-based paper-and-pencil and internet data collection methods.

  Arne Weigold, Ingrid K Weigold, Elizabeth J Russell
  [show abstract] [hide abstract]
  ABSTRACT: Self-report survey-based data collection is increasingly carried out using the Internet, as opposed to the traditional paper-and-pencil method. However, previous research on the equivalence of these methods has yielded inconsistent findings. This may be due to methodological and statistical issues present in much of the literature, such as nonequivalent samples in different conditions due to recruitment, participant self-selection to conditions, and data collection procedures, as well as incomplete or inappropriate statistical procedures for examining equivalence. We conducted 2 studies examining the equivalence of paper-and-pencil and Internet data collection that accounted for these issues. In both studies, we used measures of personality, social desirability, and computer self-efficacy, and, in Study 2, we used personal growth initiative to assess quantitative equivalence (i.e., mean equivalence), qualitative equivalence (i.e., internal consistency and intercorrelations), and auxiliary equivalence (i.e., response rates, missing data, completion time, and comfort completing questionnaires using paper-and-pencil and the Internet). Study 1 investigated the effects of completing surveys via paper-and-pencil or the Internet in both traditional (i.e., lab) and natural (i.e., take-home) settings. Results indicated equivalence across conditions, except for auxiliary equivalence aspects of missing data and completion time. Study 2 examined mailed paper-and-pencil and Internet surveys without contact between experimenter and participants. Results indicated equivalence between conditions, except for auxiliary equivalence aspects of response rate for providing an address and completion time. Overall, the findings show that paper-and-pencil and Internet data collection methods are generally equivalent, particularly for quantitative and qualitative equivalence, with nonequivalence only for some aspects of auxiliary equivalence. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
  Psychological Methods 03/2013; 18(1):53-70.
• Article: Mediation Analysis Allowing for Exposure-Mediator Interactions and Causal Interpretation: Theoretical Assumptions and Implementation With SAS and SPSS Macros.

  Linda Valeri, Tyler J Vanderweele
  [show abstract] [hide abstract]
  ABSTRACT: Mediation analysis is a useful and widely employed approach to studies in the field of psychology and in the social and biomedical sciences. The contributions of this article are several-fold. First we seek to bring the developments in mediation analysis for nonlinear models within the counterfactual framework to the psychology audience in an accessible format and compare the sorts of inferences about mediation that are possible in the presence of exposure-mediator interaction when using a counterfactual versus the standard statistical approach. Second, the work by VanderWeele and Vansteelandt (2009, 2010) is extended here to allow for dichotomous mediators and count outcomes. Third, we provide SAS and SPSS macros to implement all of these mediation analysis techniques automatically, and we compare the types of inferences about mediation that are allowed by a variety of software macros. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
  Psychological Methods 02/2013;
• Article: "Many test of significance: New methods for controlling type I errors": Correction to Keselman et al. (2011).

  H J Keselman, Charles W Miller, Burt Holland
  [show abstract] [hide abstract]
  ABSTRACT: Reports an error in "Many tests of significance: New methods for controlling type I errors" by H. J. Keselman, Charles W. Miller and Burt Holland (Psychological Methods, 2011[Dec], Vol 16[4], 420-431). The R code for arriving at adjusted p values for one of the methods is incorrect. The specific changes that need to be made are provided in the erratum. (The following abstract of the original article appeared in record 2011-24639-001.) There have been many discussions of how Type I errors should be controlled when many hypotheses are tested (e.g., all possible comparisons of means, correlations, proportions, the coefficients in hierarchical models, etc.). By and large, researchers have adopted familywise (FWER) control, though this practice certainly is not universal. Familywise control is intended to deal with the multiplicity issue of computing many tests of significance, yet such control is conservative-that is, less powerful-compared to per test/hypothesis control. The purpose of our article is to introduce the readership, particularly those readers familiar with issues related to controlling Type I errors when many tests of significance are computed, to newer methods that provide protection from the effects of multiple testing, yet are more powerful than familywise controlling methods. Specifically, we introduce a number of procedures that control the k-FWER. These methods-say, 2-FWER instead of 1-FWER (i.e., FWER)-are equivalent to specifying that the probability of 2 or more false rejections is controlled at .05, whereas FWER controls the probability of any (i.e., 1 or more) false rejections at .05. 2-FWER implicitly tolerates 1 false rejection and makes no explicit attempt to control the probability of its occurrence, unlike FWER, which tolerates no false rejections at all. More generally, k-FWER tolerates k - 1 false rejections, but controls the probability of k or more false rejections at α =.05. We demonstrate with two published data sets how more hypotheses can be rejected with k-FWER methods compared to FWER control. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
  Psychological Methods 12/2012; 17(4):679.
• Article: Bayesian Methods for the Analysis of Small Sample Multilevel Data With a Complex Variance Structure.

  Scott A Baldwin, Gilbert W Fellingham
  [show abstract] [hide abstract]
  ABSTRACT: Inferences from multilevel models can be complicated in small samples or complex data structures. When using (restricted) maximum likelihood methods to estimate multilevel models, standard errors and degrees of freedom often need to be adjusted to ensure that inferences for fixed effects are correct. These adjustments do not address problems in estimating variance/covariance components. An alternative to the adjusted likelihood method is to use Bayesian methods, which can produce accurate inferences about fixed effects and variance/covariance parameters. In this article, the authors contrast the benefits and limitations of likelihood and Bayesian methods in the estimation of multilevel models. The issues are discussed in the context of a partially clustered intervention study, a common intervention design that has been shown to require an adjusted likelihood analysis. The authors report a Monte Carlo study that compares the performance of an adjusted restricted maximum likelihood (REML) analysis to a Bayesian analysis. The results suggest that for fixed effects, the models perform equally well with respect to bias, efficiency, and coverage of interval estimates. Bayesian models with a carefully selected gamma prior for the variance components were more biased but also more efficient with respect to estimation of the variance components than the REML model. However, the results also show that the inferences about the variance components in partially clustered studies are sensitive to the prior distribution when sample sizes are small. Finally, the authors compare the results of a Bayesian and adjusted likelihood model using data from a partially clustered intervention trial. (PsycINFO Database Record (c) 2012 APA, all rights reserved).
  Psychological Methods 11/2012;