Psychological Assessment

Published by American Psychological Association
Online ISSN: 1939-134X
Print ISSN: 1040-3590
The National Consumer Assessment of Health Plans Study (CAHPS) Benchmarking Database was used to assess the factor structure and invariance of the CAHPS 1.0 Core Survey. Separate analyses were conducted with Latino and non-Latino Caucasian consumers drawn from commercial and Medicaid sectors (N = 15,092). Results demonstrated that the 23 CAHPS 1.0 report items measure consumer reports of experiences with 5 aspects of health plan performance: Access to Care, Timeliness of Care, Provider Communication, Health Plan Consumer Service, and Office Staff Helpfulness. Four items assessed global ratings of care. Analyses revealed an identical pattern of fixed and free factor loadings across all samples. Magnitude of factor loadings and correlations among factors was essentially equivalent within a common health service sector. A higher order factor analysis revealed that rating and reports of care showed marked convergence.
The validity of the Impact of Events Scale (IES) and the Posttraumatic Stress Disorder (PTSD) Symptom Scale, Self-Report version (PSS-SR) was examined among crime victims. Both instruments performed well as screeners for PTSD. For the IES, sensitivity ranged between .93 and 1.00; for the PSS-SR, sensitivity ranged between .80 and .90. Specificity for the IES ranged between .78 and .84 and for the PSS-SR ranged between .84 and .88. Some individual items from the 2 scales performed just as well as the total scales. The authors conclude that either of these short self-report instruments or their individual items are suitable as screeners for PTSD, specifically in settings where mental health professionals are unavailable. Cross-validation of these results is necessary because of the small sample size in this study.
Hierarchical iclust analysis specifying three clusters. C cluster. This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly. 
Hierarchical iclust analysis specifying six clusters. C cluster. 
Eleven BIS-11 Parcels and One-and Two-Factor Solutions for the BIS-11 Parcels Factor models 
The Barratt Impulsiveness Scale (Version 11; BIS-11; Patton, Stanford, & Barratt, 1995) is a gold-standard measure that has been influential in shaping current theories of impulse control, and has played a key role in studies of impulsivity and its biological, psychological, and behavioral correlates. Psychometric research on the structure of the BIS-11, however, has been scant. We therefore applied exploratory and confirmatory factor analyses to data collected using the BIS-11 in a community sample (N = 691). Our goal was to test 4 theories of the BIS-11 structure: (a) a unidimensional model, (b) a 6 correlated first-order factor model, (c) a 3 second-order factor model, and (d) a bifactor model. Among the problems identified were (a) low or near-zero correlations of some items with others; (b) highly redundant content of numerous item pairs; (c) items with salient cross-loadings in multidimensional solutions; and, ultimately, (d) poor fit to confirmatory models. We conclude that use of the BIS-11 total score as reflecting individual differences on a common dimension of impulsivity presents challenges in interpretation. Also, the theory that the BIS-11 measures 3 subdomains of impulsivity (attention, motor, and nonplanning) was not empirically supported. A 2-factor model is offered as an alternative multidimensional structural representation. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
Types of risk assessments 
Key to studies used in the meta-analysis 
This review compared the accuracy of various approaches to the prediction of recidivism among sexual offenders. On the basis of a meta-analysis of 536 findings drawn from 118 distinct samples (45,398 sexual offenders, 16 countries), empirically derived actuarial measures were more accurate than unstructured professional judgment for all outcomes (sexual, violent, or any recidivism). The accuracy of structured professional judgment was intermediate between the accuracy found for the actuarial measures and for unstructured professional judgment. The effect sizes for the actuarial measures were moderate to large by conventional standards (average d values of 0.67-0.97); however, the utility of the actuarial measures will vary according to the referral question and samples assessed. Further research should identify the psychologically meaningfully factors that contribute to risk for reoffending. (PsycINFO Database Record (c) 2009 APA, all rights reserved).
This article describes a series of studies involving 2,730 participants on the development and validity testing of the Severity Indices of Personality Problems (SIPP), a self-report questionnaire covering important core components of (mal)adaptive personality functioning. Results show that the 16 facets constituted homogeneous item clusters (i.e., unidimensional and internally consistent parcels) that fit well into 5 clinically interpretable, higher order domains: self-control, identity integration, relational capacities, social concordance, and responsibility. These domains appeared to have good concurrent validity across various populations, good convergent validity in terms of associations with interview ratings of the severity of personality pathology, and good discriminant validity in terms of associations with trait-based personality disorder dimensions. Furthermore, results suggest that the domain scores are stable over a time interval of 14-21 days in a student sample but are sensitive to change over a 2-year follow-up interval in a treated patient population. Taken together, the final instrument, the SIPP-118, provides a set of 5 reliable, valid, and efficient indices of the core components of (mal)adaptive personality functioning.
Overview of the Samples, With a Total of 767 Adolescents 
Correlation Between Severity Indices of Personality Problems Domain Scores and Global Severity Index (GSI; Sample 2; n 71 a ) Domain GSI 
The Severity Indices of Personality Problems (SIPP-118; Verheul et al., 2008) is a self-report questionnaire focusing on core components of (mal)adaptive personality functioning. The SIPP-118 was developed and validated in an adult population. In adult populations, the 16 facets of the SIPP-118 fit into 5 higher order domains: self-control, identity integration, relational capacities, social concordance, and responsibility. In this study we present the 1st psychometric properties of the SIPP-118 in adolescents. We compared the SIPP-118 scores of a patient and a nonpatient sample of adolescents, and compared personality disordered and non-personality disordered adolescents. In addition, the relationship between scores on the SIPP-118 and other clinical instruments (Symptom Checklist-90-Revised; SCL-90-R; Derogatis, 1975; Dimensional Assessment of Personality Pathology-Basic Personality; DAPP-BQ; Livesley & Jackson, 2002) was investigated. The questionnaires were completed by 378 adolescent patients and 389 adolescents in the community. Facets appeared to be homogeneous, as alpha coefficients ranged from .62 to .89, indicating moderate to acceptable reliability. Also, more pathological SIPP-118 scores were found in the patient sample, and more specifically in the personality disordered sample, suggesting that the facet scores of the SIPP-118 can discriminate between various populations (divergent validity). Correlation with other clinical instruments was moderate to high (-.82 to .10). Taken together, the SIPP-118 seems to be a promising instrument measuring personality pathology in adolescents.
This article describes the development and validation of a brief self-report scale for diagnosing anorexia nervosa, bulimia nervosa, and binge-eating disorder. Study 1 used a panel of eating-disorder experts and provided evidence for the content validity of this scale. Study 2 used data from female participants with and without eating disorders (N = 367) and suggested that the diagnoses from this scale possessed temporal reliability (mean kappa = .80) and criterion validity (with interview diagnoses; mean kappa = .83). In support of convergent validity, individuals with eating disorders identified by this scale showed elevations on validated measures of eating disturbances. The overall symptom composite also showed test-retest reliability (r = .87), internal consistency (mean alpha = .89), and convergent validity with extant eating-pathology scales. Results implied that this scale was reliable and valid in this investigation and that it may be useful for clinical and research applications.
Correlations Between the PID5 Traits and the NPI-13 Subscales LA GE EE 
The most widely used measure of trait narcissism is the Narcissistic Personality Inventory (NPI), which can provide both total and subscale scores. However, with a length of 40 items, this measure may not be ideal in settings in which time or participant attention may limit the types of measures that can be administered. In response, Ames, Rose, and Anderson (2006) created the NPI-16, which provides a shorter, unidimensional measure of the construct. In the present research, we examine the reliability and validity of the NPI-16 in conjunction with a new short measure of narcissism, the NPI-13, which provides both a total score and 3 subscale scores (Leadership/Authority; Grandiose Exhibitionism; Entitlement/Exploitativeness). Across 2 studies, we demonstrate that both short measures manifest good convergent and discriminant validity and adequate overall reliability. The NPI-13 may be favored over the NPI-16 because it allows for the extraction of 3 subscales, consistent with the use of its parent measure. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
Psychopathy is characterized by diverse indicators. Clinical accounts have emphasized 3 distinct facets: interpersonal, affective, and behavioral. Research using the Psychopathy Checklist--Revised (PCL-R), however, has emphasized a 2-factor model. A review of the literature on the PCL-R and related measures of psychopathy, together with confirmatory factor analysis of PCL-R data from North American participants, indicates that the 2-factor model cannot be sustained. A 3-factor hierarchical model was developed in which a coherent superordinate factor, Psychopathy, is underpinned by 3 factors: Arrogant and Deceitful Interpersonal Style, Deficient Affective Experience, and Impulsive and Irresponsible Behavioral Style. The model was cross-validated on North American and Scottish PCL-R data, Psychopathy Screening Version data, and data derived from the Diagnostic and Statistical Manual of Mental Disorders (4th ed.; American Psychiatric Association, 1994) antisocial personality disorder field trial.
Genetic and environmental influences for psychopathic personality, ages 14-15 years. CPS Child Psychopathy Scale; APSD Antisocial Process Screening Device; PCL:YV Psychopathy Checklist: Youth Version, P parent ratings, Y youth self-reports; A additive genetic variance; C shared environmental variance. A and C estimates for the PCL:YV Interpersonal subscale are nonsignificant.
Phenotypic Correlations for Psychopathic Personality, Age 14 -15 Years
Best fitting one-factor common pathway for psychopathic personality, ages 14 –15 years. Standard- ized path estimates and 95% confidence intervals from the best fitting one-factor common pathway model for CPS Parent (parent-rated Child Psychopathy Scale), CPS Youth (youth self-reported Child Psychopathy Scale), APSD Parent (parent-rated Antisocial Process Screening Device), APSD Youth (youth self-reported Antisocial Process Screening Device), and PCL:YV (Psychopathy Checklist: Youth Version). Common additive genetic factors (A) and nonshared environmental factors (E) are depicted in circles. The latent factor is labeled Psychopathic Personality. Measured variables are depicted in rectangles. As ϭ additive genetic residual variance specific to each measure; Es ϭ nonshared environment variance specific to each measure. 
Until now, no study has examined the genetic and environmental influences on psychopathic personality across different raters and method of assessment. Participants were part of a community sample of male and female twins born between 1990 and 1995. The Child Psychopathy Scale and the Antisocial Process Screening Device were administered to the twins and their parents when the twins were 14-15 years old. The Psychopathy Checklist: Youth Version (PCL:YV) was administered and scored by trained testers. Results showed that a 1-factor common pathway model was the best fit for the data. Genetic influences explained 69% of the variance in the latent psychopathic personality factor, while nonshared environmental influences explained 31%. Measurement-specific genetic effects accounted for between 9% and 35% of the total variance in each of the measures, except for PCL:YV, where all genetic influences were in common with the other measures. Measure-specific nonshared environmental influences were found for all measures, explaining between 17% and 56% of the variance. These findings provide further evidence of the heritability in psychopathic personality among adolescents, although these effects vary across the ways in which these traits are measured, in terms of both informant and instrument used. (PsycINFO Database Record (c) 2014 APA, all rights reserved).
The aim of this study was to develop and validate manic and depressive scales carved from the full-length General Behavior Inventory (GBI). The brief version was designed to be applicable for youths and adults and to improve separation between mania and depression dimensions. Data came from 9 studies (2 youth clinical samples, aggregate N = 738, and 7 nonclinical adult samples, aggregate N = 1,756). Items with high factor loadings on the 2 extracted dimensions of mania and depression were identified from both data sets, and final item selection was based on internal reliability criteria. Confirmatory factor analyses described the 2-factor model's fit. Criterion validity was compared between mania and depression scales, and with the full-length GBI scales. For both mania and depression factors, 7 items produced a psychometrically adequate measure applicable across both aggregate samples. Internal reliability of the Mania scale was .81 (youth) and .83 (adult) and for Depression was .93 (youth) and .95 (adult). By design, the brief scales were less strongly correlated with each other than were the original GBI scales. Construct validity of the new instrument was supported in observed discriminant and convergent relationships with external correlates and discrimination of diagnostic groups. The new brief GBI, the 7 Up 7 Down Inventory, demonstrates sound psychometric properties across a wide age range, showing expected relationships with external correlates. The new instrument provides a clearer separation of manic and depressive tendencies than the original. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
Psychologists typically rely on patients' self-report of premorbid status in litigated settings. The authors examined the fidelity between self-reported and actual scholastic performance in litigating head injury claimants. The data indicated late postconcussion syndrome (LPCS) and severe closed head injury litigants retrospectively inflated scholastic performance to a greater degree than nonlitigating control groups. The LPCS group showed the highest magnitude of grade inflation, but discrepancy scores did not significantly correlate with a battery of malingering measures or with objective cerebral dysfunction. These findings support previous studies, which showed self-report is not a reliable basis for estimation of preinjury cognitive status. Retrospective inflation may represent a response shift bias shaped by an adversarial context rather than a form of malingering.
The psychometric properties of the newest version of the Temperament and Character Inventory (the TCI-R) were evaluated in a large (n = 727) community sample, as was the TCI-140, a short inventory derivative. Facets-to-scale confirmatory and exploratory factor analyses of the TCI-R did not support the organization of temperament and character facet scales within their superordinate domains. Five of the 29 facet scales also displayed relatively low internal consistency (a < .70). Factor analyses of the TCI-140 item set yielded only limited support for hypothesized item-to-scale memberships. Harm Avoidance, Novelty Seeking, and Self-Directedness items, in particular, were not well differentiated. Although psychometrically comparable, the TCI-R and the TCI-140 demonstrate many of the limitations of earlier inventory versions. Implications associated with the use of the TCI-R and TCI-140 and C. R. Cloninger's theory of personality are discussed.
Frequencies for MH-CASC diagnostic groups Frequency Percent 
Frequencies for age groups Age in years Frequency Percent
Multilevel descriptive statistics 
The aim of this study was to assess the factor structure of the Life Skills Profile-16 (LSP-16; Buckingham, Burgess, Solomon, Pirkis, & Eagar, 1998a, 1998b) with a view to meeting the assumption of statistical independence that is at significant risk of violation due to the dependency introduced to the data by pooling numerous ratings made by the same observers across independent patients. The sample consisted of 20,181 outpatients rated by 2,071 clinicians employed within 54 mental health organizations within the New South Wales public adult mental health service. To estimate the extent to which the item scores were contaminated with rater-level intraclass correlations (ICC), I fit 16 3-level multinominal ordered proportional odds intercept only models that revealed large ICCs associated with Level 2 (the rater of the LSP-16) demonstrating that a multilevel analysis was required. A multilevel confirmatory factor analysis (M-CFA) using robust weighted least squares (B. O. Muthén, du Toit, & Spisic, 1997) with polychoric correlation was used to test the fit of 2 measurement models that were hypothesized a priori. The 2 models failed to provide an acceptable fit to the sample data and within- and between-level CFAs were used to inform revisions to the 4-factor model. A 15-item version of the LSP was developed, which provided an improved approximate fit in an M-CFA. Limitations of these findings are discussed. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
This article discusses issues related to the development and dissemination of evidence-based assessment strategies for anxiety disorders and associated problems. It begins with a review of the criteria that should be considered when determining whether particular assessment procedures are evidence-based. These include such factors as reliability, validity, cost-effectiveness, consumer acceptability, utility across different populations, and ease of dissemination. The importance of considering the purpose of the assessment process when deciding whether a procedure is evidence-based is emphasized. Next, the major assessment domains that are particularly important in the area of anxiety disorders (e.g., triggers for anxiety symptoms, avoidance behaviors) are reviewed. Finally, some potential obstacles to the dissemination of evidence-based assessment procedures are discussed along with suggestions for managing these obstacles.
Taxometrics is a statistical tool that can be used to discern categories from continua. Taxometric analyses (MAXCOV and MAXEIG) were conducted in a large nonclinical sample (N=1,215) to determine whether extreme anxiety forms a distinct psychopathological category, an anxiety taxon. Anxiety was operationalized with self-report measures of subjective anxiety, anxious cognitive style, physiological arousal, and anxiety-related impairment. Procedures consistently identified a taxon with a prevalence of approximately 11%. Examination of the taxon's convergent and discriminant validity revealed that it reflects general distress rather than physiological arousal. Taxon membership showed some evidence of incremental validity.
This article describes a conceptual framework for couple-based assessment strategies grounded in empirical findings linking couple distress to a broad range of both individual and relationship characteristics. These characteristics can contribute to, exacerbate, or result from relationship problems. On the basis of these findings, the authors articulate specific targets of clinical inquiry reflecting relationship behaviors, cognitions, and affect as well as features of individual distress. Guided by this framework, empirically supported assessment strategies and techniques emphasizing relationship functioning across diverse methods are proposed, including the clinical interview, analog behavioral observation, and both self- and other-report measures. Discussion concludes with specific recommendations regarding clinical assessment of couple distress and directions for further research.
The factor structure of the Brief Symptom Inventory--18 (BSI-18; L. R. Derogatis, 2000) was investigated in a sample of adult survivors of childhood cancer enrolled in the Childhood Cancer Survivor Study (CCSS; N = 8,945). An exploratory factor analysis with a randomly chosen subsample supported a 3-factor structure closely corresponding to the 3 BSI-18 subscales: Depression, Anxiety, and Somatization. Confirmatory factor analysis with structural equation modeling validated this 3-dimensional structure in a separate subsample, though an alternative 4-factor model also fit the data. Analysis of the 3-factor model showed consistent fit in male and female participants. Compared with available community-based norms, survivors reported fewer symptoms of psychological distress. Together, results support the hypothesized 3-dimensional structure of the BSI-18 and indicate the measure may be useful in assessing psychological distress in this growing population of cancer survivors.
Confirmatory factor analysis with robust weighted least squares estimation of the 103 dichotomously scored items of the Child Behavior Checklist/6-18 (T. M. Achenbach & L. A. Rescorla, 2001) in a sample of 516 girls adopted from China (ages 6.0-15.7 years; M = 8.2, SD = 1.9) indicated that the fit of the 8-factor model was good (root-mean-square error of approximation = .047) and was slightly better than what T. M. Achenbach and L. A. Rescorla (2001) reported for the same model (.06). Support for the 2nd-order factor structure of Internalizing and Externalizing Problems also was provided. Comparisons of the mean scores for the syndromes and Internalizing, Externalizing, and Total Problems revealed mostly small differences between the sample of adopted Chinese girls and T. M. Achenbach and L. A. Rescorla's normative samples.
Distribution of item information for Items 6, 8, and 4. 
Distributions of total scale information and standard error of measurement in the clinical sample. 
Distributions of total scale information and standard error of measurement in the student sample. 
Correlations Between the Subscales and the Total Score for Different Samples Sample Somatization Depression Anxiety 
The psychometric structure of the Brief Symptom Inventory-18 (BSI-18; Derogatis, 2001) was investigated using Mokken scaling and parametric item response theory. Data of 487 outpatients, 266 students, and 207 prisoners were analyzed. Results of the Mokken analysis indicated that the BSI-18 formed a strong Mokken scale for outpatients and prisoners, indicating strong unidimensionality. For students, only the depression and anxiety items formed a medium Mokken scale. Parametric item response theory analyses showed that the best discriminating items came from the depression and anxiety subscales.
The cultural equivalence of psychological outcome measures remains a major area of investigation. The current study sought to test the factor structure and factorial invariance of the Brief Symptom Inventory-18 (BSI-18) with a sample of adult individuals of Mexican descent (N = 923) across nativity status (U.S.- vs. foreign-born), language format (English vs. Spanish), and gender. The results show that 1-factor and 3-factor measurement models provided a good fit to the data; however, a single-factor model was deemed more appropriate and parsimonious. Tests of measurement invariance and invariance of factor variances (i.e., structural invariance) indicated at least partial measurement invariance across gender, nativity status, and language format. These findings suggest that the BSI-18 operates in a similar fashion among adults of Mexican descent regardless of nativity status, language format of the survey, and gender. Clinical and practical implications for use of the BSI-18 with Latino populations are discussed. (PsycINFO Database Record (c) 2012 APA, all rights reserved).
Both the Minnesota Multiphasic Personality Inventory--2 (MMPI-2) and the Minnesota Multiphasic Personality Inventory--Adolescent (MMPI-A) may be administered to 18-year-olds. Each test was administered to 18-year-old participants classified as psychopathology present (PP) or psychopathology absent (PA) to assess (a) the degree of correspondence between the 2 test versions in yielding clinically elevated or nonclinically elevated profiles and (b) the relative accuracy of the 2 test versions in identifying the presence of psychopathology. The 2 tests produced profiles that were inconsistent in clinical elevation status in 70 of 152 participants (46%). All 70 participants with incongruent profiles had clinically elevated MMPI-2 scores and normal-range MMPI-A scores. Analyses of incongruent profiles obtained by PP and PA participants indicated that 18-year-olds were overpathologized by the MMPI-2 and underpathologized by the MMPI-A.
This study used data from 3 sites to examine the invariance and psychometric characteristics of the Brief Symptom Inventory-18 across Black, Hispanic, and White mothers of 5th graders (N = 4,711; M = 38.07 years of age, SD = 7.16). Internal consistencies were satisfactory for all subscale scores of the instrument regardless of ethnic group membership. Mean and covariance structures analysis indicated that the hypothesized 3-factor structure of the instrument was not robust across ethnic groups. It provided a reasonable approximation to the data for Black and White women but not for Hispanic women. Tests for differential item functioning (DIF) were therefore conducted for only Black and White women. Analyses revealed no more than trivial instances of nonuniform DIF but more substantial evidence of uniform DIF for 3 of the 18 items. After having established partial strong factorial invariance of the instrument, latent factor means were found to be significantly higher for Black than for White women on all 3 subscales (somatization, depression, anxiety). In conclusion, the instrument may be used for mean comparisons between Black and White women.
In order to assess the degree of cognitive decline resulting from a pathological state, such as dementia, or from a normal aging process, it is necessary to know or to have a valid estimate of premorbid (or prior) cognitive ability. The National Adult Reading Test (NART; Nelson & Willison, 1991) and the Wechsler Test of Adult Reading (WTAR; Psychological Corporation, 2001) are 2 tests developed to estimate premorbid or prior ability. Due to the rarity of actual prior ability data, validation studies usually compare NART/WTAR performance with measures of current abilities in pathological and nonpathological groups. In this study, we validate the use of WTAR scores and extend the validation of the use of NART scores as estimates of prior ability, vis-à-vis the actual prior (childhood) cognitive ability. We do this in a large sample of healthy older people, the Lothian Birth Cohort 1936 (Deary, Gow, Pattie, & Starr, 2012; Deary et al., 2007). Both NART and WTAR scores were correlated with cognitive ability tested in childhood (r = .66-.68). Scores on both the NART and the WTAR had high stability over a period of 3 years in old age (r in excess of .90) and high interrater reliability. The NART accounted for more unique variance in childhood intelligence than did the WTAR. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
The middle of the 20th century was a very exciting time to be in the field of clinical psychology and, especially, assessment psychology. Seminal publications during that time clarified the then novel distinction between hypothetical constructs and intervening variables (MacCorquodale & Meehl, 1948), elucidated the concept of construct validity (Cronbach & Meehl, 1955), and described a methodology for estimating convergent and discriminant validity of measures of constructs and the role of variance due to common methods in the associations among measures of similar and different constructs (Campbell & Fiske, 1959). These articles have been the bedrock of clinical psychological and personality psychology assessment since that time. The year 2005 marks the 50th anniversary of Cronbach and Meehl's (1955) article "Construct Validity in Psychological Tests." A submission last year by Smith (2005b) struck me as an excellent time to honor the signal contributions of these psychologists of the last century through consideration particularly of the article by Cronbach and Meehl. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Since its introduction, the Rosenberg General Self-Esteem Scale (RGSE, Rosenberg, 1965) has been 1 of the most widely used measures of global self-esteem. We conducted 4 studies to investigate (a) the goodness-of-fit of a bifactor model positing a general self-esteem (GSE) factor and 2 specific factors grouping positive (MFP) and negative items (MFN) and (b) different kinds of validity of the GSE, MFN, and MFP factors of the RSGE. In the first study (n = 11,028), the fit of the bifactor model was compared with those of 9 alternative models proposed in literature for the RGSE. In Study 2 (n = 357), the external validities of GSE, MFP, and MFN were evaluated using objective grade point average data and multimethod measures of prosociality, aggression, and depression. In Study 3 (n = 565), the across-rater robustness of the bifactor model was evaluated. In Study 4, measurement invariance of the RGSE was further supported across samples in 3 European countries, Serbia (n = 1,010), Poland (n = 699), and Italy (n = 707), and in the United States (n = 1,192). All in all, psychometric findings corroborate the value and the robustness of the bifactor structure and its substantive interpretation. (PsycINFO Database Record (c) 2015 APA, all rights reserved).
We examined the psychometric properties of the Novaco Anger Scale-Provocation Inventory (NAS-PI, 1994 version) in Dutch violent forensic psychiatric patients and secondary vocational students. A confirmatory factor analysis of the subscale structure of the NAS was carried out, reliability was investigated, and relations were calculated between NAS-PI scores and other measures of personality traits and problem behaviors. The 3-subscale structure of the original NAS could not be confirmed. However, the internal consistency of the NAS and the PI was excellent, and the test-retest reliability of the NAS was good. The validity of the NAS and the PI was supported by a meaningful pattern of correlations with alternative measures of anger and personality traits. Forensic psychiatric outpatients displayed higher NAS scores than secondary vocational students, but inpatients scored even lower than this nonclinical control group. Our preliminary conclusion is that the NAS-PI is a valuable instrument for the assessment of anger in Dutch violent forensic psychiatric patients.
This article examined the incremental validity of the Ego Impairment Index (EII), a Rorschach measure of cognition, perception, and reasoning. R. M. Dawes (1999) borrowed 2 previously published data sets to assess the incremental validity of the EII. Dawes determined that in order for the EII to be considered a valid measure, the overall EII score should "outperform" 2 of the variables that compose the index. Using this approach, Dawes reported that the EII had "deficient" (p = .059) incremental validity. In this study, the incremental validity of the EII was re-examined using 1 of the data sets used by R. M. Dawes (1999). Applying the same strategy as Dawes, the author tested the incremental validity of the EII in predicting positive symptoms of schizophrenia, a more appropriate and clinically relevant criterion measure of impairment. The EII significantly predicted positive symptoms (p < .005) even after the 2 EII variables selected by Dawes and the measure of social competence were 1st entered into the equation. The results suggest that the EII adds meaningful information in the assessment of thought disturbance.
A predominant dimensional model of general personality structure is the five-factor model (FFM). Quite a number of alternative instruments have been developed to assess the domains of the FFM. The current study compares the validity of 2 alternative versions of the Shedler and Westen Assessment Procedure (SWAP-200) FFM scales, 1 that was developed on the basis of items identified by J. Shedler and D. Westen (2004) and 1 that used items identified by R. R. McCrae, C. E. Lokenhoff, and P. T. Costa (2005). The comparative validity of both measures was examined in a sample of persons who evidenced personality-related problems in living (N = 94). The McCrae et al. Neuroticism, Agreeableness, and Openness scales were superior to their respective Shedler and Westen scales. Both research teams developed comparably valid Extraversion and Conscientiousness scales. Implications of the results for future SWAP-200 research are discussed.
The relation between self- and peer-informant reports of personality using psychometric instruments has been the focus of considerable research. The quantified judgments of clinically experienced observers such as treating clinicians have also been studied. The focus of the present article is on the measurement of 3 personality disorders (borderline, antisocial, and obsessive-compulsive) using the Shedler-Westen Assessment Procedure (SWAP-200), an instrument designed to quantify personality ratings made by clinically experienced informants, and the self-report Personality Assessment Inventory (PAI). SWAP-200 personality disorder scales showed small to medium correlations with borderline and antisocial personality disorder scales from the PAI. As predicted, SWAP-200 obsessive-compulsive personality disorder correlated negatively with these scales, suggesting discriminant validity.
This reply responds to W. M. Grove's (2001) critique of H. O. F. Veiel and R. F. Koopman's (2001) article on bias in widely used methods of estimating premorbid IQ. In this reply, the authors show that Grove is misrepresenting part of Veiel and Koopman's arguments, extend them to show that the proposed adjustment to regression estimates of IQ not only is unbiased but also is the maximum-likelihood estimate of the true IQ, and argue that Grove's notion of the acceptability of biased methods in judicial proceedings reflects a fundamental misapprehension of their nature and purpose.
H. O. F. Veiel and R. F. Koopman (2001) advance statistical and legal theses. They correctly point out that the usual regression formula for estimating a pre-event IQ underestimates high IQs and overestimates low IQs (due to regression to the mean). They call this a conditional bias and show it can be sizeable. The author takes issue with their claim that a new estimator they propose should be used in place of the usual formulas, because it negates this statistical bias. Their argument against the usual estimator conflates statistical bias and legal bias. Their discussion in favor of their new estimator mentions, but does not derive a general formula for, a gross loss of precision entailed by use of the new estimator. The author quantifies this loss of precision and, using Veiel and Koopman's numerical example, shows that their estimator quadruples error.
There has been a recent controversy regarding the validity of the Rorschach test. This comment is in response to criticism levied by R. M. Dawes (2001) on the incremental validity of the Ego Impairment Index (EII), a Rorschach-derived measure of cognition, perception, and reasoning. The Dawes articles (1999 and 2001) serve as an example of the recent practice of placing extraordinary challenges on the Rorschach test. Dawes's arguments are examined and parallel examples are provided that demonstrate the bias used to judge the validity of the EII, the Rorschach, and psychological assessment. Still, in the face of criticism, the results (see Dawes, 2001) support the incremental validity of the EII. Thus, the conclusion presented in this Comment is that it is time for us to "call the whole thing off" and end the Rorschach controversy that has occupied so much recent attention and generated so few new ideas.
R. Gomez, G. L. Burns, J. A. Walsh, and M. A. de Moura (2003) examined the degree to which parent and teacher ratings of attention-deficit/hyperactivity disorder (ADHD) symptoms are accounted for by trait, source, and error variance. The importance and limitations of Gomez et al.'s findings are discussed in the context of clinical and research assessments of children suspected of having ADHD. Gomez et al.'s findings make clear that multimethod and multisource assessment protocols should be used in diagnosing children with this disorder. Further, clinicians and researchers must avoid relying too heavily on 1 source of data when evaluating the severity and frequency of ADHD symptoms.
G. J. DuPaul (2003) offered two suggestions for additional research to understand the strong source effects reported by R. Gomez, G. L. Burns, J. A. Walsh, and M. A. de Moura (2003) in attention-deficit/hyperactivity disorder (ADHD) rating scales. The first suggestion was to determine whether the source effects represent mostly bias or accuracy. The second suggestion was to minimize source effects through the development of better ADHD rating scales. Because source effects can represent bias or accuracy, it is important to minimize the bias aspect through content validation procedures prior to attempts to determine whether source effects better reflect bias or accuracy. This comment offers various suggestions to reduce the bias in ADHD rating scales.
In 4 empirical studies, E. Stice, M. Fisher, and M. R. Lowe calculated the correlations between some widely used dietary restraint scales and food intake. Failing to find substantial negative correlations, they concluded that these scales were invalid. The current article challenges this conclusion. For one thing, there is some evidence that restrained eaters do eat less than do unrestrained eaters under controlled experimental conditions favoring self-control. Dietary restraint is also associated with tendencies toward disinhibition under conditions favoring loss of self-control; such disinhibition often masks (but does not invalidate) the construct of dietary restraint. For these and other reasons, the assessment of food intake at a single eating episode may not capture overall dietary restriction. Finally, how much one eats does not necessarily indicate whether one has eaten less than one desired to eat. The authors suggest that the existing restraint scales do in fact validly assess restriction of food intake, albeit in a more complex fashion than is evident from simple correlations in single episodes.
On the basis of taxometric analyses of data sets that they created to pose interpretive challenges, S. R. H. Beach, N. Amir, and J. J. Bau (2005) cautioned that using comparison data simulated by J. Ruscio's programs can lead to inaccurate conclusions. Careful examination of S. R. H. Beach et al.'s methods and results plus reanalysis of their data fails to substantiate this concern: Using comparison data identified the taxonic structure of S. R. H. Beach et al.'s data sets, even when the taxon base rate was very low. The authors show that J. Ruscio's simulation programs generate comparison data appropriately and that analyzing these data provides a useful interpretive aid. Additionally, the authors discuss and illustrate the effective use of the inchworm consistency test to disambiguate taxometric results for small taxa and dimensional constructs with positively skewed indicators.
Descriptions of Three Higher Cognitive Processes of Foresight, Judgment, and Insight That Regulate the Emotional Drives Relevant to Each of the Five Layers of Human Personality 
The revised Temperament and Character Inventory (TCI-R) is the third stage of development of a widely used multiscale personality inventory that began with the Tridimensional Personality Questionnaire (TPQ) and then the Temperament and Character Inventory (TCI). The author describes the third stage of the psychobiological theory of temperament and character; empirical tests of its predictions from genetics, neurobiology, psychosocial development, and clinical studies; and empirical findings that stimulated incremental changes in theory and test construction. Linear factor analysis is an inadequate method for evaluating the nonlinear and dynamical nature of the intrapsychic processes that influence human personality. Traits derived by factor analysis under the doubtful assumption of linearity are actually heterogeneous composites of rational and emotional processes that differ fundamentally in their underlying brain processes. The predictions of the psychobiological theory are strongly validated by extensive data from genetics, neurobiology, longitudinal studies of development, and clinical assessment. The distinction between temperament and character allows the TCI and TCI-R to outperform other popular personality inventories in distinguishing individuals with personality disorders from others and in describing the developmental path to well-being in terms of dynamical processes within the individual that are useful for both research and clinical practice.
Reports an error in "The MMPI-2 Restructured Clinical Scales in the assessment of posttraumatic stress disorder and comorbid disorders" by Erika J. Wolf, Mark W. Miller, Robert J. Orazem, Mariann R. Weierich, Diane T. Castillo, Jaime Milford, Danny G. Kaloupek and Terence M. Keane (Psychological Assessment, 2008[Dec], Vol 20[4], 327-340). The URL for the supplemental material was incomplete. The complete URL is (The following abstract of the original article appeared in record 2008-17693-002.) This study examined the psychometric properties of the Minnesota Multiphasic Personality Inventory-2 (MMPI-2) Restructured Clinical Scales (RCSs) in individuals with posttraumatic stress disorder (PTSD) receiving clinical services at Department of Veterans Affairs medical centers. Study 1 included 1,098 men who completed the MMPI-2 and were assessed for a range of psychological disorders via structured clinical interview. Study 2 included 136 women who completed the MMPI-2 and were interviewed with the Clinician Administered Scale for PTSD. The utility of the RCSs was compared with that of the Clinical Scales (CSs) and the Keane PTSD (PK) scale. The RCSs demonstrated good psychometric properties and patterns of associations with other measures of psychopathology that corresponded to current theory regarding the structure of comorbidity. A notable advantage of the RCSs compared with the MMPI-2 CSs was their enhanced construct validity and clinical utility in the assessment of comorbid internalizing and externalizing psychopathology. The PK scale demonstrated incremental validity in the prediction of PTSD beyond that of the RCSs or CSs. (PsycINFO Database Record (c) 2009 APA, all rights reserved).
In this reply the authors address comments by C. R. Cloninger (2008) related to their report (R. F. Farmer & L. R. Goldberg, 2008) on the psychometric properties of the revised Temperament and Character Inventory (TCI-R) and a short inventory derivative, the TCI-140. Even though Cloninger's psychobiological model has undergone substantial theoretical modifications, the relevance of these changes for the evaluation and use of the TCI-R remains unclear. Aspects of TCI-R assessment also appear to be theoretically and empirically incongruent with Cloninger's assertion that TCI-R personality domains are nonlinear and dynamic in nature. Several other core assumptions from the psychobiological model, including this most recent iteration, are nonfalsifiable, inconsistently supported, or have no apparent empirical basis. Although researchers using the TCI and TCI-R have frequently accepted the temperament-character distinction and associated theoretical ramifications, for example, the authors found little overall support for the differentiation of TCI-R domains into these 2 basic categories. The implications of these observations for TCI-R assessment are briefly discussed. (PsycINFO Database Record (c) 2008 APA, all rights reserved).
Reports an error in "Diagnostic validity of the Eppendorf Schizophrenia Inventory (ESI): A self-report screen for ultrahigh risk and acute psychosis" by Maurice A. J. Niessen, Peter M. A. J. Dingemans, Reinaud van de Fliert, Hiske E. Becker, Dorien H. Nieman and Don Linszen (Psychological Assessment, 2010[Dec], Vol 22[4], 935-944). In the first full paragraph, the references to the numbers in Table 5 are incorrect in the sentence that begins "We then proceeded with calculating accuracy measures...". A corrected version of the sentence is presented in the erratum. Also presented in the erratum are corrections to variables mentioned elsewhere in the article. The aforementioned changes are very small and do not in any way affect the findings of the research. (The following abstract of the original article appeared in record 2010-24850-009.) Providers of mental health services need tools to screen for acute psychosis and ultrahigh risk (UHR) for transition to psychosis in help-seeking individuals. In this study, the Eppendorf Schizophrenia Inventory (ESI) was examined as a screening tool and for its ability to correctly predict diagnostic group membership (e.g., help seeking, mild psychiatric complaints, highly symptomatic mood or anxiety disorder, UHR, acute psychosis). Diagnostic evaluation with established instruments was used for diagnosis in 3 research samples. UHR status was assessed with the Structured Interview for Prodromal Symptoms/Scale of Prodromal Symptoms (Miller et al., 1999) and the Bonn Scale for the Assessment of Basic Symptoms Prediction list (Gross, Huber, Klosterkötter, & Linz, 1987; Klosterkötter, Hellmich, Steinmeyer, & Schulze-Lutter, 2001). This study showed that members of different diagnostic groups rate themselves significantly differently on the ESI and its subscales. A new subscale was constructed, the UHR-Psychosis scale, that showed good utility in detecting individuals with interview-diagnosed UHR status and acute psychosis. The scale is also sensitive to the threshold between UHR and acute psychosis. Practical applications of the ESI include use as a diagnostic tool within various settings. (PsycINFO Database Record (c) 2011 APA, all rights reserved).
In a recent article in this journal, Poston and Hanson (2010) reported a meta-analysis of 17 studies on the use of psychological assessment as a therapeutic intervention (PATI) and concluded that "psychological assessment procedures--when combined with personalized, collaborative, and highly involving test feedback--have positive, clinically meaningful effects on treatment" (Poston & Hanson, 2010, p. 203). Although extant data suggest that PATI can sometimes exert positive effects, Poston and Hanson's (2010) meta-analysis may overstate the magnitude of these effects because the authors (a) included several studies that combined assessment with treatment components that are irrelevant to PATI, sometimes rendering it impossible to attribute any observed effects to PATI per se and (b) excluded numerous nonsignificant results. Moreover, the studies Poston and Hanson (2010) reviewed neglected to rule out Barnum effects as alternative explanations for client improvement, raising the possibility that PATI works for reasons other than those proposed by its advocates. We conclude that Poston and Hanson's (2010) review leaves a number of lingering questions concerning the treatment utility of PATI unanswered.
In our article by J. L. Skeem & D. J. Cooke, (2010), we outlined the dangers inherent in conflating the Psychopathy Checklist-Revised (PCL-R; R. Hare, 1991) with psychopathy itself. In their response, R. Hare and C. Neumann (2010) seemed to agree with key points that the PCL-R should not be confused with psychopathy and that criminal behavior is not central to psychopathy; at the same time, they said we provided no clear directions for theory or research. In this rejoinder, we clarify our argument that progress in understanding the unobservable construct of psychopathy hinges upon setting aside procrustean dependence on a monofocal PCL-R lens to test (a) actual theories of psychopathy against articulated validation hierarchies and (b) the relation between psychopathy and crime. In specifying these conceptual and applied directions, we hope to promote constructive dialogue, further insights, and a new generation of research that better distinguishes between personality deviation and social deviance.
Reports an error in "Psychometric properties and U.S. National norms of the Evidence-Based Practice Attitude Scale (EBPAS)" by Gregory A. Aarons, Charles Glisson, Kimberly Hoagwood, Kelly Kelleher, John Landsverk and Guy Cafri (Psychological Assessment, 2010[Jun], Vol 22[2], 356-365). There were three errors in Table 1 on p. 360. In the last row, the row label should be "Overall EBPAS mean," M = 2.73, and SD = 0.49. The revised Table 1 appears in the erratum. (The following abstract of the original article appeared in record 2010-10892-016.) The Evidence-Based Practice Attitude Scale (EBPAS) assesses mental health and social service provider attitudes toward adopting evidence-based practices. Scores on the EBPAS derive from 4 subscales (i.e., Appeal, Requirements, Openness, and Divergence) as well as the total scale, and preliminary studies have linked EBPAS scores to clinic structure and policies, organizational culture and climate, and first-level leadership. EBPAS scores are also related to service provider characteristics, including age, education level, and level of professional development. The present study examined the factor structure, reliability, and norms of EBPAS scores in a sample of 1,089 mental health service providers from a nationwide sample drawn from 100 service institutions in 26 states in the United States. The study also examined associations of provider demographic characteristics with EBPAS subscale and total scores. Confirmatory factor analysis supported a second-order factor model, and reliability coefficients for the subscales ranged from .91 to .67 (total scale = .74). The study establishes national norms for the EBPAS so that comparisons can be drawn for U.S. local as well as international studies of attitudes toward evidence-based practices. The results suggest that the factor structure and reliability are likely generalizable to a variety of service provider contexts and different service settings and that the EBPAS subscales are associated with provider characteristics. Directions for future research are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved).
J. Skeem and D. J. Cooke (2010) asserted that Hare and Neumann consider criminality to be an essential component of the psychopathy construct. The assertion, presented in the guise of a debate on the nature of psychopathy, is neither accurate nor consistent with the clinical and empirical literature on psychopathy to which Hare and Neumann have contributed. Broadly defined antisociality, not criminality per se, is considered to be part of the psychopathy construct. Skeem and Cooke also expressed concerns that the popularity of the Psychopathy Checklist-Revised (R. D. Hare, 2003) inhibits the development and use of other instruments, that it has become the construct it measures, that it deviates from its clinical roots, and that it conflates criminality with personality. These and related issues are addressed, and it is suggested that the arguments proffered by Skeem and Cooke are not convincing, nor do they provide clear directions for theory and research.
The Buddhist construct of mindfulness is a central element of mindfulness-based interventions and derives from an age-old systematic phenomenological program to investigate subjective experience. Recent enthusiasm for "mindfulness" in psychology has resulted in proliferation of self-report inventories that purport to measure mindful awareness as a trait. This paper addresses a number of intractable issues regarding these scales, in general, and also specifically highlights vulnerabilities of the adult and adolescent forms of the Mindfulness Attention Awareness Scale. These problems include (a) lack of available external referents for determining the construct validity of these inventories, (b) inadequacy of content validity of measures, (c) lack of evidence that self-reports of mindfulness competencies correspond to actual behavior and evidence that they do not, (d) lack of convergent validity among different mindfulness scales, (e) inequivalence of semantic item interpretation among different groups, (f) response biases related to degree of experience with mindfulness practice, (g) conflation of perceived mindfulness competencies with valuations of importance or meaningfulness, and (h) inappropriateness of samples employed to validate questionnaires. Current self-report attempts to measure mindfulness may serve to denature, distort, and banalize the meaning of mindful awareness in psychological research and may adversely affect further development of mindfulness-based interventions. Opportunities to enrich positivist Western psychological paradigms with a detailed and complex Buddhist phenomenology of the mind are likely to require a depth of understanding of mindfulness that, in turn, depends upon direct and long-term experience with mindfulness practice. Psychologists should consider pursuing this avenue before attempting to characterize and quantify mindfulness.
Reports an error in "Correspondence and disparity in the self- and other ratings of current and childhood ADHD symptoms and impairment in adults with ADHD" by Russell A. Barkley, Laura E. Knouse and Kevin R. Murphy (Psychological Assessment, , , np). There was an omission in the author note. The author note should have included a disclosure as follows, "Russell A. Barkley receives royalties for books, videos, and rating scales from Guilford Publications, and is the author of Barkley Adult ADHD Rating Scale-IV (BAARS-IV), Barkley Deficits in Executive Functioning Scale (BDEFS), and Barkley Functional Impairment Scale (BFIS), all published by Guilford Press." (The following abstract of the original article appeared in record 2011-04636-001.) Experts recommend that clinicians evaluating adults for attention-deficit/hyperactivity disorder (ADHD) obtain information from others who know the patient well. The authors examined correspondence between the self- and other-ratings of ADHD symptoms and impairment using 3 groups of adults recruited on the basis of their severity of ADHD: ADHD diagnosis (n = 146), clinical controls self-referring for ADHD but not diagnosed (n = 97), and community controls (n = 109). The influences of diagnostic group, informant relationship, sex of participant, IQ, and comorbid anxiety and depression on self-informant disparities were also examined. Results indicated moderate to high agreement (.59-.80) between self and others on current functioning and slightly lower levels (.53-.75) between self- and parent ratings of childhood functioning. Examination of difference scores between self- and other ratings revealed small mean disparities (-0.1 to +5.0 points) but substantial variation (SDs = -2.4 to 8.9 points) for both current and childhood ratings. Clinic referrals not diagnosed with ADHD, particularly women, had higher disparities than was evident in the ADHD and community groups. Age, IQ, and education were not associated with disparities in most ratings. Higher anxiety, in contrast, was associated with greater disparities on all current and childhood measures of both ADHD and impairment. (PsycINFO Database Record (c) 2011 APA, all rights reserved).
Lilienfeld, Garb, and Wood (2011) commented on our recently published meta-analysis (Poston & Hanson, 2010). In their thorough, fine-grained critique, they questioned the study's inclusion criteria, sampling, and results, suggesting the latter are "overstated" (p. 1048). Additionally, they discussed the Barnum effect, suggesting that it accounts, at least partially, for the potential efficacy and/or effectiveness of psychological assessment as a therapeutic intervention (PATI). In addressing Lilienfeld et al.'s (2011) concerns, we incorporate their suggestions and present new findings based on the original data set, including 15 independent and 56 dependent effect sizes (ESs). Upon reanalysis, a comparable, significant overall Cohen's d ES of 0.403 (95% CI [0.302, 0.503]) was found, thus building confidence in our original results, as well as the overall efficacy of PATI. Given PATI's potential treatment utility, we mention possible change mechanisms, above and beyond Barnum effects. And finally, in hopes of launching the field forward, we offer suggestions for future research and collaboration. (PsycINFO Database Record (c) 2011 APA, all rights reserved).
Reports an error in "A psychometric analysis of the Positive and Negative Affect Schedule for Children-Parent Version in a school sample" by Chad Ebesutani, Kelsie Okamura, Charmaine Higa-McMillan and Bruce F. Chorpita (Psychological Assessment, 2011[Jun], Vol 23[2], 406-416). There was an error in the Appendix caption (pg. 416). The corrected caption is provided in the erratum. (The following abstract of the original article appeared in record 2011-04634-001.) The current study was the 1st to examine the psychometric properties of the Positive and Negative Affect Schedule for Children-Parent Version (PANAS-C-P) using a large school-based sample of children and adolescents ages 8 to 18 (N = 606). Confirmatory factor analysis supported a 2-factor (correlated) model of positive affect (PA) and negative affect (NA). The PANAS-C-P scale scores also demonstrated acceptable internal consistency and convergent and divergent validity. The PANAS-C-P PA and NA scale scores also related to measures of anxiety and depression in a manner consistent with the tripartite model. Scale means and standard deviations were reported by grade and sex to provide normative data for the PANAS-C-P scales. Results from the present study provide initial support for the PANAS-C-P as a parent-reported perspective of youth PA and NA among school-based youths. (PsycINFO Database Record (c) 2011 APA, all rights reserved).
We address 3 critiques raised by Grossman (2011) of self-report measures of mindfulness and the Mindful Attention Awareness Scale (MAAS) and Mindful Attention Awareness Scale-Adolescent (MAAS-A) in particular. Grossman questioned whether self-report measures actually assess mindfulness, whether the construct of mindfulness can be understood apart from mindfulness training, and whether there is empirical evidence to support the validity of mindfulness measures. In response we discuss established theory that attention (and secondarily meta-awareness) is core to the meaning of mindfulness and is the central feature of the MAAS and MAAS-A. We then argue that mindfulness is an inherent capacity that varies between and within persons and is not, as Grossman claimed, a concept applicable to only a trained few. Further, as assessed by the MAAS and MAAS-A, mindfulness is associated with the same variety of outcomes as mindfulness training is theorized to yield. Finally, we provide considerable evidence that the MAAS and MAAS-A are valid instruments. We conclude that although construct measurement is inevitably imperfect, such efforts are critical to building basic knowledge and refining effective interventions. (PsycINFO Database Record (c) 2011 APA, all rights reserved).
Top-cited authors
Joshua D Miller
  • University of Georgia
Matthew W Gallagher
  • University of Houston
Terence M Keane
  • VA National Center for PTSD & Boston University
Brian P Marx
  • U.S. Department of Veterans Affairs
Herb Marsh
  • Australian Catholic University