The Predictive Validity of Three Versions of the MCAT in Relation to Performance in Medical School, Residency, and Licensing Examinations: A Longitudinal Study of 36 Classes of Jefferson Medical College
ABSTRACT The Medical College Admission Test (MCAT) has undergone several revisions for content and validity since its inception. With another comprehensive review pending, this study examines changes in the predictive validity of the MCAT's three recent versions.
Study participants were 7,859 matriculants in 36 classes entering Jefferson Medical College between 1970 and 2005; 1,728 took the pre-1978 version of the MCAT; 3,032 took the 1978-1991 version, and 3,099 took the post-1991 version. MCAT subtest scores were the predictors, and performance in medical school, attrition, scores on the medical licensing examinations, and ratings of clinical competence in the first year of residency were the criterion measures.
No significant improvement in validity coefficients was observed for performance in medical school or residency. Validity coefficients for all three versions of the MCAT in predicting Part I/Step 1 remained stable (in the mid-0.40s, P < .01). A systematic decline was observed in the validity coefficients of the MCAT versions in predicting Part II/Step 2. It started at 0.47 for the pre-1978 version, decreased to between 0.42 and 0.40 for the 1978-1991 versions, and to 0.37 for the post-1991 version. Validity coefficients for the MCAT versions in predicting Part III/Step 3 remained near 0.30. These were generally larger for women than men.
Although the findings support the short- and long-term predictive validity of the MCAT, opportunities to strengthen it remain. Subsequent revisions should increase the test's ability to predict performance on United States Medical Licensing Examination Step 2 and must minimize the differential validity for gender.
Full-textDOI: · Available from: Mohammadreza Hojat, Sep 26, 2015
- SourceAvailable from: Jun Yang
[Show abstract] [Hide abstract]
- "Another US study also indicated that scores on the admission test for medical students (i.e., the MCAT) had a positive relationship with GPA in the first two years . One recent analysis has shown that scores on all three versions of the MCAT were moderately correlated to performance measures in medical school in the US . In China, a few previous studies explored the association between NCEE total scores and academic performance in medical school, but none of them observed a significant association, which was most likely due to the use of improper statistical methods [9-11]. "
ABSTRACT: Although medical education has developed rapidly in the last decade, and the National College Entrance Examination (NCEE) is used as the "gold standard" for admission to medical college in mainland China, there is a lack of literature regarding the influence of NCEE score and other factors on the academic performance of medical students. This study aimed to examine potential predictors of first-year grade point average (GPA) for medical students. This study included 1,285 students who matriculated at a first-tier medical university in mainland China in 2011. The precollege motivational attitudes for each matriculate were investigated via questionnaire. A hierarchical linear model was fitted to regress first-year GPA on a 100-point scale on NCEE score and other student-level and major-level characteristics. NCEE score was a significant predictor of both within-major and between-major variation of first-year GPA for medical students. Majors with higher mean NCEE scores had higher mean GPAs, and higher GPAs were observed among those individuals with higher NCEE scores after controlling for major-level characteristics. First-year GPA differed by certain individual socio-demographic variables. Female students had a 2.44-higher GPA on average than did male students. NCEE repeaters had a 1.55-lower GPA than non-repeaters. First-year GPA was associated negatively with parental income but positively with academic self-concept. NCEE score is an important predictor of the first-year GPA of medical students, but it is not the sole determinant. Individual socio-demographic characteristics and major-level characteristics should be taken into account to understand better and improve the first-year GPA of medical students.BMC Medical Education 04/2014; 14(1):87. DOI:10.1186/1472-6920-14-87 · 1.22 Impact Factor
[Show abstract] [Hide abstract]
- "We found supportive evidence for the assertion that the UKCAT is better at predicting performance in the later years than in the earlier years of medical school. This is in contrast to the pattern of decline observed in studies involving the MCAT [12,14] and with generally larger validity coefficients than reported by Yates and James . "
ABSTRACT: The UK Clinical Aptitude Test (UKCAT) was designed to address issues identified with traditional methods of selection. This study aims to examine the predictive validity of the UKCAT and compare this to traditional selection methods in the senior years of medical school. This was a follow-up study of two cohorts of students from two medical schools who had previously taken part in a study examining the predictive validity of the UKCAT in first year. The sample consisted of 4th and 5th Year students who commenced their studies at the University of Aberdeen or University of Dundee medical schools in 2007. Data collected were: demographics (gender and age group), UKCAT scores; Universities and Colleges Admissions Service (UCAS) form scores; admission interview scores; Year 4 and 5 degree examination scores. Pearson's correlations were used to examine the relationships between admissions variables, examination scores, gender and age group, and to select variables for multiple linear regression analysis to predict examination scores. Ninety-nine and 89 students at Aberdeen medical school from Years 4 and 5 respectively, and 51 Year 4 students in Dundee, were included in the analysis. Neither UCAS form nor interview scores were statistically significant predictors of examination performance. Conversely, the UKCAT yielded statistically significant validity coefficients between .24 and .36 in four of five assessments investigated. Multiple regression analysis showed the UKCAT made a statistically significant unique contribution to variance in examination performance in the senior years. Results suggest the UKCAT appears to predict performance better in the later years of medical school compared to earlier years and provides modest supportive evidence for the UKCAT's role in student selection within these institutions. Further research is needed to assess the predictive validity of the UKCAT against professional and behavioural outcomes as the cohort commences working life.BMC Medical Education 04/2014; 14(1):88. DOI:10.1186/1472-6920-14-88 · 1.22 Impact Factor
[Show abstract] [Hide abstract]
- "They are felt to be measures of potential and to be independent of formal schooling, and in many ways can be regarded as overlapping with measures of basic mental ability or intelligence. Tests such as the Medical College Admission Test (MCAT), used to select medical students in the USA , measure substantive academic understanding of a range of material from biology, chemistry, and physics, and are therefore primarily measures of attainment rather than of aptitude. "
ABSTRACT: Selection of medical students in the UK is still largely based on prior academic achievement, although doubts have been expressed as to whether performance in earlier life is predictive of outcomes later in medical school or post-graduate education. This study analyses data from five longitudinal studies of UK medical students and doctors from the early 1970s until the early 2000s. Two of the studies used the AH5, a group test of general intelligence (that is, intellectual aptitude). Sex and ethnic differences were also analyzed in light of the changing demographics of medical students over the past decades. Data from five cohort studies were available: the Westminster Study (began clinical studies from 1975 to 1982), the 1980, 1985, and 1990 cohort studies (entered medical school in 1981, 1986, and 1991), and the University College London Medical School (UCLMS) Cohort Study (entered clinical studies in 2005 and 2006). Different studies had different outcome measures, but most had performance on basic medical sciences and clinical examinations at medical school, performance in Membership of the Royal Colleges of Physicians (MRCP(UK)) examinations, and being on the General Medical Council Specialist Register. Correlation matrices and path analyses are presented. There were robust correlations across different years at medical school, and medical school performance also predicted MRCP(UK) performance and being on the GMC Specialist Register. A-levels correlated somewhat less with undergraduate and post-graduate performance, but there was restriction of range in entrants. General Certificate of Secondary Education (GCSE)/O-level results also predicted undergraduate and post-graduate outcomes, but less so than did A-level results, but there may be incremental validity for clinical and post-graduate performance. The AH5 had some significant correlations with outcome, but they were inconsistent. Sex and ethnicity also had predictive effects on measures of educational attainment, undergraduate, and post-graduate performance. Women performed better in assessments but were less likely to be on the Specialist Register. Non-white participants generally underperformed in undergraduate and post-graduate assessments, but were equally likely to be on the Specialist Register. There was a suggestion of smaller ethnicity effects in earlier studies. The existence of the Academic Backbone concept is strongly supported, with attainment at secondary school predicting performance in undergraduate and post-graduate medical assessments, and the effects spanning many years. The Academic Backbone is conceptualized in terms of the development of more sophisticated underlying structures of knowledge ('cognitive capital' and 'medical capital'). The Academic Backbone provides strong support for using measures of educational attainment, particularly A-levels, in student selection.BMC Medicine 11/2013; 11(1):242. DOI:10.1186/1741-7015-11-242 · 7.25 Impact Factor