The Electronic Residency Application Service Application Can Predict Accreditation Council for Graduate Medical Education Competency-Based Surgical Resident Performance
ABSTRACT Program directors often struggle to determine which factors in the Electronic Residency Application Service (ERAS) application are important in the residency selection process. With the establishment of the Accreditation Council for Graduate Medical Education (ACGME) competencies, it would be important to know whether information available in the ERAS application can predict subsequent competency-based performance of general surgery residents.
This study is a retrospective correlation of data points found in the ERAS application with core competency-based clinical rotation evaluations. ACGME competency-based evaluations as well as technical skills assessment from all rotations during residency were collected. The overall competency score was defined as an average of all 6 competencies and technical skills.
A total of77 residents from two (one university and one community based university-affiliate) general surgery residency programs were included in the analysis. Receiving honors for many of the third year clerkships and AOA membership were associated with a number of the individual competencies. USMLE scores were predictive only of Medical Knowledge (p = 0.004). Factors associated with higher overall competency were female gender (p = 0.02), AOA (p = 0.06), overall number of honors received (p = 0.04), and honors in Ob/Gyn (p = 0.03) and Pediatrics (p = 0.05). Multivariable analysis showed honors in Ob/Gyn, female gender, older age, and total number of honors to be predictive of a number of individual core competencies. USMLE scores were only predictive of Medical Knowledge.
The ERAS application is useful for predicting subsequent competency based performance in surgical residents. Receiving honors in the surgery clerkship, which has traditionally carried weight when evaluating a potential surgery resident, may not be as strong a predictor of future success.
- [Show abstract] [Hide abstract]
ABSTRACT: Selection of surgical residents is a difficult task, and program directors are interested in identifying the best candidates. Among the qualities being sought after is the ability to acquire surgical knowledge, and eventually do well on their board examinations. During the interview process, many programs use results from the United States Medical Licensing Exam (USMLE) to identify residents they think will do well academically. The purpose of this study was to evaluate a different method of identifying such residents, through the use of a surgery-specific written exam (SSWE). A retrospective review of residents in our program between 2004 and 2012 was done. A 50-question SSWE was designed and administered to candidates on the day of their interview. Scores on the SSWE and the USMLE were compared with results on the American Board of Surgery In-Training Exam (ABSITE). Correlation coefficients were calculated and compared. Community based General Surgery residency program. Resident applicants. Forty-three residents had scores available from the SSWE, USMLE Part 1 (USMLE-1), and Part 2 (USMLE-2). There were ABSITE scores available for 38 in postgraduate year (PGY) 1. USMLE-1 had a statistically significant correlation (r = 0.327, p = 0.045) with the ABSITE score in PGY-1 (ABSITE-1), while with USMLE-2 had slightly less correlation (r = 0.314, p = 0.055) with ABSITE-1. However, the SSWE had a much stronger correlation (r = 0.656, p < 0.001) than either of them. An SSWE is a good method to identify residents who will later do well on the ABSITE. It is a better method than using the more general USMLE. Since the ABSITE has been shown to correlate with performance on board examinations, residency programs interested in identifying candidates that will do well on their board examinations, should consider incorporating an SSWE into their application process.Journal of Surgical Education 11/2012; 69(6):807-12. DOI:10.1016/j.jsurg.2012.05.011 · 1.39 Impact Factor
- [Show abstract] [Hide abstract]
ABSTRACT: Objective For the past 15 years at our institution’s general surgery residency program, 3 of the senior residents have been chosen to be awarded either (1) Best Resident in Research, (2) Best Resident in Teaching, or (3) Best Resident Overall. Considering that these awards serve as data representing outstanding performance as surgical residents, the objective of this study was to determine the association between receiving one of these awards and objective measures of performance. Methods Individual files were reviewed for the 103 residents who graduated from our institution’s general surgery program from 1994 to 2010. These data were studied as a whole, and then divided into an award-winning group and a non–award winning group and subsequently compared across several objective parameters, including The United States Medical Licensing Examination (USMLE) scores, American Board of Surgery In-Training Examination (ABSITE) scores, first-time American Board of Surgery Certifying and Qualifying Examination pass rates, Alpha Omega Alpha membership status, and number of research years, using a logistic regression model. Results Overall, 103 residents completed their general surgery residency training at our institution from 1994 to 2010, and of these residents, 16 (16%) received the Best Resident in Research award, 15 (16%) received the Best Resident in Teaching award, and 17 (17%) received the Best Resident Overall award in their final years of training. Compared with those who did not receive an award, a hypothesis-based one-tailed test revealed that award winners had a significantly lower median USMLE Step 1 scores (p = 0.04) and marginally lower median USMLE Step 2 scores (p = 0.05). Alpha Omega Alpha membership status, median ABSITE percent correct overall, first-time American Board of Surgery examination pass rates, and number of research years during residency were not significantly different between the 2 groups. Conclusion Many factors contribute to success during general surgery residency. Our study showed that higher USMLE and ABSITE scores were not associated with receiving top awards in final years of training at one institution over 15 years.Journal of Surgical Education 01/2013; 71(2). DOI:10.1016/j.jsurg.2013.07.012 · 1.39 Impact Factor
- [Show abstract] [Hide abstract]
ABSTRACT: The purpose of this study was to use meta-analysis to establish which of the information available to the resident selection committee is associated with resident or doctor performance. Multiple electronic databases were searched to 4 September 2012. Two reviewers independently selected studies that met the present inclusion criteria and extracted data in duplicate; disagreement was resolved by consensus. Risk for bias was assessed using a customised bias assessment tool. Measures of association were converted to a common effect size (Hedges' g). Meta-analysis was performed using the random-effects model for each selection strategy and all outcomes without pooling. Sensitivity analysis for each selection strategy-outcome pair was performed with pooling of effect size. Eighty studies involving a total of 41 704 participants were included in the meta-analysis. Seventeen different selection strategies and 17 outcomes were assessed across these studies. The strongest positive associations referred to examination-based selection strategies, such as the US Medical Licensing Examination (USMLE) Step 1, and examination-based outcomes, such as scores on in-training examinations. Moderate positive associations were present for medical school marks and both examination-based and subjective outcomes. Minimal or no associations were seen for the selection tools represented by interviews, reference letters and deans' letters. Standardised examination performance and medical school grades show the strongest associations with current measures of doctor performance. Deans' letters, reference letters and interviews all show a lower than expected strength of association given the relative value often assigned to them during resident doctor selection. Objective selection strategies are potentially the most useful to residency selection committees based on current evaluative methods. However, reports in the literature of validated long-term doctor performance outcomes are scant.Medical Education 08/2013; 47(8):790-800. DOI:10.1111/medu.12234 · 3.62 Impact Factor