Article
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Purpose: The existing methods for evaluating resident operative performance interrupt the workflow of the attending physician, are resource intensive, and are often completed well after the end of the procedure in question. These limitations lead to low faculty compliance and potential significant recall bias. In this study, we deployed a smartphone-based system, the Procedural Autonomy and Supervisions System, to facilitate assessment of resident performance according to the Zwisch scale with minimal workflow disruption. We aimed to demonstrate that this is a reliable, valid, and feasible method of measuring resident operative autonomy. Methods: Before implementation, general surgery residents and faculty underwent frame-of-reference training to the Zwisch scale. Immediately after any operation in which a resident participated, the system automatically sent a text message prompting the attending physician to rate the resident's level of operative autonomy according to the 4-level Zwisch scale. Of these procedures, 8 were videotaped and independently rated by 2 additional surgeons. The Zwisch ratings of the 3 raters were compared using an intraclass correlation coefficient. Videotaped procedures were also scored using 2 alternative operating room (OR) performance assessment instruments (Operative Performance Rating System and Ottawa Surgical Competency OR Evaluation), against which the item correlations were calculated. Results: Between December 2012 and June 2013, 27 faculty used the smartphone system to complete 1490 operative performance assessments on 31 residents. During this period, faculty completed evaluations for 92% of all operations performed with general surgery residents. The Zwisch scores were shown to correlate with postgraduate year (PGY) levels based on sequential pairwise chi-squared tests: PGY 1 vs PGY 2 (χ(2) = 106.9, df = 3, p < 0.001); PGY 2 vs PGY 3 (χ(2) = 22.2, df = 3, p < 0.001); and PGY 3 vs PGY 4 (χ(2) = 56.4, df = 3, p < 0.001). Comparison of PGY 4 to PGY 5 scores were not significantly different (χ(2) = 4.5, df = 3, p = 0.21). For the 8 operations reviewed for interrater reliability, the intraclass correlation coefficient was 0.90 (95% CI: 0.72-0.98, p < 0.01). Correlation of Procedural Autonomy and Supervisions System ratings with both Operative Performance Rating System items (each r > 0.90, all p's < 0.01) and Ottawa Surgical Competency OR Evaluation items (each r > 0.86, all p's < 0.01) was high. Conclusions: The Zwisch scale can be used to make reliable and valid measurements of faculty guidance and resident autonomy. Our data also suggest that Zwisch ratings may be used to infer resident operative performance. Deployed on an automated smartphone-based system, it can be used to feasibly record evaluations for most operations performed by residents. This information can be used to council individual residents, modify programmatic curricula, and potentially inform national training guidelines.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... These include the Zwisch scale, OSATS, and Ottawa scoring. . 6,7,9,10 Advantages of evaluation tools include that they can be stored for an extended period of time, can be performed for each case, provide insight into the level of assistance necessary for residents intraoperatively, and provide a standardized method for evaluating all residents which can later be translated to assess overall operative competency. Our institution utilizes the System for Improving and Measuring Procedural Learning (SIMPL) application, which is based on the Zwisch scale. ...
... Validity was not the focus of this article; however, both the Zwisch scale and SIMPL app have previously been the attention of trials which illustrate their practicality among other specialties. 9,11 In this study, we utilize our novel departmental mobile application (Wayne State University Surgery Department application) to grade intraoperative competence of plastic surgery residents. The WSU Surgery application has been customized to allow evaluations through the SIMPL application, based upon the Zwisch scale. ...
... The Zwisch scale was chosen for this study because it has previously been evaluated as both a feasible and reliable tool for intraoperative assessment. 9,11 At the start of the study, a 1-hour frame of reference training was provided to application users on how to use the smartphone application and the Zwisch grading system. This training session provides education about the application but also has been illustrated to ensure rater reliability. ...
Article
Full-text available
Background: The evaluation model of operative competence is based on aggregate tabulations of procedures and end of rotation feedback from faculty members. Procedural tabulations do not detail the level of resident involvement in the case, and end of rotation feedback is infrequent and inaccurate due to the necessity of long-term recall. Smart phone-based evaluation systems provide residents with immediate and permanent feedback for surgical encounters. In this study we examine feasibility of smart phone-based evaluations in plastic surgery residency. Methods: This was a 6-month prospective, single institution, pilot study at three teaching hospitals, assessing all PGY levels. We utilized our department mobile application (Wayne State University Surgery departmental app) which includes intraoperative evaluations based on the Zwisch scale. Pre-study and post-study surveys were conducted. An unstructured interview of the Clinical Competency Committee provided feedback of the new evaluation tool against the previous evaluation forms. Results: Eleven physicians participated in the study, resulting in 126 encounters and 184 procedures. A 10-question pre-study survey was given with answers ranging from 1 (strongly disagree) to 5 (strongly agree). The Clinical Competency Committee faculty ranked the pre-study resident assessment tool 2.82/5, while the post-study survey scored 4.64/5. Conclusion: Residents and faculty both rated the smartphone application as a useful tool for evaluating residents. The success of the application proves its feasibility within plastic surgery residency and may play an important role in rating resident operative competency in the future.
... The Zwisch scale has previously been used within cardiothoracic surgery, but the SIMPL app, which utilizes the Zwisch scale, has only been deployed at relatively few training programs and, as such, merits evaluation. [4][5][6] We describe the engagement and feasibility of using this smartphone-based app to provide operative feedback at a cardiothoracic training program over 26 months. We hypothesize that the app can be used to provide regular, documented feedback on operative skills which can be used to track trainee progress. ...
... The SIMPL app has previously been described along with the Zwisch scale. 5 ...
Article
Full-text available
Purpose Timely and high-quality feedback is important in cardiothoracic (CT) surgery education. Feedback on operative proficiency is an area for improvement in CT surgery programs. Traditional evaluations significantly lag behind operative interactions. We hypothesized that use of the System for Improving and Measuring Procedural Learning (SIMPL) app would improve operative feedback for trainees. Methods Use of SIMPL was evaluated from December 2018 to January, 2021 within an academic CT surgery training program. Ratings include level of supervision, complexity of the operation, and trainee performance. Completion was limited to 72 h after the operation. Descriptive statistics of the users and ratings are presented. Results Over 28 months, 816 evaluations were completed, and of these, 495 had a rating from both the faculty and trainee. There were 19 trainees representing post-graduate years 1–8 and 19 faculty members who received or submitted at least one evaluation over the study period. The number of evaluations for each trainee ranged from 1 to 166 and from 1 to 81 for each of the faculty. The response rate for faculty ranged from 0% to 100%. “Active help” was the most common type of supervision (50.7% by the faculty, 60.4% from the trainees). Conclusions Use of SIMPL within a CT surgery training program was feasible and engagement was observed from both trainees and faculty. SIMPL provided trainees with timely, concise feedback on operative performance. Further work will focus on correlating SIMPL ratings with pre-existing assessments of performance.
... The scale ranges from 'show and tell' to 'supervision only'. 13 Both residents and attendings evaluated resident autonomy after each surgical case. ...
... Zwisch scale of autonomy. This scale was used to guide both residents and attendings in the assessment of guidance provided to the resident during the operative encounter.13 ...
Article
Objective To implement the use of standardized preoperative briefings and postoperative debriefings for surgical cases involving residents in an effort to improve resident autonomy and skill acquisition. Design Prospective longitudinal study. Setting Johns Hopkins Department of Otolaryngology-Head and Neck Surgery. Participants Resident and attending physicians. Results Joint Huddles for Improving Resident Education (JHFIRE) tool was created and successfully implemented by 19 residents and 17 faculty members. Over the course of three data collection periods spanning an academic year, overall scores improved though not statistically significantly in the metrics of Zwisch autonomy, Resident Performance, and Objective Structured Assessment of Technical Skills (OSATS) scores. Female residents were scored significantly higher by attendings than their male counterparts in the assessment of baseline Resident Performance. Conclusions (1) JHFIRE tool implemented a standardized preoperative briefing and postoperative debriefing to improve communication and resident skill acquisition; (2) The tool was accepted and utilized throughout an academic year; (3) Zwisch, Resident Performance, and OSATS scores improved though not significantly.
... Entrustment rating forms commonly use a 4-or 5-point scale with each ascending number tied to an anchor that represents a discrete supervisory judgment or decision ordered from most to least amount of supervision provided. 1,[9][10][11][12][13] Numbers are not actually required for entrustment scales, but when they are used, they should be thought of as succinctly representing a shorthand code for a particular supervisory decision that was made; that is, the numbers serve as a label but not as a count or measure. 14 The numbers on entrustment scales can be used to efficiently document a supervisory decision that was made in the moment or record a proclamation of which level of supervision should be used in the future. ...
Article
The adoption of entrustment ratings in medical education is based on a seemingly simple premise: to align workplace-based supervision with resident assessment. Yet it has been difficult to operationalize this concept. Entrustment rating forms combine numeric scales with comments and are embedded in a programmatic assessment framework, which encourages the collection of a large quantity of data. The implicit assumption that more is better has led to an untamable volume of data that competency committees must grapple with. In this article, the authors explore the roles of numbers and words on entrustment rating forms, focusing on the intended and optimal use(s) of each, with a focus on the words. They also unpack the problematic issue of dual-purposing words for both assessment and feedback. Words have enormous potential to elaborate, to contextualize, and to instruct; to realize this potential, educators must be crystal clear about their use. The authors set forth a number of possible ways to reconcile these tensions by more explicitly aligning words to purpose. For example, educators could focus written comments solely on assessment; create assessment encounters distinct from feedback encounters; or use different words collected from the same encounter to serve distinct feedback and assessment purposes. Finally, the authors address the tyranny of documentation created by programmatic assessment and urge caution in yielding to the temptation to reduce words to numbers to make them manageable. Instead, they encourage educators to preserve some educational encounters purely for feedback, and to consider that not all words need to become data.
... These evaluations are submitted through the SIMPL OR smart-phone application, which is a validated operative assessment tool collectively developed and maintained by the SIMPL collaborative. 7,8 The SIMPL OR application has been successfully piloted in one Urology residency program 9 and has been adopted by 17 other Urology residency programs in the United States amassing 9,853 unique evaluations through September 2021. ...
... We therefore do not capture those events in which the resident was operating with the trainer in the "Passive help" role. 31 Despite this, we feel that when considering a graduating chief's comfort with transition to independent operative practice, having operated successfully independently before graduation is an important component of confidence. In this scenario the resident has an opportunity to safely rehearse their future role with an attending surgeon in range. ...
Article
Objective Surgical resident autonomy during training is paramount to independent practice. We sought to determine prevalence of general surgery resident autonomy for surgeries commonly performed on emergency general surgery services and identify trends with time. Design We queried ACS-NSQIP for patients undergoing one of 7 emergency general surgery operations. We evaluated trends in independent operating (defined as a resident operating alone, without attending having scrubbed) over the study period. Other outcomes of interest: operative time, 30-day-mortality and complications. Setting The ACS-NSQIP database. Participants Patients undergoing one of 7 emergency general surgery operations. Results Data regarding resident involvement was only available for the years 2005-2010. 90,790 operations were performed, 922 (1%) by residents operating independently. Appendectomy accounted for 61% independent cases. Independent resident operating was associated with a longer operative time (65 versus 58 minutes, p < 0.001), but lower risk of bleeding requiring transfusion (p < 0.001) and progressive renal insufficiency (p = 0.02). Independent operating was not associated with increased risk of complications/mortality. Conclusion Independent resident operating is rare, even with increasing attention to its importance, and is not associated with increased complications or mortality. National data on this subject is old and not currently collected. There is need for a national registry on resident involvement to understand the current effect of independent operating on outcomes.
... These scales permit programs to incorporate explicitly, trust-based assessments for specific professional activities at each stage of training. This is important for clarity among all stakeholders (including learners) in determining learner progression and in facilitating competencybased advancement of learner responsibilities (George et al. 2014;Weller et al. 2017). ...
... 50% of critical portions of the procedure: show and tell, active help, passive help, and supervision only (online supplementary data TABLE 1 describes the levels). 15 If the autonomy level was greater than show and tell, the trainee's performance was then rated on a 5-level modified Dreyfus scale: unprepared/critical deficiency, inexperienced with procedure, intermediate performance, practice ready performance, and exceptional performance (online supplementary data TABLE 2 describes the levels). 12 Faculty/residents could see the other party's evaluation after both assessments were submitted. ...
Article
Background Gender disparities are prevalent in medicine, but their impact on surgical training is not well studied. Objective To quantify gender disparities in trainee intraoperative experiences and explore the variables associated with ratings of surgical autonomy and performance. Methods From September 2015 to May 2019, attending surgeons and trainees from 71 programs assessed trainee autonomy on a 4-level Zwisch scale and performance on a 5-level modified Dreyfus scale after surgical procedures. Multivariable regression models were used to examine the association of trainee gender with autonomy and performance evaluations. Results A total of 3255 trainees and attending surgeons completed 94 619 evaluations. Attendings gave lower ratings of operative autonomy to female trainees than male trainees when controlling for training level, attending, and surgical procedure (effect size B = −0.0199, P = .008). There was no difference in ratings of autonomy at the beginning of training (P = .32); the gap emerged as trainees advanced in years (B = −0.0163, P = .020). The gender difference in autonomy was largest for the most complex cases (B = −0.0502, P = .002). However, there was no difference in attending ratings of surgical performance for female trainees compared to male trainees (B = −0.0124, P = .066). Female trainees rated themselves as having less autonomy and worse performance than males when controlling for training level, attending, procedure, case complexity, and attending ratings (autonomy B = −0.0669, P < .001; performance B = −0.0704, P < .001). Conclusions While there was no significant difference in ratings of operative performance, a small difference between ratings of operative autonomy for female and male surgical trainees was identified.
... When guidance of the attending surgeon was necessary to ensure patient safety, the degree of this intervention was evaluated by the attending surgeon at the end of the procedure using the Zwisch scale. This scale defines the level of guidance provided to the resident during a case, and it has 4 levels: (1) show and tell, (2) active help, (3) passive help, and (4) supervision only [12]. All videos requesting more than passive help were excluded. ...
Article
Study Objective The technical conduct of total laparoscopic hysterectomy (LH) is critical to surgical outcomes. This study explored the validity evidence of an objective scale specific to the assessment of technical skills (H-OSATS) for seven tasks of the LH with salpingo-oophorectomy procedure performed in the operating room. Design Observational cohort study. Setting Two academic hospitals in France: Marseille and Montpellier. Patients Three groups of operators (novice, intermediate, and experienced surgeons) were video-recorded during their live performances of LH on a simple case. For each group, a dozen unedited videos were obtained for the following tasks: division of the round ligament, division of the infundibulopelvic (IP) ligament, creation of the bladder flap, opening of the posterior peritoneum, division of the uterine vessels, colpotomy, and closure of the vault. Interventions Two qualified raters blindly assessed each video using the H-OSATS rating scale. Inter-rater reliability and test–retest reliability were calculated as measures of internal structure. In a separate round of evaluation, the raters provided a global competent/noncompetent decision for each performance. As a measure of consequential validity, a pass/fail score was set for each task using the contrasting group method. Measurements and Main Results Three tasks (creation of the bladder flap, colpotomy, and closure of the vault) displayed sound validity evidence: a meaningful total score difference among the three groups of experience and between intermediate and experienced surgeons, reliability outcomes of >0.7, and a pass/fail score with a theoretical false positive rate of <10%. Conclusion The validity evidence of the H-OSATS rating scale differed for separate evaluations of the seven tasks. Three tasks (i.e., creation of the bladder flap, colpotomy, and closure of the vault) revealed sound validity evidence, including at the level of the attending surgeon, whereas other tasks were more consistent with low-stakes formative evaluation standards.
... This confirms prior work showing that formal processes such as a time-out to encourage preoperative educational engagement can promote entrustment. 23 Tools such as OpTrust 24 and the Zwisch scale 25,26 have been developed to assess entrustment behaviors using a common language and structure. They also facilitate resident-attending engagement and implicitly encourage entrustment. ...
Article
Background General surgery residents may be underprepared for practice, due in part to declining operative autonomy during training. The factors that influence entrustment of autonomy in the operating room are unclear. Objective To identify and compare the factors that residents and faculty consider influential in entrustment of operative autonomy. Methods An anonymous survey of 29-item Likert-type scale (1–7, 1 = strongly disagree, 7 = strongly agree), 9 multiple-choice, and 4 open-ended questions was sent to 70 faculty and 45 residents in a large ACGME-approved general surgery residency program comprised of university, county, and VA hospitals in 2018. Results Sixty (86%) faculty and 38 (84%) residents responded. Faculty were more likely to identify resident-specific factors such as better resident reputation and higher skill level as important in fostering entrustment. Residents were more likely to identify environmental factors such as a focus on efficiency and a litigious malpractice environment as impeding entrustment. Both groups agreed that work hour restrictions do not decrease autonomy and entrustment does not increase risk to patients. More residents considered low faculty confidence level as a barrier to operative autonomy, while more faculty considered lower resident clinical skill as a barrier. Improvement in resident preparation for cases was cited as an important intervention that could enhance entrustment. Conclusions Differences in perspectives exist between general surgery residents and faculty regarding entrustment of autonomy. Residents cite environmental and attending-related factors, while faculty cite resident-specific factors as most influential. Residents and faculty both agree that entrustment is integral to surgical training.
... Ayuda pasiva fue un esquema que en su mayoría se llevó a cabo con residentes de segundo año (33,3%) y residentes de ultimo año (30,7%), mientras que solo supervisión se asignó a residentes de quinto año en el 76,9% de los casos, observaciones antes vistas por Meyerson et al. 8 así como por George et al. 9 , en las que el mayor porcentaje de solo supervisión lo representó el grupo de residentes de quinto año, con un 44% y 23,2%, respectivamente. ...
Article
Full-text available
Introducción: Para valorar la autonomía de los residentes quirúrgicos se han desarrollado dife- rentes herramientas entre las que destacan la escala System for Improving and Measuring Procedural Learning (SIMPL). Esta herramienta consiste en una evaluación de 3 preguntas para el residente y para el cirujano, con la que se evalúa el desempen ̃o de los residentes en pro- cedimientos quirúrgicos de diversas complejidades. El objetivo de este estudio fue analizar la autonomía de los residentes de cirugía general utilizando la escala SIMPL. Métodos: Basado en la herramienta SIMPL se evaluó la autonomía de un grupo de residentes pertenecientes al programa de cirugía general del Tecnológico de Monterrey en un lapso de 6 meses. Resultados: Se hicieron un total de 124 evaluaciones. El 41,12% (n = 51) fueron residentes meno- res y el 58,8% (n = 73) fueron residentes mayores. Se observó una diferencia significativa y una alta tendencia a requerir demuestra y explica como forma de guía durante la cirugía para los residentes menores (n=22; 39,22%; p=0,0093; OR=2,977). Así mismo, se observaron mayo- res tasas de residentes que requirieron ayuda activa (n=13; 25,49%; r=1,57) y ayuda pasiva (n = 18; 35,29%; r = 1,35) en este grupo. Se observó que la totalidad de los residentes que fueron solamente supervisados fueron del grupo de los residentes mayores (n = 26; 35,62%; p < 0,001). Conclusión: Se obtuvieron evaluaciones objetivas de los residentes de cirugía general. SIMPL es una herramienta eficiente, ya que adapta e integra el desarrollo de un procedimiento quirúrgico para medir la autonomía del residente con el objetivo de evaluar si el residente puede llevar a cabo un procedimiento sin supervisión.
... The O-score includes an evaluation of 8 steps of the surgical procedure using a 5-point scale, as outlined in Table V. The P-score is a summative evaluation using a 5-level assessment based on the work of Zwischenberger 14,15 , as outlined in Table VI. To compare these 2 tools, the ABOS and AOA/CORD collaborated in a study from January to June 2017, comparing the O-tool and P-tool webbased evaluation assessments as completed by faculty 12 . ...
Article
Full-text available
The American Board of Orthopaedic Surgery (ABOS) is the national organization charged with defining education standards for graduate medical education in orthopaedic surgery. The purpose of this article is to describe initiatives taken by the ABOS to develop assessments of competency of residents to document their progress toward the independent practice of orthopaedic surgery and provide feedback for improved performance during training. These initiatives are called the ABOS Knowledge, Skills, and Behavior Program. Web-based assessment tools have been developed and validated to measure competence. These assessments guide resident progress through residency education and better define the competency level by the end of training. The background and rationale for these initiatives and how they serve as steps toward competency-based education in orthopaedic residency education in the United States will be reviewed with a vision of a hybrid of time and competency-based orthopaedic residency education that will remain 5 years in length, with residents assessed using standardized tools.
... Through such an adaptive learning environment, the resident is provided with the opportunity to garner a skill set which is more durable, comprehensive, and translatable. Lastly, incorporating objective feedback mechanisms such as the System for Improving Procedural Learning (SIMPL) or the 4-level Zwisch scale appear integral to surgical curricula, regardless of the surgical approach being applied [19,20]. Such applications allow the resident to be provided with immediate and consistent performance evaluations, creating a real-time awareness of their perceived level of autonomy and competence for a particular case. ...
Article
Full-text available
Purpose of Review To review the integration of robotics in urology residency programs and evaluate how it has impacted a graduates’ level of surgical competence. Recent Findings Surgical technique training has shown a dramatic shift towards robotics with the most profound occurring in oncology. However, integration of robotics is not uniform across programs nor even among residents themselves. Robotics require graduates to garner a broader skill set within the same prescribed training time. Unfortunately, in this modern era, graduates are feeling more ill-equipped to start independent practice and show an increased need to pursue fellowship training to achieve technical proficiency. Summary The dissemination of robotics in residency programs has gone unchecked. Modulating existing training structures through (1) development of procedure- and surgical technique-specific target metrics for graduation and (2) integration of a formalized robotic curriculum may improve the overall quality and outcome of the educational experience.
Article
Purpose: Workplace-based assessment (WBA) serves a critical role in supporting Competency-Based Medical Education (CBME) by providing assessment data to inform competency decisions and support learning. Many WBA systems have been developed, but little is known about how to effectively implement WBA. Filling this gap is important for creating suitable and beneficial assessment processes that support large-scale use of CBME. As a step toward filling this gap, the authors describe what is known about WBA implementation and use to identify knowledge gaps and future directions. Method: The authors used Arksey and O'Malley's six-stage scoping review framework to conduct the review, including: (1) identifying the research question, (2) identifying relevant studies, (3) study selection, (4) charting the data, (5) collating, summarizing, and reporting the results, and (6) consulting with relevant stakeholders. Results: In 2019-2020, the authors searched and screened 726 papers for eligibility using defined inclusion and exclusion criteria. 163 met inclusion criteria. The authors identified five themes in their analysis: (1) Many WBA tools and programs have been implemented, and barriers are common across fields and specialties; (2) Theoretical perspectives emphasize the need for data-driven implementation strategies; (3) User perceptions of WBA vary and are often dependent on implementation factors; (4) Technology solutions could provide useful tools to support WBA; and (5) Many areas of future research and innovation remain. Conclusions: Knowledge of WBA as an implemented practice to support CBME remains constrained. To remove these constraints, future research should aim to generate generalizable knowledge on WBA implementation and use, address implementation factors, and investigate remaining knowledge gaps.
Article
Introduction We present our experience developing and embedding a registry-based module for resident feedback. Methods At our institution, entering operative data into the institutional quality collaborative registry is standard practice. In February 2019, a surgical education module was embedded into the registry to capture procedure-specific resident operative assessments. Faculty engagement with the sugical education module was assessed during its first year in existence (February 2019-February 2020). Results In total, 1074 of 1269 (85%) operative assessments were completed by 27 faculty via the surgical education registry module. Median faculty engagement rate with the module following resident-assisted procedures was 91% [IQR 76%-100%]. Residents received a median of 7 operative assessments [IQR 2-19] over the study period. Conclusion By embedding a surgical education module into an existing surgical quality collaborative registry, procedure-specific operative assessments can be routinely captured.
Article
PURPOSE This study investigates the role of procedure difficulty on attending ratings of supervised levels of independence and procedural performance amongst general surgery residents, while accounting for case complexity. METHODS Attending ratings for residents were obtained from System for Improving and Measuring Procedural Learning (SIMPL) database. Current procedural terminology (CPT) codes were used to match procedures to a corresponding work relative value unit (wRVU) as a surrogate for procedure difficulty. Three categories of wRVU (<13.07, 13.07-22, >22) were identified using recursive partitioning. Procedures were also divided into ‘Core’ or ‘Advanced’ as defined by the American Board of Surgery Surgical Council on Resident Education (SCORE). Temporal advancement in resident skill was accounted for through academic quarterly analysis. A generalized estimating equations (GEE) approach was used to form separate multivariable logistic regression models for meaningful autonomy (MA) and satisfactory performance (SP) adjusted for potential clustering by program, subject, and rater. Models were further adjusted for core/advanced procedures, attending rated complexity, and academic quarter. RESULTS A total of 33,281 ratings were analyzed. Overall, 51.6% were rated as MA and 44.4% as SP. For core procedures, surgical residents rated as MA (53.5%) and SP (45.7%), which was twice as high as those for advance procedures (MA-29.2%, SP-29.0%). MA and SP both decreased with increasing wRVU (Figure 2 &3). Using a wRVU<13.07 as a reference, the adjusted odds ratios of MA and SP were significantly lower with increasing procedure difficulty, 0.44 for wRVU 13.07-22.0 and 0.24 for wRVU >22.00 (Table 3). Post graduate year (PGY) 5 residents in the final quarter of training obtain MA in 95.5% and SP 92.9% for core procedures with wRVU <13.07 (Table 4). CONCLUSION Increasing procedural difficulty is independently associated with decreases in meaningful autonomy and satisfactory performance. As residents approach graduation the level of meaningful autonomy and satisfactory performance both reach high levels for common core procedures but decrease as procedural difficulty increases.
Article
Innovations in surgical education follow advancing clinical technology. New surgical methods have prompted demand for systematic methods to leverage computing power and internet tools to achieve proficiency-based training goals. Virtual reality, high-fidelity patient simulation, web-based resources to facilitate performance assessment, and telementoring have become mainstream practices, although patient outcomes benefits are not well studied. Remote virtual meeting and mentoring have had transformative effects on resident experiences, the full effects of which remain to be seen. Highlights • Proficiency-based virtual reality training methods are effective in the development of surgical skills. • Simulation methods improve clinical performance but evidence of improved patient outcomes is limited. • Telementoring and telepresence tools have proven to be valuable and increasingly utilized adjuncts to more traditional education method. • The COVID-19 pandemic has resulted in adoption of remote learning methods with potentially lasting implications to surgical education.
Article
Wildland fire literacy is the capacity for wildland fire professionals to understand and communicate fundamentals of fuel and fire behavior within the socio-ecological elements of the fire regime. While wildland fire literacy is best developed through education, training, and experience in wildland fire science and management, too often, development among early-career professionals is deficient in one or more aspects of full literacy. We report on a hands-on prescribed fire methods workshop designed to provide training and experience in measuring and conducting prescribed fire, with a focus on grassland ecosystems. The workshop was held in March 2022 at The Nature Conservancy’s Dunn Ranch Prairie in northern Missouri. It consisted of hands-on training and experience in measuring fuels, fire weather, and fire behavior. Prescribed fire operations training facilitated both hands-on learning and vicarious learning by rotating squad roles among several small sub-units on the first day of live fire exercises. Participants then gained experience as crew members for two larger prescribed burns (60 and 200 ha). We report here on the successes and lessons learned from the perspectives of both participants and the instructor cadre for what was widely regarded as a successful workshop.
Article
Objective A resident-run minor surgery clinic was developed to increase resident procedural autonomy. We evaluated whether 1) there was a significant difference between complications and patient satisfaction when procedures were independently performed by surgical residents vs. a surgical attending and 2) if participation was associated with an increase in resident procedural confidence. Design Third year general surgery residents participated in a weekly procedure clinic from 2014-2018. Post-procedure complications and patient satisfaction were compared between patients operated on by residents vs. the staff surgeon. Residents were surveyed regarding their confidence in independently performing a variety of clinic-based patient care tasks. Setting Massachusetts General Hospital General in Boston, MA; an academic tertiary care general surgery residency program. Participants Post-graduate year three general surgery residents that ran the clinic as part of a general surgery rotation. Results 1230 patients underwent 1592 procedures (612 in resident clinic, 980 in attending clinic). There was no significant difference in the 30-day complication rate between patients operated on by the resident vs. attending (2.5% vs. 1.9%, p = 0.49). 459 patient satisfaction surveys were administered with a 79.1% response rate. There was no significant difference in the overall quality of care rating between residents and the attending surgeon (87.5% top-box rating vs. 93.1%, p = 0.15). Twenty-one residents completed both a pre- and post-rotation survey (77.8% response rate). The proportion of residents indicating that they could independently perform a variety of patient care tasks significantly increased across the rotation (all p < 0.05). Conclusion Mid-level general surgery residents can independently perform office-based procedures without detriment to safety or patient satisfaction. The resident-run procedure clinic serves as an environment for residents to grow in confidence in both technical and non-technical skills. Given the high rate at which patients provide resident feedback, future work may investigate how to best incorporate patient derived evaluations into resident assessment.
Article
The operating room continues to be the location where surgical residents develop both technical and nontechnical skills, ultimately culminating with them being capable of safe and independent practice. The process of intraoperative instruction is, by necessity, moving from an apprentice-based model where skills are acquired somewhat randomly through repeated exposure and evaluation is done in a global gestalt fashion. Modern surgical education demands that intraoperative instruction be intentional and that evaluation provides formative and summative feedback. This chapter describes some best practice approaches to intraoperative teaching and evaluation.
Article
Background: High-quality workplace-based assessments are essential for competency-based surgical education. We explored education leaders' perceptions regarding faculty competence in assessment. Methods: Surgical education leaders were surveyed regarding which areas faculty needed improvement, and knowledge of assessment tools. Respondents were queried on specific skills regarding (a)importance in resident/medical student education (b)competence of faculty in assessment and feedback. Results: Surveys (n = 636) were emailed, 103 responded most faculty needed improvement in: verbal (86%) and written (83%) feedback, assessing operative skill (49%) and preparation for procedures (50%). Cholecystectomy, trauma laparotomy, inguinal herniorrhaphy were "very-extremely important" in resident education (99%), but 21-24% thought faculty "moderately to not-at-all" competent in assessment. This gap was larger for non-technical skills. Regarding assessment tools, 56% used OSATS, 49% Zwisch; most were unfamiliar with all non-technical tools. Summary: These data demonstrate a significant perceived gap in competence of faculty in assessment and feedback, and unfamiliarity with assessment tools. This can inform faculty development to support competency-based surgical education.
Article
Objectives Lack of autonomy in the operating room (OR) during general surgery residency is a major contributing factor to low confidence operating independently after graduation. Although attempts to address decreased autonomy and development of entrustment in the OR are being made in general surgery programs, this issue has not been examined thoroughly in vascular surgery. We sought to determine barriers and opportunities for developing operative autonomy during vascular surgery training by surveying program directors (PDs) and trainees (integrated residents and fellows) in U.S. vascular surgery training programs. Methods An anonymous electronic survey was sent via email to all PDs (n=155) and trainees (n=516) in U.S. vascular surgery training programs. Demographics, academic characteristics, and responses regarding factors impacting the development of entrustment were collected. Results Thirty-five PDs and one-hundred trainees completed the survey (22.5% and 19.4% response rate, respectively). Sixty percent of trainees were integrated residents and 40% were fellows. Twenty percent of PDs and 33% of trainees were female, and 5% of all PDs and trainees were from underrepresented minorities. The single most positive factor affecting the development of autonomy according to trainees and PDs is familiarity of the faculty with the trainee. Both PDs and trainees thought the trainee's preparation for the case positively affected development of autonomy; however, more PDs believed that involvement with preoperative preparation in particular (marking the patient, consenting the patient, filling out a history and physical, prepping and draping the patient) was important (p<0.05). PDs believed that duty-hour limitations negatively affected the trainee's ability to develop autonomy in the OR, whereas more trainees believed that hospital or OR efficiency policies played a negative role (p<0.05). Finally, compared with trainees, PDs believed that the appropriate amount of time for safe struggle before the attending should take over the case was when OR efficiency was compromised or at any moment the trainee is unsure of themselves (p<0.05); trainees believed that the attending should take over the case after the limit of their skill set or troubleshooting ability was reached (p<0.05). Conclusions Familiarity of the attending physician with the trainee is an important positive factor for development of entrustment and autonomy in vascular surgery trainees. Duty-hour limitations and belief of the need for hospital efficiency may negatively impact operative independence of trainees. An open discussion about balancing OR efficiency and trainees’ safe struggle is essential to address the growth of independent operative skills in vascular surgery trainees.
Article
Workplace-based assessments and entrustment scales have two primary goals: providing formative information to assist students with future learning; and, determining if and when learners are ready for safe, independent practice. To date, there has not been an evaluation of the relationship between these performance-relevant information pieces in veterinary medicine. This study collected quantitative and qualitative data from a single cohort of final-year students (n = 27) across in-training evaluation reports (ITERs) and entrustment scales in a distributed veterinary hospital environment. Here we compare progression in scoring and performance within and across student, within and across method of assessment, over time. Narrative comments were quantified using the Completed Clinical Evaluation Report Rating (CCERR) instrument to assess quality of written comments. Preliminary evidence suggests that we may be capturing different aspects of performance using these two different methods. Specifically, entrustment scale scores significantly increased over time, while ITER scores did not. Typically, comments on entrustment scale scores were more learner specific, longer, and used more of a coaching voice. Longitudinal evaluation of learner performance is important for learning and demonstration of competence; however, the method of data collection could influence how feedback is structured and how performance is ultimately judged.
Article
Background The impact of general surgery resident participation on operative case time and postoperative complications has been broadly studied in the United States. Although surgical trainee involvement in international humanitarian surgical care is escalating, there is limited information as to how this participation affects care rendered. This study examines the impact of trainee involvement on case length and immediate postoperative complications with regard to operations in low- and middle-income settings. Methods A retrospective chart review was conducted of humanitarian surgeries completed during annual short-term surgical missions performed by the International Surgical Health Initiative to Ghana and Peru. Between 2017 and 2019, procedures included inguinal hernia repairs and total abdominal hysterectomies (TAHs). Operative records were reviewed for case type, duration, and immediate postoperative complications. Cases were categorized as involving two attending co-surgeons (AA) or one attending and resident assistant (RA). Results There were 135 operative cases between 2017 and 2019; the majority (82%) involved a resident assistant. There were no statistically significant differences in case times between the attending assistant (AA) and resident assistant (RA) cohorts in both case types. All 23 postoperative complications were classified as Clavien-Dindo Grade I. In addition, resident assistance did not lead to a statistically significant increase in complication rate; 26% in the AA cohort versus 74% in the RA cohort (P = 0.3). Conclusions This pilot study examining 135 operative cases over 2 y of humanitarian surgeries demonstrates that there were no differences in operative duration or complication rates between the AA and RA cohorts. We propose that surgical trainee involvement in low- and middle-income settings do not adversely impact operative case times or postoperative complications.
Article
Background: The "Surgery for Abdomino-thoracic ViolencE (SAVE)" animate lab engages surgical residents in the management of penetrating injuries in a team setting. Senior residents, representing post graduate year (PGY) 3-5, assume the role of team leader and facilitate the junior residents, PGY1-2, in operative management of simulated penetrating wounds. Residents completed five scenarios with increasing level of difficulty within set time limits. Senior residents were evaluated on their team's ability to "SAVE" their patient within the time allotted as well as their communication and leadership skills. Methods: General, Vascular, Urology, and Plastic Surgery residents (n= 79) were divided into 25 teams of 3-4 residents by "resident-scores" (R-scores, the sum of the team members' PGY), to create balanced teams with comparable years of clinical experience. Residents completed assessments of their senior resident's leadership ability and style. Results: Evaluation of a resident's desired learning style changed across PGY with junior residents preferring more hands-on guidance compared to senior residents preferring only verbal correction. Resident leadership evaluations demonstrated team leaders of varied resident years achieved the highest scores. Greater differences in the mismatch between autonomy provided to and desired by junior residents correlated to greater junior resident discomfort in expressing their opinion, confidence, and leadership ratings of senior residents. However, greater autonomy mismatch also correlated to more rapid time to task completion. Conclusion: Different from our expectations, clinical experience alone did not define team leader success. Leadership is a powerful influence on the outcome of team performance and may be a skill which can transcend overall clinical experience. A match between desired and provided resident autonomy and team cohesion may demonstrate a stronger effect on team success in stressful operative situations, such as trauma resuscitation. Enhancement of leadership skills early in residency training may represent an important focus for trauma surgery education. Level of evidence: IV, economic/decision.
Article
Over the last decade, strict duty hour policies, pressure for increased work related value units from faculty, and the apprenticeship model of education have coalesced to make opportunities for intraoperative teaching more challenging. Evidence is emerging that graduating residents are not exhibiting competence by failing to recognize major complications, and perform routine operations independently. In this pilot study, we combine Vygotsky's social learning theory with a modified version of the competency-based scale called TAGS to study 1 single operation, anterior cervical discectomy and fusion, with 3 individual residents taught by a single faculty member. In order for the 3 residents to achieve “Solo and Observe” in all 4 zones of proximal development, the number of cases required was 10 cases for postgraduate year (PGY)-3a, 19 cases for PGY 3b, and 22 cases for the PGY 2. In this pilot study, the time required to complete an independent 2-level anterior cervical discectomy and fusion by the residents correlated with the number of cases to reach competence. We demonstrate the Surgical Autonomy Program's ability to track neurosurgical resident's educational progress and the feasibility of using the Surgical Autonomy Program (SAP) to teach residents in the operating room and provide immediate formative feedback. Ultimately, the SAP represents a paradigm shift towards a modern, scalable competency-focused subspecialty teaching, evaluation and assessment tool that provides increases in resident's autonomy and metacognitive skills, as well as immediate formative feedback.
Article
Objective In the United States, the majority of colorectal procedures are performed primarily by nonfellowship trained general surgeons. Given that surgical technique and experience affect patient outcomes, it is important that general surgeons are well-trained to perform colorectal surgery operations. In this study, we evaluated how prepared general surgery residents were to perform colorectal procedures upon graduating residency. Design This was a retrospective observational cohort study. Attending ratings of residents’ intraoperative performance were collected with the System for Improving and Measuring Procedural Learning application from 9/2015 to 9/2018. Descriptive analyses and Bayesian mixed models were used to determine a resident's probability of being deemed competent upon graduating residency, controlling for core vs. advanced procedure, case complexity, and rater and resident effects. Setting Faculty and residents within 30 teaching institutions within the Procedural Learning and Safety Collaborative (PLSC). Patients We sampled colorectal procedures and categorized them as core or advanced based on American Board of Surgery designations. Results A total of 564 residents were rated after 2102 operations (82% core, 18% advanced). A resident in their fifth year of clinical training had a 93% (95% CI 85-97%) adjusted probability of competent performance after a core procedure and 75% (95% CI 55-89%) after an advanced procedure. Conclusions General surgery residents were not universally deemed competent to perform colorectal procedures even at the end of residency. These gaps were more pronounced for advanced colorectal procedures. Current graduation requirements should be carefully reviewed to ensure residents are appropriately trained to meet the needs of their communities. Additionally, advanced training remains a critical resource for surgeons who will perform complex colorectal procedures in practice.
Article
Full-text available
Background: Effective cardiothoracic surgical training requires an emphasis on education through timely feedback and constructive criticism. Aims: Review of the implementation of the System for Improving and Measuring Procedural Learning application within a cardiothoracic surgical training program. Materials & Methods: Here, we review a report by Bergquist et al. to the Journal of Cardiac Surgery detailing their implementation of the System for Improving and Measuring Procedural Learning application. Conclusion: It is feasible for the SIMPL application to be implemented with acceptable use in a cardiothoracic training program.
Article
Full-text available
Objective We surveyed otolaryngology program directors (PDs) and recent otolaryngology residency graduates on the operative autonomy of graduating residents and their comfort with independent practice. Methods An anonymous survey was sent to otolaryngology PDs and recent graduates of training programs (members of the Young Physicians Section [YPS] of the American Academy of Otolaryngology-Head and Neck Surgery Foundation). Questions were developed around the 14 key indicator procedures (KIPs) defined by the Accreditation Council for Graduate Medical Education. Results Fifty PDs (43% of PDs) and 152 recent graduates (6% of YPS members) responded. Over 90% of participating PDs felt their graduating residents were either somewhat or extremely comfortable performing 12 out of 14 KIPs. Among the 12 procedures PDs felt their graduating residents were comfortable performing, 57% to 95% of recent graduates also felt either somewhat or extremely comfortable performing them by graduation. Similarly, at least 90% of responding PDs felt their residents achieved meaningful autonomy in the last 2 months of residency prior to graduation for 11 of 14 KIPs. For these same 11 procedures, 74% to 95% of recent graduates indicated they achieved meaningful autonomy. The procedures that PDs and recent graduates felt required the most surgical assistance were ossiculoplasty/stapedectomy, rhinoplasty, and mastoidectomy. All PDs agreed or strongly agreed that graduating residents are comfortable operating and taking call as general otolaryngologists, compared to 86% and 93% of recent graduates. Conclusion Most PDs and recent graduates agree that residents are well-prepared for general otolaryngology practice with the exception of select KIPs. Level of evidence: 4.
Article
OBJECTIVE Accurate recognition of patient-related complexity of an operation is critical for appropriate surgical decision making. It is not yet understood whether general surgery residents are able to accurately assess the relative complexity of a given operative case. This study investigates the agreement of case complexity ratings between residents and attending surgeons and explores whether resident-related factors correlate with any discordance in perception of patient-related operative complexity. DESIGN Residents and attending surgeons rated the relative complexity of completed cases on a 3 point scale via the SIMPL (Society for Improving Medical Professional Learning) operative assessment smartphone app. Additional trainee demographic data, autonomy ratings, and performance ratings were also obtained from the SIMPL registry for each rated case. Complexity agreement was defined as an equal rating between the resident and attending and assigned a value of zero. Over-estimate ratings were assigned a positive value and under-estimate ratings were assigned a negative value. Trends in complexity agreement were analyzed using descriptive statistics and mixed-effects models. RESULTS A total of 43,179 general surgery cases were rated by 1946 categorical general surgery residents and 1520 attending surgeons between 2015 and 2020. Residents and attendings agreed on case complexity in 63.23% of cases, while the residents overestimated complexity in 13.37% of cases and underestimated complexity in 23.40% of cases. Every level of resident except post-graduate year 2 had similar rates of agreement about the complexity of a procedure, while residents who received a higher autonomy rating were more likely to be in agreement with the faculty raters (OR 1.12, 95% CI 1.06-1.19). CONCLUSIONS The results of this study suggest that general surgery residents inaccurately perceive the patient-related complexity of a given case approximately one third of the time. Greater experience and operative autonomy appear to be associated with higher complexity agreement. Future research into factors influencing perceived case complexity may provide insight into how to best implement new teaching for surgical residents regarding the concept of case complexity.
Article
Background The European Society of Gynaecological Oncology (ESGO) and partners are committed to improving the training for gynecologic oncology fellows. The aim of this survey was to assess the type and level of training in cervical cancer surgery and to investigate whether the Laparoscopic Approach to Cervical Cancer (LACC) trial results impacted training in radical surgery for gynecologic oncology fellows. Methods In June 2020, a 47-question electronic survey was shared with European Network of Young Gynaecologic Oncologists (ENYGO) members. Specialist fellows in obstetrics and gynecology, and gynecologic oncology, from high- and low-volume centers, who started training between January 1, 2017 and January 1, 2020 or started before January 1, 2017 but finished their training at least 6 months after the LACC trial publication (October 2018), were included. Results 81 of 125 (64.8%) respondents were included. The median time from the start of the fellowship to completion of the survey was 28 months (range 6–48). 56 (69.1%) respondents were still fellows-in-training. 6 of 56 (10.7%) and 14 of 25 (56.0%) respondents who were still in training and completed the fellowship, respectively, performed ≥10 radical hysterectomies during their training. Fellows trained in an ESGO accredited center had a higher chance to perform sentinel lymph node biopsy (60.4% vs 30.3%; p=0.027). There was no difference in the mean number of radical hysterectomies performed by fellows during fellowship before and after the LACC trial publication (8±12.0 vs 7±8.4, respectively; p=0.46). A significant reduction in number of minimally invasive radical hysterectomies was noted when comparing the period before and after the LACC trial (38.5% vs 13.8%, respectively; p<0.001). Conclusion Exposure to radical surgery for cervical cancer among gynecologic oncology fellows is low. Centralization of cervical cancer cases to high-volume centers may provide an increase in fellows’ exposure to radical procedures. The LACC trial publication was associated with a decrease in minimally invasive radical hysterectomies performed by fellows.
Article
OBJECTIVE There is no standard way in which physicians teach or evaluate surgical residents intraoperatively, and residents are proving to not be fully competent at core surgical procedures upon graduating. The Surgical Autonomy Program (SAP) is a novel educational model that combines a modified version of the Zwisch scale with Vygotsky’s social learning theory. The objective of this study was to establish preliminary validity evidence that SAP is a reliable measure of autonomy and a useful tool for tracking competency over time. METHODS The SAP breaks each surgical case into 4 parts, or zones of proximal development (ZPDs). Residents are evaluated on a 4-tier autonomy scale (TAGS scale) for each ZPD in every surgical case. Attendings were provided with a teaching session about SAP and identified appropriate ZPDs for surgical cases under their area of expertise. All neurosurgery residents at Duke University Hospital from July 2017 to July 2021 participated in this study. Chi-square tests and ordinal logistic regression were used for the analyses. RESULTS Between 2017 and 2021, there were 4885 cases logged by 27 residents. There were 30 attendings who evaluated residents using SAP. Faculty completed evaluations on 91% of cases. The ZPD of focus directly correlated with year of residency (postgraduate year) (χ ² = 1221.1, df = 15, p < 0.001). The autonomy level increased with year of residency (χ ² = 3553.5, df = 15, p < 0.001). An ordinal regression analysis showed that for every year increase in postgraduate year, the odds of operating at a higher level of independence was 2.16 times greater (95% CI 2.11–2.21, p < 0.001). The odds of residents performing with greater autonomy was lowest for the most complex portion of the case (ZPD3) (OR 0.18, 95% CI 0.17–0.20, p < 0.001). Residents have less autonomy with increased case complexity (χ ² = 160.28, df = 6, p < 0.001). Compared with average cases, residents were more likely to operate with greater autonomy on easy cases (OR 1.44, 95% CI 1.29–1.61, p < 0.001) and less likely to do so on difficult cases (OR 0.72, 95% CI 0.67–0.77, p < 0.001). CONCLUSIONS This study demonstrates preliminary evidence supporting the construct validity of the SAP. This tool successfully tracks resident autonomy and progress over time. The authors’ smartphone application was widely used among surgical faculty and residents, supporting integration into the perioperative workflow. Wide implementation of SAP across multiple surgical centers will aid in the movement toward a competency-based residency education system.
Article
OBJECTIVE We hypothesized that a Chief Resident Service educational model provides safe care for patients compared to that received on standard academic services where rotating residents adopt the practices and preferences of their attending. DESIGN We retrospectively identified patients undergoing inguinal hernia repairs from July 2016 through June 2019 and matched Chief's service patients to standard academic service patients 1:1 on CPT, sex and age. We compared patient characteristics, recurrence rates, outcomes and complications. SETTING Tertiary care center, single institution. PARTICIPANTS Overall, 77 patients undergoing inguinal hernia repairs (66% open and 34% laparoscopic) on the Chief's service matched successfully to 77 standard academic service patients during the study period. RESULTS Age, BMI and ASA were similar between the services, but Chief's service patients were less likely to be current smokers (1.3% vs. 24.7%) and more likely to be former smokers (59.7% vs. 26.0%) than standard academic service patients (p < 0.01). Patients presenting with incarcerated hernias (5.2% vs. 9.1%), recurrent (10.4% vs. 5.2%) and bilateral hernias (19.5% vs. 10.4%) were similar between the Chief's service and standard academic services, respectively (all p > 0.05). Operative times were longer for the Chief's service for open (123 min vs. 67, p < 0.01) and laparoscopic (112 min vs. 79, p = 0.02) repairs. Recurrence rates (6.5% vs. 3.9%, p = 0.47) and complications including infection, seroma or hematoma requiring evacuation and need for reoperation were similarly low (p > 0.05) between the Chief's and standard academic services, respectively. Despite low complication rates, Chief's service patients were more likely to present to the ED post-op (14.3% vs. 1.3%; p = 0.001), but readmission rates were similarly low (2.6% vs. 0%, p = 0.09). CONCLUSIONS Providing general surgery chief residents with a supervised opportunity to direct, plan and provide surgical care in clinic and the operating room, as a transition to independent practice following graduation, is safe for patients presenting with inguinal hernias. Concerns about patient safety should not be a barrier to maximizing entrustment for the evaluation and operative management of select core general surgery diagnoses and operations.
Article
Full-text available
Importance: Concern regarding surgical trainees’ operative autonomy has increased in recent years, emphasizing patient safety and preparation for independent practice. Regarding abdominal wall reconstruction (AWR), long-term outcomes of fellow autonomy have yet to be delineated. Objectives: To evaluate the long-term outcomes of AWRs performed by fellows and compare them with those of AWRs performed by assistant, associate, and senior-level professors. Design, Setting, and Participants: This retrospective cohort study included patients who underwent AWR for ventral hernias or repair of tumor resection defects at a 710-bed tertiary cancer center between March 1, 2005, and June 30, 2019. The analysis was conducted between January 2020 and December 2021. Exposure: Academic rank of primary surgeon. Main Outcomes and Measures: The primary outcome was hernia recurrence. Secondary outcomes were surgical site occurrence, surgical site infection, length of hospital stay, unplanned return to the operating room, and 30-day readmission. Multivariable hierarchical models were constructed to identify predictive factors. Results: Of 810 consecutive patients, 720 (mean [SD] age, 59.8 [11.5] years; 375 female [52.1%]) met the inclusion criteria. Mean (SD) body mass index was 31.4 (6.7), and mean (SD) follow-up time was 42 (29) months. Assistant professors performed the most AWRs (276 [38.3%]), followed by associate professors (169 [23.5%]), senior-level professors (157 [21.8%]), and microsurgical fellows (118 [16.4%]). Compared with fellows and more junior surgeons, senior-level professors tended to operate on significantly older patients (mean [SD] age, 59.9 [10.9] years; P = .03), more patients with obesity (103 [65.6%]; P = .003), and patients with larger defects (247.9 [216.0] cm; P < .001), parastomal hernias (27 [17.2%]; P = .001), or rectus muscle violation (53 [33.8%]; P = .03). No significant differences were found for hernia recurrence, surgical site occurrence, surgical site infection, 30-day readmission rates, or length of stay among the fellows and assistant, associate, and senior-level professors in adjusted models. Compared with fellows, assistant professors (OR, 0.22; 95% CI, 0.08-0.64) and senior-level professors (OR, 0.20; 95% CI, 0.06-0.69) had lower rates of unplanned return to the operating room. Conclusions and Relevance: This cohort study provides evidence-based reassurance that providing fellows with autonomy in performing AWRs does not compromise long-term patient outcomes. These findings may incite efforts to increase appropriate surgical trainee autonomy, thereby empowering future generations of competent, independent surgeons.
Article
Background: A subset of Entrustable Professional Activities (EPAs) has been developed for general surgery. We aim to contribute validity evidence for EPAs as an assessment framework for general surgery residents, including concurrent validity compared to ACGME milestones, the current gold standard for evaluating competency. Study design: This is a cross-sectional study in a general surgery training program within a tertiary academic medical center. EPA assessments were submitted using a mobile app and scored on a numerical scale, mirroring milestones. EPA score distribution was analyzed with respect to post-graduate year (PGY) level and phase of care. Proportional odds logistic regression identified significant predictors. Spearman rank and Wilcoxon rank tests were used for comparisons with milestone ratings. Results: From August 2018 to December 2019, 320 assessments were collected. EPA scores increased by PGY level. Operative phase EPA scores were significantly lower than nonoperative phase scores. PGY level, operative phase, and case difficulty significantly influenced entrustment scoring. EPA scores demonstrated strong correlation with nonoperative milestones patient care-1, medical knowledge-1, interpersonal and communication skills-2, interpersonal and communication skills-3, professionalism-1, professionalism-3, and practice-based learning and improvement-2 (ρ > 0.5, p < 0.05) and a weaker correlation with operative milestones patient care-3 and medical knowledge-2 (ρ < 0.5, p < 0.05). Conclusions: The influence of PGY level and operative phase on entrustment scoring supports the validity of EPAs as a formative evaluation framework for general surgery resident performance. In addition, evident correlations between EPA scores and respective milestone ratings provide concurrent validity evidence.
Article
Introduction Leadership is necessary for effective health care teams, particularly for surgeons. Trainees similarly must acquire foundational leadership skills to maximize effectiveness. However, surgical leadership is rarely formally assessed, particularly for junior trainees. We aimed to establish themes of communication, perception and engagement styles, as well as strengths and weaknesses among junior surgical residents at a single institution. Methods The Data Dome Inc. (datadome.com) DISC personality assessment was administered in 2018-2021 to junior residents at an academic general surgery training program at a single institution. Resident demographics were recorded, and themes from deidentified reports were analyzed by year (PGY-1 and PGY-2) using JMP 16 Pro Text Explorer. Results PGY-1 communication was most frequently described as “accomplished best by well-defined avenues” with “duties and responsibilities of others who will be involved explained” in “friendly terms.” PGY-2 communication involved “deal [ing] with people,” “strong feelings about a particular problem,” and being “good at giving verbal and nonverbal feedback.” In ideal environments, PGY-1s self-perceived as “good listener [s],” “good-natured,” and “team player [s].” However, under stress, PGY-1s were perceived by others as “poor listener [s],” “self-promoter [s],” “detached,” and “insensitive.” In ideal environments, PGY-2s were also “good listener [s],” “good-natured,” and “team player [s].” However, under stress, PGY-2 external perception was “overly confident,” “poor listener [s],” and “self-promoter [s].” Conclusions Clear expectations, friendly work environments, and opportunities to succeed are key to effectively train junior surgical residents. In environments where time is often a limited resource, surgical simulation, stress training, and standardized teaching methods from attending surgeons are needed to develop competent trainees.
Article
Background Data defining the utility of the system for improving and measuring procedural learning (SIMPL) in surgical education is limited. The aim of this pilot study is to describe the impact of SIMPL on resident and faculty perspectives regarding operative feedback. Methods Residents and faculty were surveyed prior to and 6 months after SIMPL implementation. Associations were analyzed using [Formula: see text] for categorical and Student’s t-test for continuous variables. Statistical significance was defined as P-value < .05. Results The proportion of residents receiving intraoperative feedback at least once/day increased significantly (35% to 73%, P = .025); there was a trend toward increased postoperative feedback (15% to 33%, P = .201). Faculty reported an increase in intraoperative (55% to 91%, P = .041) and postoperative feedback (21% to 64%, P = .020). Satisfaction with intraoperative feedback improved from a score of 3.50 ± 1.05 to 3.93 ± .62, although not statistically significant ( P = .181). Satisfaction with postoperative feedback improved significantly from 2.85 ± .93 to 3.50 ± .65 ( P = .032). The proportion of faculty who felt they were providing effective feedback increased (53% to 91%, P = .032). The proportion of residents who perceived that feedback enhanced their surgical skill increased as well (65% to 93%, P = .048). Discussion Despite a modest increase in frequency of postoperative feedback, perceived quality of feedback improved substantially after implementation of SIMPL. Introduction of SIMPL also increased the amount of feedback provided by faculty intraoperatively. SIMPL, via direct and indirect effects, has a positive impact on the resident operative learning environment. Further work is necessary to examine the influence this may have on resident operative skill and patient outcomes.
Article
Background General surgery residents commonly engage in research years after the second (Post-postgraduate year 2 [PostPGY2]) or third (PostPGY3) clinical training year. The impact of dedicated research training timing on training experience is unknown. Our aim was to examine the progression of residents’ perceived meaningful operative autonomy and evaluate career satisfaction, in relation to research timing. Methods Categorical surgery residents with 2-year research requirements were surveyed regarding perceived autonomy for laparoscopic appendectomy, laparoscopic cholecystectomy, and right hemicolectomy and satisfaction with the impact of dedicated research training on professional development. Meaningful operative autonomy was defined as Zwisch scores ≥3 (passive help or supervision only). Results Residents from 17 programs participated (n = 233, 30.6%); 48% were PostPGY2. PostPGY3 residents were more likely to perceive meaningful operative autonomy when starting dedicated research training (laparoscopic appendectomy: 98% vs 74%, P < .001; laparoscopic cholecystectomy: 87% vs 48%, P < .001; right hemicolectomy: 27% vs 3%, P < .001). Meaningful operative autonomy declined during dedicated research training but was still higher for PostPGY3 residents for laparoscopic appendectomy (84% vs 42%, P < .001) and laparoscopic cholecystectomy (68% vs 30%, P < .001). By PGY4, PostPGY2 residents reported rates of meaningful operative autonomy comparable to PostPGY3 through training completion. A higher proportion of PostPGY3 residents reported dedicated research training satisfaction (90% vs 78%, P = .01). Training at PostPGY3 programs (odds ratio, 3.06, 95% confidence interval, 1.38–6.80) and postresearch training stage (compared with preresearch residents, odds ratio, 3.25, 95% confidence interval, 1.06–10.0) were independently associated with satisfaction. Conclusion Significant differences existed in the progression of perceived operative autonomy and dedicated research training satisfaction between PostPGY2 and PostPGY3 residents. These results could help surgical educators make individualized decisions regarding research timing to promote surgical skill acquisition and resident well-being.
Article
Full-text available
Objective: Surgical education is changing in an era of new regulations and evolving training cultures. We sought to understand the factors that affect operative experiences during otolaryngology residency. Methods: From December 2019 to December 2020, five otolaryngology training programs used the SIMPL OR smartphone application to evaluate residents after each operation. Residents and attendings rated the trainee's autonomy on a 4-level Zwisch scale, performance on a 5-level scale, and case complexity on a 3-level scale. We examined associations between ratings of autonomy and performance with variables including postgraduate year (PGY), case complexity, gender, week of the academic year (AY), and whether multiple procedures were logged. Results: 78 attendings and 92 residents logged 2984 evaluations. PGY level and week of the AY were positively associated with attending ratings of autonomy and performance (PGY3 vs. PGY2: B = 0.63, p < .001 for autonomy and B = 1.05, p < .001 for performance; week of the AY: B = 0.013, p = .002 for autonomy; B = 0.025, p < .001 for performance). Multiple procedures logged and increasing case complexity were negatively associated with attending ratings (multiple procedures: B = -0.19, p = .04 for autonomy and B = -0.48, p < .001 for performance; hardest vs. easiest 1/3 of cases: B = -1.01, p < .001 for autonomy and B = -0.59, p < .001 for performance). Attending and trainee genders were not associated with attending ratings of autonomy or performance. Conclusion: Resident autonomy and performance were positively associated with PGY level and week of the academic year, and negatively associated with case complexity and multiple procedures. These findings highlight the need to align training level with case complexity to promote quality operative experiences. Level of evidence: 2.
Article
OBJECTIVE Workplace-based assessment is increasingly prevalent in surgical education, especially for assessing operative skill. With current implementations, not all observed clinical performances are assessed, in part because trainees often have discretion about when they seek assessment. As a result, these samples of observed operative performances may not be representative of the full breadth of experience of surgical trainees. Therefore, analyses of these samples may be biased. We aimed to benchmark patterns of procedures logged in the SIMPL operative performance assessment system against records of trainee experience in Accreditation Council for Graduate Medical Education (ACGME) case logs. DESIGN We analyzed SIMPL longitudinal intraoperative performance assessments from categorical trainees in US general surgery residency programs. We compared overall patterns of how procedures are logged in SIMPL and in ACGME case logs using a Pearson correlation, and we examined differences in how individual procedures are logged in each system using Fisher's exact test. RESULTS Total procedure frequency from the SIMPL dataset was strongly correlated with total procedure frequency from ACGME case logs (r = 0.86, 95% CI 0.80-0.90). A subset of these procedures (10 of 116 procedures) was logged more frequently in the SIMPL dataset. These 10 procedures accounted for 56% of SIMPL observations and 30% of ACGME logged cases. Case complexity was comparable for assessments initiated by residents and faculty. CONCLUSIONS Samples of intraoperative performance ratings gathered using the SIMPL application largely resemble ACGME case logs. There is no evidence to indicate that residents preferentially select fewer complex cases for assessment.
Article
Objective There is concern that current otolaryngology residents may not receive adequate surgical training. We aimed to characterize residents’ surgical experiences at 5 academic centers performing the 14 key indicator procedures (KIPs) outlined by the Accreditation Council for Graduate Medical Education. Study Design Prospective study. Setting Five otolaryngology training programs. Methods Data were gathered from December 2019 to December 2020 with a smartphone application from the Society for Improving Medical Professional Learning. After each operation, residents and faculty rated trainee autonomy on a 4-level Zwisch scale and performance on a 5-level modified Dreyfus scale. Results Residents and attendings (n = 92 and 78, respectively) logged 2984 evaluations. Attending ratings of resident autonomy and performance increased with training level ( P < .001). Resident self-assessments of autonomy and performance were lower than paired attending assessments ( P < .001). Among attending evaluations of KIPs performed by senior residents (postgraduate year 4 or 5), 55% of cases were performed with meaningful autonomy (passive help or supervision only). Similarly, attendings rated 55% of these cases as a practice-ready or exceptional performance. Senior residents had meaningful autonomy for ≥50% of cases for most KIPs, with the exception of flaps and grafts (40%), pediatric/adult airway (39%), and stapedectomy/ossiculoplasty (33%). Similarly, senior residents received practice-ready or exceptional performance ratings for ≥50% of cases across all KIPs other than pediatric/adult airway (42%) and stapedectomy/ossiculoplasty (33%). Conclusion In this multicenter study, resident surgical autonomy and performance varied across otolaryngology KIPs. The development of nationwide benchmarks will help programs and residents set educational goals. Level of evidence 2.
Article
Background Implicit bias is a key factor preventing the advancement and retention of women and underrepresented minorities in academic surgery. Purpose We examined the role of implicit bias in the technical component of the residency performance evaluation. The Fundamentals of Laparoscopic Surgery (FLS) score, an objective measure of technical performance, was compared to the subjective technical skills (TS) score given by attending surgeons. Procedures FLS scores and the average TS scores from chief resident evaluations at a university program were analyzed from 2015 to 2019 (n = 29 residents; female 22%, underrepresented minorities 27%). The average TS score for each resident was calculated, scores dichotomized above and below the mean for the program and analyzed across gender and racial identity. Main Findings There were no significant differences in FLS or TS scores between male and female trainees or racial identity. The Kappa correlation coefficient between the 2 dichotomized scores was significantly lower for female (-0.50) versus male (0.23) trainees (p < 0.01); it was not significantly different between racial groups (p = 0.34). Principal Conclusions There was statistically significant difference in agreement between the FLS and TS scores of individual female and male trainees, suggesting the presence of implicit bias in our pilot study. Further research with a larger sample size is warranted. Objective To investigate the presence of implicit bias against women and underrepresented minorities in the technical component of the residency performance evaluation. We hypothesized that women and underrepresented racial minorities would have lower subjective technical skills (TS) scores as compared to their objective FLS scores, relative to the mean for the training program. Design FLS scores and the average TS scores from chief resident performance evaluations were analyzed from 2015-2019. Both FLS and the average TS scores were dichotomized above and below the mean for the program and analyzed across gender and racial identity. Research was approved by institutional IRB. Setting This study was conducted at the University of Arizona General Surgery Residency Program at Banner University Medical Center in Tucson, Arizona. This is a tertiary care university training program. Participants Educational records of graduated general surgery chief residents from 2015 to 2019 were accessed for the study. We analyzed 37 TS scores from attending performance evaluations and 29 FLS scores reported to the program during the study period (22% female, 27% underrepresented racial minorities). Results There were no significant differences in FLS or TS scores between male and female trainees or racial identity. The Kappa correlation coefficient between the 2 dichotomized scores was significantly lower for female (-0.50) versus male (0.23) trainees (p < 0.01); it was not significantly different between racial groups (p = 0.34). Conclusions There was a statistically significant difference in agreement between the FLS and TS score of individual female and male trainees, suggesting the presence of implicit bias in this pilot study. Further research with a larger sample size is warranted.
Article
Background We evaluate the association between attending surgeon involvement and clinical outcome in elective inguinal hernia repairs performed by residents. Methods Patients undergoing initial elective unilateral inguinal hernia repair between 2004 and 2019 were identified using the Veterans Administration Surgical Quality Improvement Program Database. The level of attending surgeon involvement was categorized as active (attending scrubbed [AS]) or passive (supervising the resident's performance but not scrubbed [ANS]). AS and ANS herniorrhaphies were 1:1 propensity matched for patient demographics, comorbidities, surgical approach, resident postgraduate level, and year of repair. Rates of complication and recurrence for matched cohorts were compared by standard methods. Results 30,784 patients met inclusion criteria. 5136 (17%) repairs were performed without the attending scrubbed. On comparison of matched-cohorts, overall complication rates (1.7% vs 1.2%, p = 0.07) and rates of recurrence (1.9% vs 1.4%, p = 0.041) for patients undergoing herniorrhaphy AS were statistically similar to those performed ANS. Conclusion Supervised independence in elective inguinal hernia repair performed by surgical residents is not associated with inferior clinical outcomes.
Article
Background There is a lack of understanding of the scope and purpose of teaching assistant cases, impact on patients and safety, as well as the facilitators or barriers to resident participation in these cases. Methods Four databases (PubMed, Embase, Web of Science, and the Education Resources Information Center), were searched. The references of identified resources were additionally hand-searched. 10 articles were identified and considered in the literature review. Results The TA case literature focuses on case numbers and safety. The discussions of papers allude to perceived benefits of TA cases. The literature review reveals that residents are more likely to be granted TA opportunities if they show themselves worthy of entrustment. Conclusions The work elucidates aspects of TA cases that have not previously been emphasized or highlighted. The literature review can serve to inform attending surgeons and trainees how to optimize the opportunities teaching assistant cases can afford.
Article
OBJECTIVE Gender disparities have demonstrated influence on several areas of medical trainee academic performance and surgeon professional attainment. The impact of gender on perceived operative autonomy and performance of urology residents is not well understood. This single-institution pilot study explores this relationship by evaluating urology faculty and resident assessment of resident operative autonomy and performance using the Society for Improving Medical Professional Learning app. DESIGN Using Society for Improving Medical Professional Learning, trainees in a single urology residency program were assessed in operative cases on three scales (autonomy, performance, and case complexity). Intraoperative assessments were completed by both faculty and residents (self-evaluation). Respective evaluations were compared to explore differences in ratings by gender. SETTING University of Michigan Health, Ann Arbor, MI. PARTICIPANTS University of Michigan Urology Residents and Faculty. RESULTS A total of 516 evaluations were submitted from 18 urology residents and 20 urology faculty. Self-reported ratings among female and male residents did not differ significantly for autonomy (p = 0.20) or performance (p = 0.82). Female and male residents received overall similar autonomy ratings that were not significantly different from female faculty (p = 0.66) and male faculty (p = 0.81). For female residents, there was no significant difference in performance ratings by faculty gender (p = 0.20). This finding was consistent when the resident was male (p = 0.70). CONCLUSIONS At our institution, there is no overall gender-based difference in self-rated or faculty-rated operative autonomy or performance among urology trainees. Understanding relevant facets of institutional culture as well as educational strategies between faculty and residents may identify factors contributing to this outcome.
Chapter
Residency program directors learn methods that foster surgical training. These include: how to provide objective assessments of trainees skills at surgical performances, provide objective feedback and remediation for clinical surgery and simulation skills, trainee preparations for surgery as general knowledge of the surgical procedure as a CEVi, creating simulation practice exercises, identifying basic surgical skills to learn as simulations, how to create a “storyboard” of the surgical procedure, OR room layout and adjust OR table, and medication suite. This training also discusses how trainees may work with different attendings to perform the same surgical case and presents basic endoscopic skills, such as how to assemble and use an endoscope. Portable briefcases for trainees after leaving residency may be created.
Article
OBJECTIVE Workplace-based assessments (WBAs) are used in multiple surgical specialties to facilitate feedback to residents as a form of formative assessment. The validity evidence to support this purpose is limited and has yet to include investigations of how users interpret the assessment and make rating decisions (response processes). This study aimed to explore the validity evidence based on response processes for a WBA in surgery. DESIGN Semi-structured interviews explored the reasonings and strategies used when answering questions in a surgical WBA, the System for Improving and Measuring Procedural Learning (SIMPL). Interview questions probed the interpretation of the three assessment questions and their respective answer categories (level of autonomy, operative performance, case complexity). Researchers analyzed transcripts using directed qualitative content analysis to generate themes. SETTING Single tertiary academic medical center. PARTICIPANTS Eight residents and 13 faculty within the Department of Otolaryngology—Head and Neck Surgery participating in a 6-month pilot of SIMPL. RESULTS We identified four overarching themes that that characterized faculty and resident response processes while completing SIMPL: (1) Faculty and resident users had similar content-level interpretations of the questions and corresponding answer choices; (2) Users employed a variety of cognitive, behavioral, and emotional processes to make rating decisions; (3) Contextual factors influenced ratings; and (4) Tensions during interpretation contributed to rating uncertainty. CONCLUSIONS Response processes are a key source of evidence to support the validity for the formative use of WBAs. Evaluating response process evidence should go beyond basic content-level analysis as contextual factors and tensions that arise during interpretation also play a large role in rating decisions. Additional work and a continued critical lens are needed to ensure that WBAs can truly meet the needs for formative assessment.
Article
OBJECTIVE Structured performance evaluations are important for the professional development and personal growth of resident learners. This process is formalized by the Accreditation Council for Graduate Medical Education milestones assessment system. The primary aim of this study was to understand the current feedback delivery mechanism by exploring the culture of feedback, the mechanics of delivery, and the evaluation of the feedback itself. METHODS Face-to-face interviews were conducted with 10 neurosurgery residents exploring their perceptions of summative feedback. Coded data were analyzed qualitatively for overriding themes using the matrix framework method. A priori themes of definition of feedback, feedback delivery, and impact of feedback were combined with de novo themes discovered during analysis. RESULTS Trainees prioritized formative over summative feedback. Summative and milestone feedback were criticized as being vague, misaligned with practice, and often perceived as erroneous. Barriers to implementation of summative feedback included perceived veracity of feedback, high interrater variability, and the inconstant adoption of a developmental progression model. Gender bias was noted in degree of feedback provided and language used. CONCLUSIONS Trainee perception of feedback provided multiple areas of improvement. This paper can serve as a baseline to study improvements in the milestone feedback process and optimize learning.
Article
PurposeThere is no standard program for laparoscopic surgery training in Japan, and competency in these procedures does not require the acquisition of board certification. The purpose of this survey was to investigate the current status of laparoscopic surgery training in Japan.MethodsA questionnaire survey was mailed to 2296 members of the Japan Society for Endoscopic Surgery who were between postgraduate year 3 and 10. The questionnaire inquired about laparoscopic surgical training conditions, operation case numbers, and autonomy in eight laparoscopic procedures.ResultsThe total response rate was 28.1%. The number of cases required to perform procedures independently was demonstrated. Most participants felt confident in performing laparoscopic appendectomy and cholecystectomy; however, they felt less confident about performing laparoscopic colectomy and gastrectomy.Conclusions The information from this survey may be useful for surgical educators, surgical societies, and the board certification council for rebuilding the surgical training system in Japan.
Article
Full-text available
General surgery residency programs are facing multiple pressures, including attracting and retaining residents. Despite the importance of resident perspectives in designing effective responses to these pressures, understanding of residents' views is limited. To profile US general surgery residents; characterize resident attitudes, experiences, and expectations regarding training; and examine differences by sex and training year. Cross-sectional study of all general surgery residents completing a survey in January 2008 following administration of the American Board of Surgery In-Training Examination. Resident satisfaction; perceived supports, strains and concern; career motivations; and professional expectations. Of 5345 categorical general surgery residents, 4402 (82.4%) responded, representing 248 of 249 surgical residency programs. Most respondents expressed satisfaction with training (3686 [85.2%]; 95% confidence interval [CI], 84.1%-86.3%) and supportive peer relationships (3433 [84.2%]; 95% CI, 83.1%-85.3%). However, residents also reported unmet needs and apprehensions about training and careers. Worry that they will not feel confident performing procedures independently was reported by 1185 (27.5%; 95% CI, 26.2%-28.8%), while 2681 (63.8%; 95% CI, 62.4%-65.3%) reported that they must complete specialty training to be competitive. Perceptions of program support differ, with men more likely than women to report that their program provides support (2188 [74.5%] vs 895 [65.6%]; P < .001), and that they can turn to faculty when having difficulties (2193 [74.5%] vs 901 [66.4%]; P < .001). Reports of having considered leaving training in the prior year differed significantly across years (P < .001), highest in postgraduate year 2 (19.2%) and lowest in postgraduate year 5 (7.2%). General surgery residents' attitudes, experiences, and expectations regarding training reflect both high levels of satisfaction and sources of strain. These factors vary by sex and training year.
Article
The practice of general surgery has undergone a marked evolution in the past 20 years, which has been inadequately recognized and minimally addressed. The changes that have occurred have been disruptive to residency training, and to date there has been inadequate compensation for these. Evidence is now emerging of significant issues in the overall performance of recent graduates from at least 3 sources: the evaluation of external agents who incorporate these graduates into their practice or group, the opinions of the residents themselves, and the performance of graduates on the oral examination of the American Board of Surgery during the past 8 years. The environmental and technological causes of the present situation represent improvements in care for patients, and are clearly irreversible. Hence, solutions to the problems must be sought in other areas. To address the issues effectively, greater recognition and engagement are needed by the surgical community so that effective solutions can be crafted. These will need to include improvements in the efficiency of teaching, with the assumption of greater individual resident responsibility for their knowledge, the establishment of more defined standards for knowledge and skills acquisition by level of residency training, with flexible self-assessment available online, greater focus of the curriculum on current rather than historical practice, increased use of structured assessments (including those in a simulated environment), and modifications to the overall structure of the traditional 5-year residency.
Article
The American Board of Surgery has mandated intraoperative assessment of general surgery residents, yet the time required to train faculty to accurately and reliably complete operating room performance evaluation forms is unknown. Outside of surgical education, frame-of-reference (FOR) training has been shown to be an effective training modality to teach raters the specific performance indicators associated with each point on a rating scale. Little is known, however, about what form and duration of FOR training is needed to accomplish reliable ratings among surgical faculty. Two groups of surgical faculty separately underwent either an accelerated 1-hour (n = 10) or immersive four-hour (n = 34) FOR faculty development program. Both programs included a formal presentation and a facilitated discussion of sample behaviors for each point on the Zwisch operating room performance rating scale (see DaRosa et al.(8)). The immersive group additionally participated in a small group exercise that included additional practice. After training, both groups were tested using 10 video clips of trainees at various levels. Responses were scored against expert consensus ratings. The 2-sided Mann-Whitney U test was used to compare between group means. All trainees were faculty members in the Department of Surgery of a large midwestern private medical school. Faculty undergoing the 1-hour FOR training program did not have a statistically different mean correct response rate on the video test when compared with those undergoing the 4-hour training program (88% vs 80%; p = 0.07). One-hour FOR training sessions are likely sufficient to train surgical faculty to reliably use a simple evaluation instrument for the assessment of intraoperative performance. Additional research is needed to determine how these results generalize to different assessment instruments.
Article
To assess readiness of general surgery graduate trainees entering accredited surgical subspecialty fellowships in North America. A multidomain, global assessment survey designed by the Fellowship Council research committee was electronically sent to all subspecialty program directors. Respondents spanned minimally invasive surgery, bariatric, colorectal, hepatobiliary, and thoracic specialties. There were 46 quantitative questions distributed across 5 domains and 1 or more reflective qualitative questions/domains. There was a 63% response rate (n = 91/145). Of respondent program directors, 21% felt that new fellows arrived unprepared for the operating room, 38% demonstrated lack of patient ownership, 30% could not independently perform a laparoscopic cholecystectomy, and 66% were deemed unable to operate for 30 unsupervised minutes of a major procedure. With regard to laparoscopic skills, 30% could not atraumatically manipulate tissue, 26% could not recognize anatomical planes, and 56% could not suture. Furthermore, 28% of fellows were not familiar with therapeutic options and 24% were unable to recognize early signs of complications. Finally, it was felt that the majority of new fellows were unable to conceive, design, and conduct research/academic projects. Thematic clustering of qualitative data revealed deficits in domains of operative autonomy, progressive responsibility, longitudinal follow-up, and scholarly focus after general surgery education.
Article
The operating room (OR) remains primarily a master/apprenticeship-based learning environment for surgical residents. Changes in surgical education and health care systems challenge faculty to efficiently and effectively graduate residents truly competent in operations classified by the Surgical Council on Resident Education as "common essential" and "uncommon essential." Program directors are charged with employing resident evaluation systems that yield useful data, yet feasible enough to fit into a busy surgical faculty member's workflow. This paper proposes a simple model for teaching and assessing residents in the operating room to guide faculty and resident interaction in the OR, and designating a resident's earned level of autonomy for a given procedure. The system as proposed is supported by theories associated with motor skill acquisition and learning.
Article
Purpose: Most assessment of surgical trainees is based on measures of knowledge, with limited evaluation of their competence to actually perform various surgical procedures. In this study, the authors evaluated a tool they designed to assess a trainee's competence to perform an entire surgical procedure independently, regardless of procedure type or postgraduate year (PGY). Method: In phase 1, the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) was piloted in the University of Ottawa's Division of Orthopaedic Surgery. In phase 2, the refined 11-item tool (8 items rated on a 5-point competency scale, 1 item assessing procedural competence, 2 feedback items) was used in the Divisions of Orthopaedic Surgery and General Surgery to assess residents' performance on 11 common procedures. Quantitative and qualitative analyses were conducted. Results: In phase 2, 34 orthopaedic and general surgeons assessed the performance of 37 residents in 163 procedures. ANOVA demonstrated an effect of PGY. Post hoc analysis found that total procedure scores for PGYs 1 and 2 were lower than those for PGY 3 (P<.001), and PGY 3 scores were lower than those for PGYs 4 and 5 (P<.02). Analysis of qualitative data indicated that the rating scale was practical and useful for surgeons and residents. Conclusions: This novel evaluation tool successfully discriminated between junior and senior residents and identified surgical competency across various PGY levels regardless of procedure type. Multiple sources of evidence support the O-SCORE as a valid tool for the assessment of trainee operative competency.
Article
Threats to the current form of surgical training in the academic medical center include financial pressures from the government and managed care organizations. A diminishing medical student interest in surgical careers has been noted. The constraints of managed care hold the potential to introduce weaknesses in surgical training in the academic medical center.
Article
Resident evaluation traditionally involves global assessments including clinical performance, professional behavior, technical skill, and number of procedures performed. These evaluations lack objective assessment of operative skills. We describe an operative performance rating system (OPRS) designed to provide objective operative performance ratings using a sentinel procedure format. Ten-item procedure-specific rating instruments were developed. Items included technical skills, operative decision making, and general items. A 1 to 5 (5 = excellent) scale was used for evaluation. Six procedures had sufficient forms returned to allow evaluation. Inter-rater reliability was determined by having faculty evaluators view 2 videotaped operations. Return rates for the Internet-based form were full-time faculty (92%), volunteer faculty (27%), and overall (67%). Reliability, (average interitem correlation), and total procedures evaluated were excisional biopsy, 0.90, (0.48), 77; open inguinal herniorraphy, 0.94, (0.62), 51; laparoscopic cholecystectomy, 0.95, (0.64), 75; small-bowel and colon resection, 0.92, (0.58), 30; parathyroidectomy, 0.70, (0.19), 30; and lumpectomy, 0.92, (0.51), 38. Years of training accounted for 25% to 57% of the variation in scores. Inter-rater variability was observed; however, the average rater agreement was reliable. Internet-based management made obtaining the data feasible. The OPRS complements traditional evaluations by providing objective assessment of operative decision-making and technical skills. Interitem correlations indicate the average rating of items provides a reliable indicator of resident performance. The OPRS is useful in tracking resident development throughout postgraduate training and offers a structured means of certifying operative skills.
Article
Competency-based surgical residency training is rapidly becoming the norm across surgical specialties. Ensuring that graduating surgeons are competent to deliver the necessary services and skills to their patients remains a seminal objective of training programs. Defining surgical competence, the measures used to assess and quantify that competence, and the criteria used to judge whether it has been achieved are critical issues. The bar that surgical residency programs have established is, and must continue to be, set very high. Definitions of competency differ across disciplines. In education, two approaches are recognized. According to the behaviorist approach, competence is assessed by precise measures of performance, generally documented by checklists. The integrated (holistic) approach defines competence as a complex combination of personal attributes. Assessments of competence also fall under two categories: the traditional scientific paradigm, emphasizing objectivity and reproducibility, and the judgment paradigm, reflecting the need to assess clinical competence in the final stages of medical training. In surgery, competence is the ability to successfully apply professional knowledge, skills, and attitudes to new situations as well as to familiar tasks. A critical step in assessing surgical competency is developing methodology for competency evaluation and certification. Matching different aspects of surgical competency with the appropriate assessment instruments is the theme of the contemporary evaluation process, with emphasis on a whole-task approach and the assessment of professional judgment. An effective assessment program will incorporate several competency elements, using multiple sources of information to assess competencies on multiple occasions, at various levels, and in different settings.
Article
To identify the most prevalent patterns of technical errors in surgery, and evaluate commonly recommended interventions in light of these patterns. The majority of surgical adverse events involve technical errors, but little is known about the nature and causes of these events. We examined characteristics of technical errors and common contributing factors among closed surgical malpractice claims. Surgeon reviewers analyzed 444 randomly sampled surgical malpractice claims from four liability insurers. Among 258 claims in which injuries due to error were detected, 52% (n = 133) involved technical errors. These technical errors were further analyzed with a structured review instrument designed by qualitative content analysis. Forty-nine percent of the technical errors caused permanent disability; an additional 16% resulted in death. Two-thirds (65%) of the technical errors were linked to manual error, 9% to errors in judgment, and 26% to both manual and judgment error. A minority of technical errors involved advanced procedures requiring special training ("index operations"; 16%), surgeons inexperienced with the task (14%), or poorly supervised residents (9%). The majority involved experienced surgeons (73%), and occurred in routine, rather than index, operations (84%). Patient-related complexities-including emergencies, difficult or unexpected anatomy, and previous surgery-contributed to 61% of technical errors, and technology or systems failures contributed to 21%. Most technical errors occur in routine operations with experienced surgeons, under conditions of increased patient complexity or systems failure. Commonly recommended interventions, including restricting high-complexity operations to experienced surgeons, additional training for inexperienced surgeons, and stricter supervision of trainees, are likely to address only a minority of technical errors. Surgical safety research should instead focus on improving decision-making and performance in routine operations for complex patients and circumstances.
Attitudes, training experiences, and professional expectations of us general surgery residents: a national survey
  • H Yeo
  • K Viola
  • D Berg
Yeo H, Viola K, Berg D, et al. Attitudes, training experiences, and professional expectations of us general surgery residents: a national survey. J Am Med Assoc. 2009;302(12):1301-1308.
Ensuring an Effective Physician Workforce for the US-Summary of a Conference sponsored by the Josiah Macy Jr. Foundation
  • M Johns
Johns M. Ensuring an Effective Physician Workforce for the US-Summary of a Conference sponsored by the Josiah Macy Jr. Foundation, held in Atlanta, GA, 2011.