Article
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Purpose: The existing methods for evaluating resident operative performance interrupt the workflow of the attending physician, are resource intensive, and are often completed well after the end of the procedure in question. These limitations lead to low faculty compliance and potential significant recall bias. In this study, we deployed a smartphone-based system, the Procedural Autonomy and Supervisions System, to facilitate assessment of resident performance according to the Zwisch scale with minimal workflow disruption. We aimed to demonstrate that this is a reliable, valid, and feasible method of measuring resident operative autonomy. Methods: Before implementation, general surgery residents and faculty underwent frame-of-reference training to the Zwisch scale. Immediately after any operation in which a resident participated, the system automatically sent a text message prompting the attending physician to rate the resident's level of operative autonomy according to the 4-level Zwisch scale. Of these procedures, 8 were videotaped and independently rated by 2 additional surgeons. The Zwisch ratings of the 3 raters were compared using an intraclass correlation coefficient. Videotaped procedures were also scored using 2 alternative operating room (OR) performance assessment instruments (Operative Performance Rating System and Ottawa Surgical Competency OR Evaluation), against which the item correlations were calculated. Results: Between December 2012 and June 2013, 27 faculty used the smartphone system to complete 1490 operative performance assessments on 31 residents. During this period, faculty completed evaluations for 92% of all operations performed with general surgery residents. The Zwisch scores were shown to correlate with postgraduate year (PGY) levels based on sequential pairwise chi-squared tests: PGY 1 vs PGY 2 (χ(2) = 106.9, df = 3, p < 0.001); PGY 2 vs PGY 3 (χ(2) = 22.2, df = 3, p < 0.001); and PGY 3 vs PGY 4 (χ(2) = 56.4, df = 3, p < 0.001). Comparison of PGY 4 to PGY 5 scores were not significantly different (χ(2) = 4.5, df = 3, p = 0.21). For the 8 operations reviewed for interrater reliability, the intraclass correlation coefficient was 0.90 (95% CI: 0.72-0.98, p < 0.01). Correlation of Procedural Autonomy and Supervisions System ratings with both Operative Performance Rating System items (each r > 0.90, all p's < 0.01) and Ottawa Surgical Competency OR Evaluation items (each r > 0.86, all p's < 0.01) was high. Conclusions: The Zwisch scale can be used to make reliable and valid measurements of faculty guidance and resident autonomy. Our data also suggest that Zwisch ratings may be used to infer resident operative performance. Deployed on an automated smartphone-based system, it can be used to feasibly record evaluations for most operations performed by residents. This information can be used to council individual residents, modify programmatic curricula, and potentially inform national training guidelines.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... We used the Zwisch Scale and the System for Improving and Measuring Procedural Learning (SIMPL) Performance Scale in order to assess directly and indirectly the residents' autonomy. [3][4][5] The Zwisch Scale was created to evaluate how much the mentor needs to guide the trainee and it is composed of 4 levels: Show and Tell, Active Help, Passive Help, and Supervision Only. 4 The SIMPL Performance Scale measures the trainees' ability to independently reproduce procedures similar to the procedure they just attended and consists of 5 degrees: Unprepared/Critical Deficiency, Unfamiliar with Procedure, Intermediate Performance, Practice Ready, and Exceptional Performance. 5 All the residents involved in our study, both from group A and group B, belonged to level 4 and 5 of the Performance Scale and have been classified by the tutor as level 4 of the Zwisch Scale. ...
... [3][4][5] The Zwisch Scale was created to evaluate how much the mentor needs to guide the trainee and it is composed of 4 levels: Show and Tell, Active Help, Passive Help, and Supervision Only. 4 The SIMPL Performance Scale measures the trainees' ability to independently reproduce procedures similar to the procedure they just attended and consists of 5 degrees: Unprepared/Critical Deficiency, Unfamiliar with Procedure, Intermediate Performance, Practice Ready, and Exceptional Performance. 5 All the residents involved in our study, both from group A and group B, belonged to level 4 and 5 of the Performance Scale and have been classified by the tutor as level 4 of the Zwisch Scale. ...
... All procedures in both groups were carried out by a standard direct visualization of the RLN on each side along its route. The Intermittent Nerve Integrity Monitor (NIM-Response 2.0 System, Medtronic Xomed, Jacksonville, Florida) was used in the group B. The standard intermittent IONM analysis was performed stimulating both the vagus nerve and the RLN before, during and after thyroid resection, using the technique already described in many previous studies [4,6,10], and reporting results and stimulation curves on a pdf record enclosed to the operative report. Any alteration of the curve or loss of signal was reported on the patient chart. ...
Article
Full-text available
Introduction Recurrent laryngeal nerve (RNL) identification constitutes the standard in thyroidectomy. Intraoperative nerve monitoring (IONM) has been introduced as a complementary tool for RLN functionality evaluation. The aim of this study is to establish how routine use of IONM can affect the learning curve (LC) in thyroidectomy. Methods Patients undergoing total thyroidectomy performed by surgery residents in their learning curve course in 2 academic hospitals, were divided into 2 groups: Group A, including 150 thyroidectomies performed without IONM by 3 different residents, and Group B, including 150 procedures with routine use of intermittent IONM, by other 3 different residents. LC was measured by comparing operative time (OT), its stabilization during the development of the LC, perioperative complication rate. Results As previously demonstrated, the LC was achieved after 30 procedures, in both groups, with no differences due to the use of IONM. Similarly, there were no significant differences among the 2 groups, and between subgroups independently matched, for both OT and complications, even when comparing RLN palsy. Direct nerve visualization and IONM assessment rates were comparable in all groups, and no bilateral RLN palsy (transient or permanent) were reported. No case of interrupted procedure to unilateral lobectomy, due to evidence of RLN injury, was reported. Conclusions The study demonstrates that the use of IONM thyroid surgery, despite requiring a specific training with experienced surgeons, does not particularly affect the learning curve of residents approaching this kind of surgery, and for this reason its routine use should be encouraged even for trainees.
... Second, SIMPL creates a repository of longitudinal data so that various stakeholders, including residents, faculty, and programs, can track resident autonomy and performance in operative procedures over time for summative purposes. While both elements are beneficial, the developers believe that the most useful aspect of SIMPL is its ability to function as a formative assessment to facilitate operative feedback, stimulate conversations, guide resident learning, and enhance faculty teaching [17][18][19], which is the focus of this study. ...
... A SIMPL assessment consists of three questions: (1) a rating of resident operative autonomy using the 4 Zwisch levels from Show & Tell to Supervision Only [1,18]; ...
Article
Full-text available
Purpose Workplace-based assessments (WBAs) of trainee operative skills are widely used in surgical education as formative assessments to facilitate feedback for learning, but the evidence to support this purpose is mixed. Further evaluation of the consequences of assessment use and score interpretation is needed to understand if there is alignment between the intended and actual impacts of assessment. This study examines consequences validity evidence for an operative WBA, exploring whether WBA use is consistent with the goals of formative assessment for learning. Methods Eight residents and 9 faculty within the Department of Otolaryngology—Head and Neck Surgery at a tertiary institution completed semi-structured interviews after participating in a pilot of a surgical WBA, the System for Improving and Measuring Procedural Learning in the OR (SIMPL OR). Residents received feedback from attendings via both scores (performance and autonomy ratings) and recorded dictations. Interview questions explored faculty and resident perceptions of feedback behaviors and perceived impacts on their teaching or learning practices. Three researchers analyzed transcripts using directed qualitative content analysis to generate themes and evaluated how the perceived impacts aligned with formative purposes for assessment and score use. Results Both faculty and residents identified intended impacts of formative assessment, including (1) greater emphasis on feedback, (2) support for a postoperative feedback routine, and (3) facilitation of case-specific reflection. Residents also used score and verbal feedback for (1) calibrating case perceptions and (2) benchmarking performance to an external standard. The recorded dictations supported feedback by (1) providing context for ratings, (2) facilitating review of dictated feedback, and (3) prompting faculty for deliberate feedback. Unintended impacts included: (1) emotional discomfort during the assessment process, (2) increased feedback frequency but not diversity or quality, (3) inadequate support for feedback conversations, and (4) limited next steps for teaching or learning. Assessment usage declined over the pilot period. Conclusions The validity evidence gathered in this study suggests an operative WBA can be used for formative purposes to improve perceptions of feedback, but unintended consequences and implementation challenges limited ultimate impacts on teaching and learning. User perspectives can add important elements to consequences validity evidence and should be further evaluated in different implementation settings to better understand how WBAs can achieve their formative goals.
... The Zwisch scale [3][4][5] was used for participants' self-evaluation of operative autonomy in performing advanced pediatric endoscopic surgical procedures. This is a simple scale that rates competency in performing the procedure according to four levels, as follows: ...
Article
Full-text available
Purpose To ensure the safe prevalence of pediatric endoscopic surgery in Japan, a training curriculum should be established. In addition, the number of pediatric surgical cases is decreasing due to the decreasing birth rate in Japan, and it is necessary to clarify the number of surgical cases required for young pediatric surgeons to achieve autonomy in pediatric endoscopic surgery. Methods An online nationwide survey was conducted among young pediatric surgeons with 3–15 years of clinical experience in Japan. We assessed training experience, opinions concerning the ideal training curriculum, and the correlation between surgical experience and the level of autonomy for pediatric endoscopic surgeries. Results One hundred seventy participants responded to the survey (response rate: 35.2%). Only 18% answered that their training facility had a regular educational off-the-job training program. Ninety percent of respondents answered that an educational curriculum for pediatric endoscopic surgery was necessary. It took 11–20 cases to achieve autonomy in laparoscopic appendectomy and laparoscopic inguinal hernia repair. Conclusion This survey revealed that off-the-job training programs were insufficient. The results of this study are expected to aid in the establishment of an effective curriculum for pediatric endoscopic surgery in the era of declining birth rates.
... Debriefing should involve both personal reflection by the learner and constructive feedback by the teacher. Incorporating feedback instruments such as the SIMPL tool (which utilizes a smartphone-based application) or Zwisch scale can help encourage the "debriefing" component and increase discussions of operative autonomy [99,100]. Concise and focused educational interventions such as this may be most feasible to promote learner autonomy in intra-operative judgment and decision-making. ...
Article
Full-text available
Purpose of Review The education of surgeons takes place within a unique surgical culture, and also forms this culture, with its inherent challenges and values. We aim to describe recent surgical education innovations that seek to address the challenges and promote the values of surgical culture. Recent Findings The high stakes of surgical training has prompted educational strategies such as simulation-based practice, communication tools for shared decision-making, and standardized flipped-classroom didactic content. Resilience curricula and formalized mentorship both help ameliorate burnout and the “surgical personality.” The evolution of morbidity and mortality conferences fosters the value of non-abandonment among surgeons, and the inclusion of team-based skills in surgical training is helping surgeons develop the strengths of interdisciplinary cooperation and leadership. In order to promote acquisition of surgical autonomy, perioperative teaching frameworks and entrustable professional activities are being implemented. Summary Surgical culture is evolving; surgical educators continue to innovate within this culture, meeting particular challenges and promoting surgical values.
... EPAs have been particularly well documented and validated for procedure-based tasks (especially in surgical specialties). 14,22 In pathology, the entrusted skill can vary quite broadly from a procedure, to writing a report, to managing a clinical event or activity. This study demonstrated a framework for adjusting entrustment scales to match procedure-based tasks, clinical situation/management tasks, and written report tasks. ...
Article
Full-text available
Entrustable professional activities (EPAs) are observable activities that define the practice of medicine and provide a framework of evaluation that has been incorporated into US medical school curricula in both undergraduate and graduate medical education. This manuscript describes the development of an entrustment scale and formative and summative evaluations for pathology EPAs, outlines a process for faculty development that was employed in a pilot study implementing two Anatomic Pathology and two Clinical Pathology EPAs in volunteer pathology residency programs, and provides initial validation data for the proposed pathology entrustment scales. Prior to implementation, faculty development was necessary to train faculty on the entrustment scale for each given activity. A “train the trainer” model used performance dimension training and frame of reference training to train key faculty at each institution. The session utilized vignettes to practice determination of entrustment ratings and development of feedback for trainees as to strengths and weaknesses in the performance of these activities. Validity of the entrustment scale is discussed using the Messick framework, based on concepts of content, response process, and internal structure. This model of entrustment scales, formative and summative assessments, and faculty development can be utilized for any pathology EPA and provides a roadmap for programs to design and implement EPA assessments into pathology residency training.
... 30 This concept, coupled with continuous postoperative feedback from faculty, is crucial for enhancing not only residents' satisfaction and self-perception but also build their confidence to excel in operating room sessions. 29,31,32 We found that the presence of protected time for research was a significant determinant of satisfaction among senior residents, resulting in a higher satisfaction rate (100%) compared to those without research-allocated time (4.2%). Consistent with the literature, Chan et al. reported that residents were less satisfied (w20% lower) with research opportunities compared to clinical education opportunities. ...
... Objectivity was also noted with a high intra-observer agreement (P < .0001) on the autonomy Zwisch scale. 19 Similarly, a general surgery study looked at SIMPL assessments completed by both faculty and residents with a high overall correlation between residents and faculty for case complexity (r = 0.76, P < .0001), technical performance (r = 0.66, P < .0001), ...
Article
The Society for Improving Medical Professional Learning (SIMPL) Collaborative is a non-profit, educational, quality improvement consortium focused on developing tools, curricula, and policies to improve physician training. The goal is to provide educators and learners with convenient, reliable, and valid evaluation tools for frequent, real-time workplace assessment and feedback. The SIMPL Operating Room application provides this platform. It was developed to provide high-quality, time-sensitive, procedural feedback. Its objective is to facilitate intra-rotation corrections. SIMPL is available to all residency programs and has matured to include over 175 residency programs, involving 19 different specialties, 4000 trainees, 5000 attendings, and 354,800 evaluations in 3 countries. At least 52 peer-reviewed manuscripts have used the evolving database. We have performed an expert narrative review of the entire SIMPL literature (primary research studies, reviews, and websites) to discuss the unique lessons learned from this large collaborative experience. SIMPL can be the core of a competency-based operative skills assessment that is incorporated into medical training, assessment, and certification. Higher quality feedback is provided via SIMPL compared to the routine end-of-rotation evaluations with a corrective comment 60% of the time versus 15%, respectively. A high overall correlation between residents and faculty within case complexity has also been documented (r = 0.76, P < .0001), technical performance (r = 0.66, P < .0001), and autonomy (r = 0.56, P < .0001). The goal of the SIMPL initiative is to use resident performance to drive continuous quality improvement of the individual, programs, and larger medical system.
... The Zwisch scale has been used as an assessment of intraoperative faculty guidance and resident autonomy in general surgery residency. 19 It is a graduated approach that includes four categories -show and tell, active help, passive help, and supervision only. Pittelkow et al conducted a pilot study evaluating neuromodulation surgical skills in pain medicine fellows where a predetermined set of criteria were established, and faculty were instructed to use the Zwisch scale as a guide to evaluate fellow surgical performance during a neuromodulation case. ...
Article
Full-text available
Objective Targeted drug delivery (TDD) via intrathecal drug delivery systems (IDDS) exposure and clinical adoption remains low despite multiple well-designed trials that demonstrate safety, efficacy, reliability, and cost-saving benefits. This study aims to understand the possible contributing factors starting with Pain Medicine fellowship training. Materials and Methods An internet-based, anonymous pilot survey was distributed to pain medicine fellows enrolled in an Accreditation Council for Graduate Medical Education (ACGME) accredited pain medicine training program during the 2021–2022 academic year. Fellowship programs were identified using published online ACGME accreditation data. The survey was distributed via email to fellowship program directors and coordinators and was made available through pain medicine societies. Results Seventy-one of four hundred and twenty-three pain medicine fellows (17% response rate) completed the survey. Nine percent of respondents evidence-informed opinion coincided with the most recent Polyanalgesic Consensus Conference (PACC) guidelines recommendations for IDDS treatment indications. Fifty-one percent of respondents felt there was an unmet need for IDDS training. About one-third of respondents felt that lack of curriculum, faculty, and cases were barriers to IDDS use, respectively. Thirty-one percent of fellows reported sufficient training for IDDS in their fellowship programs. The majority (70%) of respondents somewhat or strongly support direct training by IDDS manufacturers. Conclusion A wide variability exists surrounding IDDS training during ACGME accredited pain medicine fellowship. Insufficient case exposure and lack of a standardized curriculum may play a role in future therapy adoption. The results from this study call for a more standardized training approach with an emphasis on adequate clinical exposure, utilization of peer reviewed educational curriculum and supplemental material to aid pain medicine fellows’ education.
... 6 Following this study, SIMPL training was implemented as a 1-hour, in-person faculty and trainee session, with an additional hour to train the program coordinator and supportive faculty and residents labeled "local champions." 7 In 2020, this approach was adapted to Zoom, where faculty and trainees are shown 6 brief operative videos illustrating both open and laparoscopic examples for the Zwisch scale levels. In all cases, these sessions were led by a volunteer group of experienced SIMPL educators labeled "SIMPL ambassadors" who had been trained to onboard new member programs. ...
Article
Full-text available
Over the past decade, medical education has shifted from a time-based approach to a competency-based approach for surgical training. This transition presents many new systemic challenges. The Society for Improving Medical Professional Learning (SIMPL) was created to respond to these challenges through coordinated collaboration across an international network of medical educators. The primary goal of the SIMPL network was to implement a workplace-based assessment and feedback platform. To date, SIMPL has developed, implemented, and sustained a platform that represents the earliest and largest effort to support workplace-based assessment at scale. The SIMPL model for collaborative improvement demonstrates a potential approach to addressing other complex systemic problems in medical education.
... La evaluación se realizó entre 2015-2016 mediante la escala de Zwisch, un instrumento validado internacionalmente para determinar el nivel de autonomía y supervisión operatoria. La escala permite evaluar cuatro niveles: i) demuestra y explica (el cirujano realiza el procedimiento al residente; ii) ayuda activa (el residente participa activamente en la cirugía sin realizar pasos críticos); iii) ayuda pasiva (el residente realiza el procedimiento y el cirujano lo asiste en partes críticas) y; iv) solo supervisión (el residente realiza el procedimiento con supervisión a distancia del cirujano o con ayuda de un residente menor) 43 . ...
Article
Full-text available
Introducción. El currículo para la formación del cirujano general exige precisión, ajuste al contexto y factibilidad. En 2022, la World Society of Emergency Surgery formuló cinco declaraciones sobre el entrenamiento en cirugía digestiva mínimamente invasiva de emergencia que puede contribuir a estos propósitos. El objetivo del presente artículo fue examinar el alcance de estas declaraciones para la educación quirúrgica en Colombia. Métodos. Se analizó desde una posición crítica y reflexiva el alcance y limitaciones para Colombia de cada una de las declaraciones de la World Society of Emergency Surgery, con base en la evidencia empírica publicada durante las últimas dos décadas en revistas indexadas nacionales e internacionales. Resultados. La evidencia empírica producida en Colombia durante el presente siglo permite identificar que el país cuenta con fundamentos del currículo nacional en cirugía general, formulado por la División de Educación de la Asociación Colombiana de Cirugía en 2021; un sistema de acreditación de la educación superior; un modelo de aseguramiento universal en salud; infraestructura tecnológica y condiciones institucionales que pueden facilitar la adopción exitosa de dichas declaraciones para el entrenamiento de los futuros cirujanos en cirugía digestiva mínimamente invasiva de emergencia. No obstante, su implementación requiere esfuerzos mayores e inversión en materia de simulación quirúrgica, cooperación institucional y fortalecimiento del sistema de recertificación profesional. Conclusión. La educación quirúrgica colombiana está en capacidad de cumplir con las declaraciones de la World Society of Emergency Surgery en materia de entrenamiento en cirugía digestiva mínimamente invasiva de emergencia.
... Case complexity is determined by comparing to the "average" case: lowest 1/3 in complexity, average complexity (middle 1/3), and highest 1/3 in complexity. Case supervision is determined based on the Zwisch scale where 1-show and tell, 2-active help, 3-passive help, 4-supervision only [13]. Case performance is evaluated on a 5-point scale in which 1-critical deficiency, 2-inexperienced with procedure, 3-intermediate performance, 4practice-ready performance, 5-exceptional performance. ...
Article
Full-text available
Background Studies suggest that there are key differences in operative experience based on a trainee’s gender. A large-scale self-efficacy (SE) survey, distributed to general surgery residents after the American Board of Surgery In-Training Examination in 2020, found that female gender was associated with decreased SE in graduating PGY5 residents for all 4 laparoscopic procedures included on the survey (cholecystectomy, appendectomy, right hemicolectomy, and diagnostic laparoscopy). We sought to determine whether these differences were reflected at the case level when considering operative performance and supervision using an operative assessment tool (SIMPL OR).Methods Supervision and performance data reported through the SIMPL OR platform for the same 4 laparoscopic procedures included in the SE survey were aggregated for residents who were PGY5s in 2020. Independent t-tests and multiple linear regression were used to determine the relationship between trainee gender and supervision/performance ratings.ResultsFor laparoscopic cases in aggregate (n = 2708), male residents rated their performance higher than females (3.57 vs. 3.26, p < 0.001, 1 = critical deficiency, 5 = exceptional performance) and reported less supervision (3.15 vs. 2.85, p < 0.001, 1 = show and tell, 4 = supervision only); similar findings were seen when looking at attending reports of resident supervision and performance. A multiple linear regression model showed that attending gender did not significantly predict resident-reported supervision or performance levels, while case complexity and trainee gender significantly affected both supervision and performance (p < 0.001).DiscussionFemale residents perceive themselves to be less self-efficacious at core laparoscopic procedures compared to their male colleagues. Comparison to more case-specific data confirm that female residents receive more supervision and lower performance ratings. This may create a domino effect in which female residents receive less operative independence, preventing the opportunity to establish SE. Further research should identify opportunities to break this cycle and consider gender identity beyond the male/female construct.Graphical abstract
... The Zwisch scale [3][4][5] was used for participants' self-evaluation of their operative autonomy for performing advanced pediatric endoscopic surgical procedures. This is a simple scale that rates the competency in performing the procedure according to four levels, as follows: ...
Article
Full-text available
PurposeTo ensure the safe spread of pediatric endoscopic surgery, it is essential to build a training curriculum, and a survey of the current situation in Japan is necessary. The present study assessed an efficient training curriculum by clarifying instructor class pediatric surgeons’ experiences, including autonomy when performing advanced endoscopic surgeries.Methods An online nationwide questionnaire survey was conducted among pediatric surgeons who had Endoscopic Surgical Skill Qualification (ESSQ) and board-certified instructors who had skills comparable to ESSQ. We assessed participants’ training experience, opinions concerning the ideal training curriculum, and the correlation between surgical experience and the level of autonomy. The Zwisch scale was used to assess autonomy.ResultsFifty-two participants responded to the survey (response rate: 86.7%). Only 57.7% of the respondents felt that they had received sufficient endoscopic surgery training. Most respondents considered an educational curriculum for endoscopic surgery including off-the-job training essential during the training period. Autonomy had been acquired after experiencing two to three cases for most advanced endoscopic surgeries.Conclusion This first nationwide survey in Japan showed that instructor class pediatric surgeons acquired autonomy after experiencing two to three for most advanced endoscopic surgeries. Our findings suggest that training, especially off-the-job training, has been insufficient.
... The Zwisch scale self-evaluation The Zwisch scale [3,4,5] was used for participants' self-evaluation of their operative autonomy for performing advanced pediatric endoscopic surgical procedures. This is a simple scale that rates the competency in performing the procedure according to four levels, as follows: ...
Preprint
Full-text available
Purpose: To ensure the safe spread of pediatric endoscopic surgery, it is essential to build a training curriculum, and a survey of the current situation in Japan is necessary. The present study assessed an efficient training curriculum by clarifying instructor class pediatric surgeons’ experiences, including autonomy when performing advanced endoscopic surgeries. Methods: An online nationwide questionnaire survey was conducted among pediatric surgeons who had Endoscopic Surgical Skill Qualification (ESSQ) and board-certified instructors who had skills comparable to ESSQ. We assessed participants’ training experience, opinions concerning the ideal training curriculum and the correlation between surgical experience and the level of autonomy. The Zwisch scale was used to assess autonomy. Results: Fifty-two participants responded to the survey (response rate: 86.7%). Only 57.7% of the respondents felt that they had received sufficient endoscopic surgery training. Most respondents considered an educational curriculum for endoscopic surgery including off-the-job training essential during the training period. Autonomy had been acquired after experiencing two to three cases for most advanced endoscopic surgeries. Conclusion: This first nationwide survey in Japan showed that instructor class pediatric surgeons acquired autonomy after experiencing two to three for most advanced endoscopic surgeries. Our findings suggest that training, especially off-the-job training, has been insufficient. (Present: 200/ Max. 200 words)
... Given the lack of proven alternatives, rater-based assessment remains the primary mechanism for performance evaluation in orthopedics. [9][10][11][12][13] The flaws of this current technical standard are especially pronounced in the assessment of psychomotor skills. 14,15 One specific core skill, fluoroscopic wire navigation, presents a unique challenge to assessors. ...
Article
Full-text available
Psychomotor skill and decision‐making efficiency in surgical wire navigation can be objectively evaluated by analysis of intraoperative fluoroscopic image sequences. Prior work suggests that such image‐based behavior analysis of operating room (OR) performance can predict performer experience level (R ² = 0.62) and agree with expert opinion (the current standard) on the quality of a final implant construct (R ² = 0.59). However, it is unclear how objective image‐based evaluation compares with expert assessments for entire technical OR performances. This study examines the relationships between three key variables: (1) objective image‐based criteria, (2) expert opinions, and (3) performing surgeon experience level. A paired‐comparison survey of seven experts done based upon eight OR fluoroscopic wire navigation sequences shows that the experts’ preferences are best explained by objective metrics that reflect psychomotor and decision‐making behaviors which are counter‐productive to successful implant placement, like image count (R ² = 0.83) and behavior tally (R ² = 0.74). One such behavior, adjustments away from goal, uniquely correlated well with all three key variables: the fluoroscopic image‐based analysis composite score (R ² = 0.40), expert consensus (R ² = 0.76), and performer experience (R ² = 0.41). These results confirm that experts view less efficient technical behavior as indicative of lesser technical proficiency. However, while expert assessments of technical skill were reliable and consistent, neither individual nor consensus expert opinion appears to correlate with performer experience (R ² = 0.11). This article is protected by copyright. All rights reserved.
Article
Full-text available
Information and data are accelerating the implementation of competency-based medical education. The adoption of precision education can contribute to this purpose. This article discusses the extent to which precision surgical education can be used in assessing the minimum reliability standards of future surgeons — given the advent of Entrustable Professional Activities — and as an option to strengthen the career trajectory of residents.
Article
BACKGROUND AND OBJECTIVES The Accreditation Council for Graduate Medical Education (ACGME) requires neurosurgery residents to reach a set number of cases in specified procedure types (case minimums) before graduation and mandates completion of Milestones. We used the Surgical Autonomy Program, a validated method of autonomy-based resident evaluation, to determine the number of cases it took for residents to become competent and compared these with the ACGME case minimums. METHODS We collected data from neurosurgery residents at Duke University on 7 procedures (tumor craniotomy, trauma craniotomy, ventriculoperitoneal shunt, anterior cervical discectomy and fusion (ACDF), posterior cervical fusion (PCF), discectomy/laminectomy, and posterior thoracolumbar spinal fusion [PSF]). We defined competency as being graded at the highest autonomy level in the Surgical Autonomy Program by attending neurosurgeons for the first and second time and determined the case volume on the ACGME case log when these were achieved. These results were analyzed with summary statistics. RESULTS The median case volume among residents (N = 4-8) for the first and second competency rating (and ACGME minimum) for each procedure type was found to be: tumor: 44.5 and 64.5 (min. 60), trauma: 21 and 30 (min. 60), ventriculoperitoneal shunt: 11.3 and 13 (min. 20), ACDF: 30 and 32.5 (min. 20), PCF: 24 and 40 (min. 30), discectomy/laminectomy: 28 and 36 (min. 30), and PSF: 51 and 54 (min. 30). CONCLUSION We found variation in the case numbers to reach competency and that for some procedures (tumor, ACDF, PCF, discectomy/laminectomy, and PSF), most residents required more cases than the ACGME case minimums to achieve competency. The ACGME case minimums may not accurately reflect the number of cases required for neurosurgical residents to reach competency. To promote trainee-centered education, individualized, competency-based evaluation systems may be better determining readiness for graduation, including a system that builds off the established ACGME Milestones.
Article
Background The proportion of women surgeons is increasing, but studies show that women in surgical residency are granted less autonomy than men. Objective We utilized the Surgical Autonomy Program (SAP), an educational framework, to evaluate gender differences in self-reported autonomy, attending-reported autonomy, and operative feedback among US neurosurgical residents. Methods The SAP tracks resident progression and guides teaching in neurosurgery. Surgeries are divided into zones of proximal development (opening, exposure, critical portion, and closure). Postoperatively, resident autonomy is rated on a 4-point scale by the resident and the attending for each part of the case, or zone. We utilized data from July 2017 to February 2024 from 8 institutions. Ordinal regression was used to evaluate the odds of self- and attending-evaluated autonomy, accounting for gender, training year, case difficulty, and institution. Differences between attending assessment and self-assessment were calculated across time. Chi-square analyses were used to measure any differences in feedback given to men and women. Results From 128 residents (32 women, 25%), 11894 cases were included. Women were granted less autonomy (OR 0.81; 95% CI 0.74-0.89; P<.001) and self-evaluated as having less autonomy (OR 0.73; 95% CI 0.67-0.80; P<.001). The odds of women operating at higher autonomy were similar to the odds of operating on a hard case compared to average difficulty (OR 0.77; 95% CI 0.71-0.84; P<.001). Men’s and women’s self-assessment became closer to attending assessment over time, with women improving more quickly for the critical portions of surgeries. Women residents received meaningful postoperative feedback on fewer cases (women: 74.2%, men: 80.5%; X2=31.929; P<.001). Conclusions Women operated with lower autonomy by both attending and self-assessment, but the assessment gap between genders decreased over time. Women also received less feedback from their attendings.
Article
Full-text available
Background: The Zwisch Score is a recognized tool for objectively assessing resident competency, particularly in measuring faculty guidance. However, there hasn't been a study in Ethiopia yet to assess surgical residents' operative experience using this standardized objective method. Objective: To assess the operative experience of general surgery residents’ using Zwish score in Yekatit 12Hospital Medical College. Methods: A cross-sectional study design was employed, involving all general surgery residents enrolled in the residency program at Yekatit 12 Hospital Medical College from January 2020 to January 2023. Primary data collection utilized online shared questionnaires, with data entry and analysis conducted using SPSS. Result: In Yekatit 12Hospital Medical College, there are 18 Surgeons and 44 surgery residents. Senior residents tend to give junior residents scores of show and tell (33.3% for year 1, 50% for year 2), while year 1 residents most commonly rate themselves as providing active help (18.4%), and year 2 residents rate themselves as show and tell (59.3%). When comparing senior residents' scores with their own, seniors commonly rate themselves as providing passive help (30% for year 3, 65.4% for year 4), and residents rate themselves similarly (31% for year 3, 62.1% for year 4). Conclusion: There are variations between scores given by junior residents and seniors, but senior residents' self-assessments align closely with those of the seniors. This suggests that residents tend to accurately evaluate their skills as they progress through their residency.
Article
Purpose: The System for Improving and Measuring Procedure Learning (SIMPL), a smartphone-based operative assessment application, was developed to assess the intraoperative performance of surgical residents. This study aims to examine the reliability of the SIMPL assessment and determine the optimal number of procedures for a reliable assessment. Methods: In this retrospective observational study, we analyzed data collected between 2015 and 2023 from 4,616 residents across 94 General Surgery Residency programs in the United States that utilized the SIMPL smartphone application. We employed multivariate generalizability theory and initially conducted generalizability studies to estimate the variance components associated with procedures. We then performed decision studies to estimate the reliability coefficient and the minimum number of procedures required for a reproducible assessment. Results: We estimated that the reliability of the assessment of surgical trainees’ intraoperative autonomy and performance using SIMPL exceeded 0.70. Additionally, the optimal number of procedures required for a reproducible assessment was 10, 17, 15, and 17 for postgraduate year (PGY) 2, PGY 3, PGY 4, and PGY 5, respectively. Notably, the study highlighted that the assessment of residents in their senior years necessitated a larger number of procedures compared to those in their junior years. Conclusion: The study demonstrated that the SIMPL assessment is reliably effective for evaluating the intraoperative performance of surgical trainees. Adjusting the number of procedures based on the trainees’ training stage enhances the assessment process’s accuracy and effectiveness.
Article
Purpose This study examines the gaps in early career physician readiness for independent practice after graduating from their final graduate medical education (GME) program. Method The authors conducted a literature search of 4 online databases (PubMed, Scopus, Health Business Elite, and Education Resources Information Center) using the following terms: population terms ( GME, fellow, resident , and others), early career terms ( onboarding, hiring, early career, ready , and others), readiness terms ( readiness, preparedness, knowledge, skills, competence ), and independence terms ( attending, physician, independent practice ). The databases were searched on March 12, 2024, for articles that explored GME graduate gaps in readiness for independent practice; assessment tools, curricula, or curricular need aimed at gaps in early career physician readiness; or an area where GME graduates need more knowledge and skills. They extracted specific gaps in preparedness and categorized them using existing competency frameworks. Results The search returned 116 articles addressing gaps in preparedness for independent practice among recent GME graduates. Surgery yielded more articles than any other specialty (43 [37%]). Overall, 192 individual gaps were extracted; the greatest number of gaps came from patient care (75 [39%]) followed by personal and professional development (44 [23%]). The most frequently identified gaps were procedural independence (10 occurrences), practice management (9 occurrences), and billing (7 occurrences). Conclusions Despite advances in GME, learners still struggle when transitioning to independent practice. Personal and professional development is a useful categorization for many gaps and should be considered for inclusion as a GME competency. Systematic assessment of new-to-practice attendings could help stakeholders better understand the true outcomes of GME programs. Concerted investment by specialty societies may drive greater understanding and innovative solutions. Additional study could help address the challenges in the GME-to-practice transition.
Chapter
Clinical and technical skill sets are necessary for the successful completion of a surgical residency training program. Each clinical encounter is an opportunity to expand clinical knowledge and practice decision making. Applying fundamentals of history-taking, physical exam technique, and test interpretation is key to skill development. Technical skills involve practice in the clinical and operative settings, but must also be practiced outside the patient care setting to allow for adequate repetition, comfort, and safety. Many of these skills will be taught, but the development and honing of these skill sets will require self-motivation, feedback, and deliberate practice. For both clinical and technical skill development, there are resources and simulation tools to foster competency and graduated autonomy. Additionally, models of micro-assessments provide potential for greater entrustability. While progressing through training, it is crucial to take an active role in utilizing peer and faculty mentors to identify opportunities for professional growth. This chapter will discuss the foundational principles of surgical training, identify core clinical and technical skills, and describe how the development of these various skills will allow for optimal patient care.
Article
Background A national survey of general surgery residents revealed significant self-assessed deficits in preparation for independent practice, with only 7.7% of graduating postgraduate year 5 residents (n=1145) reporting self-efficacy for all 10 commonly performed operations surveyed. Objective We sought to understand why this phenomenon occurs. We hypothesized that self-efficacy would be positively correlated with both operative independence and case volume. Methods We compared 3 independent datasets: case information for the same 10 previously surveyed operations for residents graduating in 2020 (dataset 1), operative independence data obtained through the SIMPL OR app, an operative self-assessment tool (dataset 2), and case volume data obtained through the Accreditation Council for Graduate Medical Education National Data Report (dataset 3). Operations were categorized into high, middle (mid), and low self-efficacy tiers; analysis of variance was used to compare operative independence and case volume per tier. Results There were significant differences in self-efficacy between high (87.7%), mid (68.3%), and low (25.4%) tiers (P=.008 [95% CI 6.2, 32.7] for high vs mid, P<.001 for high vs low [49.1, 75.6], and P<.001 for mid vs low [28.7, 57.1]). The percentage of cases completed with operative independence followed similar trends (high 32.7%, mid 13.8%, low 4.9%, P=.006 [6.4, 31.4] for high vs mid, P<.001 [15.3, 40.3] for high vs low, P=.23 [-4.5, 22.3] for mid vs low). The total volume of cases decreased from high to mid to low self-efficacy tiers (average 91.8 to 20.8 to 11.1) but did not reach statistical significance on post-hoc analysis. Conclusions In this analysis of US surgical residents, operative independence was strongly correlated with self-efficacy.
Article
Importance A competency-based assessment framework using entrustable professional activities (EPAs) was endorsed by the American Board of Surgery following a 2-year feasibility pilot study. Pilot study programs’ clinical competency committees (CCCs) rated residents on EPA entrustment semiannually using this newly developed assessment tool, but factors associated with their decision-making are not yet known. Objective To identify factors associated with variation in decision-making confidence of CCCs in EPA summative entrustment decisions. Design, Setting, and Participants This cohort study used deidentified data from the EPA Pilot Study, with participating sites at 28 general surgery residency programs, prospectively collected from July 1, 2018, to June 30, 2020. Data were analyzed from September 27, 2022, to February 15, 2023. Exposure Microassessments of resident entrustment for pilot EPAs (gallbladder disease, inguinal hernia, right lower quadrant pain, trauma, and consultation) collected within the course of routine clinical care across four 6-month study cycles. Summative entrustment ratings were then determined by program CCCs for each study cycle. Main Outcomes and Measures The primary outcome was CCC decision-making confidence rating (high, moderate, slight, or no confidence) for summative entrustment decisions, with a secondary outcome of number of EPA microassessments received per summative entrustment decision. Bivariate tests and mixed-effects regression modeling were used to evaluate factors associated with CCC confidence. Results Among 565 residents receiving at least 1 EPA microassessment, 1765 summative entrustment decisions were reported. Overall, 72.5% (1279 of 1765) of summative entrustment decisions were made with moderate or high confidence. Confidence ratings increased with increasing mean number of EPA microassessments, with 1.7 (95% CI, 1.4-2.0) at no confidence, 1.9 (95% CI, 1.7-2.1) at slight confidence, 2.9 (95% CI, 2.6-3.2) at moderate confidence, and 4.1 (95% CI, 3.8-4.4) at high confidence. Increasing number of EPA microassessments was associated with increased likelihood of higher CCC confidence for all except 1 EPA phase after controlling for program effects (odds ratio range: 1.21 [95% CI, 1.07-1.37] for intraoperative EPA-4 to 2.93 [95% CI, 1.64-5.85] for postoperative EPA-2); for preoperative EPA-3, there was no association. Conclusions and Relevance In this cohort study, the CCC confidence in EPA summative entrustment decisions increased as the number of EPA microassessments increased, and CCCs endorsed moderate to high confidence in most entrustment decisions. These findings provide early validity evidence for this novel assessment framework and may inform program practices as EPAs are implemented nationally.
Article
Purpose: Supervisors may be prone to implicit (unintentional) bias when granting procedural autonomy to trainees due to the subjectivity of autonomy decisions. The authors aimed to conduct a systematic review and meta-analysis to assess the differences in perceptions of procedural autonomy granted to physician trainees based on gender and/or race. Method: MEDLINE, Embase, CENTRAL, Scopus, and Web of Science were searched (search date: January 5, 2022) for studies reporting quantitative gender- or race-based differences in perceptions of procedural autonomy of physician trainees. Reviewers worked in duplicate for article selection and data abstraction. Primary measures of interest were self-reported and observer-rated procedural autonomy. Meta-analysis pooled differences in perceptions of procedural autonomy based on trainee gender. Results: The search returned 2,714 articles, of which 16 were eligible for inclusion. These reported data for 6,109 trainees (median 90 per study) and 2,763 supervisors (median 54 per study). No studies investigated differences in perceptions of autonomy based on race. In meta-analysis of disparities between genders in autonomy ratings (positive number favoring female trainees), pooled standardized mean differences were -0.12 (95% confidence interval [CI] = -0.19, -0.04; P = .003; n = 10 studies) for trainee self-rated autonomy and -0.05 (95% CI = -0.11, 0.01; P = .07; n = 9 studies) for supervisor ratings of autonomy. Conclusions: Limited evidence suggests that female trainees perceived that they received less procedural autonomy than did males. Further research exploring the degree of gender- and race-based differences in procedural autonomy, and factors that influence these differences, is warranted.
Article
Importance As the surgical education paradigm transitions to entrustable professional activities, a better understanding of the factors associated with resident entrustability are needed. Previous work has demonstrated intraoperative faculty entrustment to be associated with resident entrustability. However, larger studies are needed to understand if this association is present across various surgical training programs. Objective To assess intraoperative faculty-resident behaviors and determine if faculty entrustment is associated with resident entrustability across 4 university-based surgical training programs. Design, Setting, and Participants This cross-sectional study was conducted at 4 university-based surgical training programs from October 2018 to May 2022. OpTrust, a validated tool designed to assess both intraoperative faculty entrustment and resident entrustability behaviors independently, was used to assess faculty-resident interactions. A total of 94 faculty and 129 residents were observed. Purposeful sampling was used to create variation in type of operation performed, case difficulty, faculty-resident pairings, faculty experience, and resident training level. Main Outcomes and Measures Observed resident entrustability scores (scale 1-4, with 4 indicating full entrustability) were compared with reported measures (faculty level, case difficulty, resident postgraduate year [PGY], resident gender, observation month) and observed faculty entrustment scores (scale 1-4, with 4 indicating full entrustment). Path analysis was used to explore direct and indirect effects of the predictors. Associations between resident entrustability and faculty entrustment scores were assessed by pairwise Pearson correlation coefficients. Results A total of 338 cases were observed. Cases observed were evenly distributed by faculty experience (1-5 years’ experience: 67 [20.9%]; 6-14 years’ experience: 186 [58%]; ≥15 years’ experience: 67 [20.9%]), resident PGY (PGY 1: 28 [8%]; PGY 2: 74 [22%]; PGY 3: 64 [19%]; PGY 4: 40 [12%]; PGY 5: 97 [29%]; ≥PGY 6: 36 [11%]), and resident gender (female: 183 [54%]; male: 154 [46%]). At the univariate level, PGY (mean [SD] resident entrustability score range, 1.44 [0.46] for PGY 1 to 3.24 [0.65] for PGY 6; F = 38.92; P < .001) and faculty entrustment (2.55 [0.86]; R ² = 0.94; P < .001) were significantly associated with resident entrustablity. Path analysis demonstrated that faculty entrustment was associated with resident entrustability and that the association of PGY with resident entrustability was mediated by faculty entrustment at all 4 institutions. Conclusions and Relevance Faculty entrustment remained associated with resident entrustability across various surgical training programs. These findings suggest that efforts to develop faculty entrustment behaviors may enhance intraoperative teaching and resident progression by promoting resident entrustability.
Article
Full-text available
Purpose of Review With the continued growth of the field of surgery, procedural education must be prioritized. The aim of this article is to review methods for procedural teaching as well as tools for operative performance assessment. Recent Findings There are many techniques for approaching peri-procedural education, all of which have their advantages and disadvantages. Currently, data suggest that trainees and educators do not regularly engage in peri-procedural educational discussion. Regarding assessment, immediate post-procedural feedback is recommended. This should be for single, directly observed encounters, and ideally should be combined with dialogue or written (or dictated) commentary from the educator. Summary With technological innovation, patient and case complexity, and time constraints for both educators and trainees, the need for formalized educational strategies has never been stronger. A number of techniques that can be used for peri-procedural teaching and assessment have been reviewed in this chapter and can be implemented based on institutional, educator, and learner preferences.
Article
Full-text available
Background Assessment of the Core Entrustable Professional Activities for Entering Residency requires direct observation through workplace‐based assessments (WBAs). Single‐institution studies have demonstrated mixed findings regarding the reliability of WBAs developed to measure student progression towards entrustment. Factors such as faculty development, rater engagement and scale selection have been suggested to improve reliability. The purpose of this investigation was to conduct a multi‐institutional generalisability study to determine the influence of specific factors on reliability of WBAs. Methods The authors analysed WBA data obtained for clerkship‐level students across seven institutions from 2018 to 2020. Institutions implemented a variety of strategies including selection of designated assessors, altered scales and different EPAs. Data were aggregated by these factors. Generalisability theory was then used to examine the internal structure validity evidence of the data. An unbalanced cross‐classified random‐effects model was used to decompose variance components. A phi coefficient of >0.7 was used as threshold for acceptable reliability. Results Data from 53 565 WBAs were analysed, and a total of 77 generalisability studies were performed. Most data came from EPAs 1 ( n = 17 118, 32%) 2 ( n = 10 237, 19.1%), and 6 ( n = 6000, 18.5%). Low variance attributed to the learner (<10%) was found for most (59/77, 76%) analyses, resulting in a relatively large number of observations required for reasonable reliability (range = 3 to >560, median = 60). Factors such as DA, scale or EPA were not consistently associated with improved reliability. Conclusion The results from this study describe relatively low reliability in the WBAs obtained across seven sites. Generalisability for these instruments may be less dependent on factors such as faculty development, rater engagement or scale selection. When used for formative feedback, data from these instruments may be useful. However, such instruments do not consistently provide reasonable reliability to justify their use in high‐stakes summative entrustment decisions.
Article
General surgery training programs should provide equitable experiences to both categorical and preliminary residents, as up to 60% of non-designated preliminary general surgery residents will go on to complete surgical residencies. However, self-reported data suggest that preliminary general surgery residents perceive inferior operative experiences when compared with their categorical peers. In this study, we used workplace-based data to determine if there are differences in operative experiences between categorical and preliminary general surgery residents. This study included operative evaluations for all categorical and preliminary PGY-1 and PGY-2 residents in the Society for Improving Medical Professional Learning (SIMPL) data registry from 2015 to 2023. Comparison of proportions and linear mixed models were used to compare frequency of feedback (both numerical and narrative) as well as mean operative autonomy and operative performance ratings between groups (categorical, designated preliminary, non-designated preliminary) within each PGY cohort. 49,737 faculty and resident operative evaluations from 65 institutions were included. Faculty were more likely to complete a SIMPL evaluation for categorical PGY-1’s than for designated or non-designated preliminary PGY-1’s (58 vs. 50% and 58 vs. 48%, respectively; both p < 0.01), and equally as likely to complete a SIMPL evaluation for categorical and preliminary PGY-2’s (54 vs. 52%, p = 0.11). Though performance ratings did not differ by PGY-1 group, both faculty and resident ratings of resident autonomy were lower for non-designated preliminary than categorical PGY-1’s (both p < 0.01). In the PGY-2 cohort, faculty ratings of performance and autonomy were higher for categorical compared to non-designated preliminary residents. Though preliminary and categorical general surgery PGY-1’s have similar operative performance ratings, preliminary PGY-1’s (specifically, non-designated preliminary PGY-1’s) have less operative autonomy and are less likely to receive operative feedback than their categorical peers. For PGY-2 residents, preliminary residents continue to have lower operative autonomy ratings, though performance ratings are also lower compared to their categorical peers. More equitable operative experiences may accelerate preparation of preliminary residents for future careers in surgery or other procedural fields.
Article
Assessment of trainees’ operative autonomy is challenging. However, workplace-based assessment (WBA) systems have made it possible to capture longitudinal data on trainees’ operative autonomy. We evaluated the ability of prior WBA operative autonomy ratings to predict future autonomy ratings for pediatric surgery fellows. WBA data from two pediatric surgery training programs were analyzed using Bayesian mixed effects models to evaluate the relationship between prior cumulative autonomous operative experiences and the probability of being granted meaningful autonomy in a subsequent operation rated using a WBA. Cumulative autonomous experience was modeled as a fixed effect, while procedure, complexity, fellow, and attending surgeon were all modeled as random effects. Marginal predictions were generated and visualized to identify the number of prior autonomous operative experiences needed to achieve a 95% probability of being granted autonomy on a subsequent rating for three procedures: laparoscopic gastrostomy, laparoscopic inguinal hernia repair, and laparoscopic pyloromyotomy. At site one, 843 of 1111 (76%) evaluations were rated as meaningfully autonomous for 9 fellows. At site two, 201 of 234 evaluations (86%) were rated as meaningfully autonomous for 3 fellows. Both sites identified similar expectations for autonomy based on cumulative autonomous experiences. The number of previously autonomous ratings needed to achieve a 95% probability of being granted meaningful autonomy for a subsequent procedure were 12 (site one and two) for laparoscopic gastrostomy, 10 (site one) and 14 (site two) for laparoscopic inguinal hernia repair, and 9 (site one) and 13 (site two) for laparoscopic pyloromyotomy. Prior operative autonomy ratings appear effective in predicting the probability of being granted meaningful autonomy in subsequent procedures across multiple faculty and fellow groups. The approach demonstrated in this paper could support establishing minimum case number requirements and monitoring of fellows’ developing entrustability.
Article
This Viewpoint suggests measures to improve surgical resident autonomy and thereby produce capable and resilient surgeons.
Article
Our residents expressed dissatisfaction with operative autonomy and faculty feedback regarding technical skills. They reported variability among faculty regarding allowed operative autonomy. Our goals were to establish a shared mental model among residents and faculty regarding intraoperative performance expectations. We asked faculty to assign a level of expected autonomy (Zwisch scale) for various steps of common procedures according to the resident post-graduate year. Through an iterative process, the maps were standardized across service lines. The resulting "Autonomy Maps" were distributed to the faculty and residents. We held educational sessions and set expectations for use. Selected benchmarks were incorporated into resident end-of-rotation assessment forms. Initial operative case mapping identified variability in faculty expectations for a given post-graduate year and procedure. Residents reported improved satisfaction with understanding expectations regarding operative performance. Establishing autonomy benchmarks facilitated more specific feedback regarding residents' technical skills. Faculty expectations for resident operative autonomy are variable. Autonomy Maps provide structure for a shared mental model between faculty and residents for progressive operative autonomy and serve as a framework for expectations that improve resident satisfaction. Case-specific technical benchmarks are useful tools for assessing residents' technical milestones.
Article
PurposeThere is growing concern over the readiness of orthopedic surgical residents and fellows for independent surgical practice upon completion of their training. This study aims to explore orthopedic surgery (OS) trainees’ experience of accessing operative autonomy by eliciting their perceptions and techniques implemented to gain autonomy.MethodsOS residents and fellows were invited to participate in focus group interviews via a convenience sampling approach. A non-faculty facilitator led the discussions using an interview guide to prompt conversation. All interviews were recorded, de-identified, and then transcribed. Three investigators iteratively analyzed transcripts to identify emerging themes until thematic saturation was achieved. All interviews were performed at Ohio State University Wexner Medical Center, an academic medical institution, in Columbus, Ohio.ResultsA total of 16 residents and 2 fellows participated. Two themes emerged: (1) optimal setting: trainees were allowed more operative autonomy in trauma and on-call cases than elective cases, though they perceived it was their responsibility to earn autonomy; (2) techniques: two techniques promote trainees’ access to autonomy, including trainee-initiated techniques (i.e., building relationship, preoperative planning, knowing attending preferences, and effective communication); and (3) faculty-initiated techniques (i.e., setting expectations, indications conference, and providing graduated autonomy).Conclusions Our study findings suggest OS trainees tend to access least autonomy in elective OS cases. Although trainees perceived earning autonomy as their responsibility, faculty and resident development is recommended to enhance teaching and learning techniques to increase trainees’ practice readiness.
Article
Full-text available
General surgery residency programs are facing multiple pressures, including attracting and retaining residents. Despite the importance of resident perspectives in designing effective responses to these pressures, understanding of residents' views is limited. To profile US general surgery residents; characterize resident attitudes, experiences, and expectations regarding training; and examine differences by sex and training year. Cross-sectional study of all general surgery residents completing a survey in January 2008 following administration of the American Board of Surgery In-Training Examination. Resident satisfaction; perceived supports, strains and concern; career motivations; and professional expectations. Of 5345 categorical general surgery residents, 4402 (82.4%) responded, representing 248 of 249 surgical residency programs. Most respondents expressed satisfaction with training (3686 [85.2%]; 95% confidence interval [CI], 84.1%-86.3%) and supportive peer relationships (3433 [84.2%]; 95% CI, 83.1%-85.3%). However, residents also reported unmet needs and apprehensions about training and careers. Worry that they will not feel confident performing procedures independently was reported by 1185 (27.5%; 95% CI, 26.2%-28.8%), while 2681 (63.8%; 95% CI, 62.4%-65.3%) reported that they must complete specialty training to be competitive. Perceptions of program support differ, with men more likely than women to report that their program provides support (2188 [74.5%] vs 895 [65.6%]; P < .001), and that they can turn to faculty when having difficulties (2193 [74.5%] vs 901 [66.4%]; P < .001). Reports of having considered leaving training in the prior year differed significantly across years (P < .001), highest in postgraduate year 2 (19.2%) and lowest in postgraduate year 5 (7.2%). General surgery residents' attitudes, experiences, and expectations regarding training reflect both high levels of satisfaction and sources of strain. These factors vary by sex and training year.
Article
The practice of general surgery has undergone a marked evolution in the past 20 years, which has been inadequately recognized and minimally addressed. The changes that have occurred have been disruptive to residency training, and to date there has been inadequate compensation for these. Evidence is now emerging of significant issues in the overall performance of recent graduates from at least 3 sources: the evaluation of external agents who incorporate these graduates into their practice or group, the opinions of the residents themselves, and the performance of graduates on the oral examination of the American Board of Surgery during the past 8 years. The environmental and technological causes of the present situation represent improvements in care for patients, and are clearly irreversible. Hence, solutions to the problems must be sought in other areas. To address the issues effectively, greater recognition and engagement are needed by the surgical community so that effective solutions can be crafted. These will need to include improvements in the efficiency of teaching, with the assumption of greater individual resident responsibility for their knowledge, the establishment of more defined standards for knowledge and skills acquisition by level of residency training, with flexible self-assessment available online, greater focus of the curriculum on current rather than historical practice, increased use of structured assessments (including those in a simulated environment), and modifications to the overall structure of the traditional 5-year residency.
Article
The American Board of Surgery has mandated intraoperative assessment of general surgery residents, yet the time required to train faculty to accurately and reliably complete operating room performance evaluation forms is unknown. Outside of surgical education, frame-of-reference (FOR) training has been shown to be an effective training modality to teach raters the specific performance indicators associated with each point on a rating scale. Little is known, however, about what form and duration of FOR training is needed to accomplish reliable ratings among surgical faculty. Two groups of surgical faculty separately underwent either an accelerated 1-hour (n = 10) or immersive four-hour (n = 34) FOR faculty development program. Both programs included a formal presentation and a facilitated discussion of sample behaviors for each point on the Zwisch operating room performance rating scale (see DaRosa et al.(8)). The immersive group additionally participated in a small group exercise that included additional practice. After training, both groups were tested using 10 video clips of trainees at various levels. Responses were scored against expert consensus ratings. The 2-sided Mann-Whitney U test was used to compare between group means. All trainees were faculty members in the Department of Surgery of a large midwestern private medical school. Faculty undergoing the 1-hour FOR training program did not have a statistically different mean correct response rate on the video test when compared with those undergoing the 4-hour training program (88% vs 80%; p = 0.07). One-hour FOR training sessions are likely sufficient to train surgical faculty to reliably use a simple evaluation instrument for the assessment of intraoperative performance. Additional research is needed to determine how these results generalize to different assessment instruments.
Article
To assess readiness of general surgery graduate trainees entering accredited surgical subspecialty fellowships in North America. A multidomain, global assessment survey designed by the Fellowship Council research committee was electronically sent to all subspecialty program directors. Respondents spanned minimally invasive surgery, bariatric, colorectal, hepatobiliary, and thoracic specialties. There were 46 quantitative questions distributed across 5 domains and 1 or more reflective qualitative questions/domains. There was a 63% response rate (n = 91/145). Of respondent program directors, 21% felt that new fellows arrived unprepared for the operating room, 38% demonstrated lack of patient ownership, 30% could not independently perform a laparoscopic cholecystectomy, and 66% were deemed unable to operate for 30 unsupervised minutes of a major procedure. With regard to laparoscopic skills, 30% could not atraumatically manipulate tissue, 26% could not recognize anatomical planes, and 56% could not suture. Furthermore, 28% of fellows were not familiar with therapeutic options and 24% were unable to recognize early signs of complications. Finally, it was felt that the majority of new fellows were unable to conceive, design, and conduct research/academic projects. Thematic clustering of qualitative data revealed deficits in domains of operative autonomy, progressive responsibility, longitudinal follow-up, and scholarly focus after general surgery education.
Article
The operating room (OR) remains primarily a master/apprenticeship-based learning environment for surgical residents. Changes in surgical education and health care systems challenge faculty to efficiently and effectively graduate residents truly competent in operations classified by the Surgical Council on Resident Education as "common essential" and "uncommon essential." Program directors are charged with employing resident evaluation systems that yield useful data, yet feasible enough to fit into a busy surgical faculty member's workflow. This paper proposes a simple model for teaching and assessing residents in the operating room to guide faculty and resident interaction in the OR, and designating a resident's earned level of autonomy for a given procedure. The system as proposed is supported by theories associated with motor skill acquisition and learning.
Article
Purpose: Most assessment of surgical trainees is based on measures of knowledge, with limited evaluation of their competence to actually perform various surgical procedures. In this study, the authors evaluated a tool they designed to assess a trainee's competence to perform an entire surgical procedure independently, regardless of procedure type or postgraduate year (PGY). Method: In phase 1, the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) was piloted in the University of Ottawa's Division of Orthopaedic Surgery. In phase 2, the refined 11-item tool (8 items rated on a 5-point competency scale, 1 item assessing procedural competence, 2 feedback items) was used in the Divisions of Orthopaedic Surgery and General Surgery to assess residents' performance on 11 common procedures. Quantitative and qualitative analyses were conducted. Results: In phase 2, 34 orthopaedic and general surgeons assessed the performance of 37 residents in 163 procedures. ANOVA demonstrated an effect of PGY. Post hoc analysis found that total procedure scores for PGYs 1 and 2 were lower than those for PGY 3 (P<.001), and PGY 3 scores were lower than those for PGYs 4 and 5 (P<.02). Analysis of qualitative data indicated that the rating scale was practical and useful for surgeons and residents. Conclusions: This novel evaluation tool successfully discriminated between junior and senior residents and identified surgical competency across various PGY levels regardless of procedure type. Multiple sources of evidence support the O-SCORE as a valid tool for the assessment of trainee operative competency.
Article
Threats to the current form of surgical training in the academic medical center include financial pressures from the government and managed care organizations. A diminishing medical student interest in surgical careers has been noted. The constraints of managed care hold the potential to introduce weaknesses in surgical training in the academic medical center.
Article
Resident evaluation traditionally involves global assessments including clinical performance, professional behavior, technical skill, and number of procedures performed. These evaluations lack objective assessment of operative skills. We describe an operative performance rating system (OPRS) designed to provide objective operative performance ratings using a sentinel procedure format. Ten-item procedure-specific rating instruments were developed. Items included technical skills, operative decision making, and general items. A 1 to 5 (5 = excellent) scale was used for evaluation. Six procedures had sufficient forms returned to allow evaluation. Inter-rater reliability was determined by having faculty evaluators view 2 videotaped operations. Return rates for the Internet-based form were full-time faculty (92%), volunteer faculty (27%), and overall (67%). Reliability, (average interitem correlation), and total procedures evaluated were excisional biopsy, 0.90, (0.48), 77; open inguinal herniorraphy, 0.94, (0.62), 51; laparoscopic cholecystectomy, 0.95, (0.64), 75; small-bowel and colon resection, 0.92, (0.58), 30; parathyroidectomy, 0.70, (0.19), 30; and lumpectomy, 0.92, (0.51), 38. Years of training accounted for 25% to 57% of the variation in scores. Inter-rater variability was observed; however, the average rater agreement was reliable. Internet-based management made obtaining the data feasible. The OPRS complements traditional evaluations by providing objective assessment of operative decision-making and technical skills. Interitem correlations indicate the average rating of items provides a reliable indicator of resident performance. The OPRS is useful in tracking resident development throughout postgraduate training and offers a structured means of certifying operative skills.
Article
Competency-based surgical residency training is rapidly becoming the norm across surgical specialties. Ensuring that graduating surgeons are competent to deliver the necessary services and skills to their patients remains a seminal objective of training programs. Defining surgical competence, the measures used to assess and quantify that competence, and the criteria used to judge whether it has been achieved are critical issues. The bar that surgical residency programs have established is, and must continue to be, set very high. Definitions of competency differ across disciplines. In education, two approaches are recognized. According to the behaviorist approach, competence is assessed by precise measures of performance, generally documented by checklists. The integrated (holistic) approach defines competence as a complex combination of personal attributes. Assessments of competence also fall under two categories: the traditional scientific paradigm, emphasizing objectivity and reproducibility, and the judgment paradigm, reflecting the need to assess clinical competence in the final stages of medical training. In surgery, competence is the ability to successfully apply professional knowledge, skills, and attitudes to new situations as well as to familiar tasks. A critical step in assessing surgical competency is developing methodology for competency evaluation and certification. Matching different aspects of surgical competency with the appropriate assessment instruments is the theme of the contemporary evaluation process, with emphasis on a whole-task approach and the assessment of professional judgment. An effective assessment program will incorporate several competency elements, using multiple sources of information to assess competencies on multiple occasions, at various levels, and in different settings.
Article
To identify the most prevalent patterns of technical errors in surgery, and evaluate commonly recommended interventions in light of these patterns. The majority of surgical adverse events involve technical errors, but little is known about the nature and causes of these events. We examined characteristics of technical errors and common contributing factors among closed surgical malpractice claims. Surgeon reviewers analyzed 444 randomly sampled surgical malpractice claims from four liability insurers. Among 258 claims in which injuries due to error were detected, 52% (n = 133) involved technical errors. These technical errors were further analyzed with a structured review instrument designed by qualitative content analysis. Forty-nine percent of the technical errors caused permanent disability; an additional 16% resulted in death. Two-thirds (65%) of the technical errors were linked to manual error, 9% to errors in judgment, and 26% to both manual and judgment error. A minority of technical errors involved advanced procedures requiring special training ("index operations"; 16%), surgeons inexperienced with the task (14%), or poorly supervised residents (9%). The majority involved experienced surgeons (73%), and occurred in routine, rather than index, operations (84%). Patient-related complexities-including emergencies, difficult or unexpected anatomy, and previous surgery-contributed to 61% of technical errors, and technology or systems failures contributed to 21%. Most technical errors occur in routine operations with experienced surgeons, under conditions of increased patient complexity or systems failure. Commonly recommended interventions, including restricting high-complexity operations to experienced surgeons, additional training for inexperienced surgeons, and stricter supervision of trainees, are likely to address only a minority of technical errors. Surgical safety research should instead focus on improving decision-making and performance in routine operations for complex patients and circumstances.
Attitudes, training experiences, and professional expectations of us general surgery residents: a national survey
  • H Yeo
  • K Viola
  • D Berg
Yeo H, Viola K, Berg D, et al. Attitudes, training experiences, and professional expectations of us general surgery residents: a national survey. J Am Med Assoc. 2009;302(12):1301-1308.
Ensuring an Effective Physician Workforce for the US-Summary of a Conference sponsored by the Josiah Macy Jr. Foundation
  • M Johns
Johns M. Ensuring an Effective Physician Workforce for the US-Summary of a Conference sponsored by the Josiah Macy Jr. Foundation, held in Atlanta, GA, 2011.