No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Reliability, Validity, and Feasibility of the Zwisch Scale for the Assessment of Intraoperative Performance
Abstract
Purpose:
The existing methods for evaluating resident operative performance interrupt the workflow of the attending physician, are resource intensive, and are often completed well after the end of the procedure in question. These limitations lead to low faculty compliance and potential significant recall bias. In this study, we deployed a smartphone-based system, the Procedural Autonomy and Supervisions System, to facilitate assessment of resident performance according to the Zwisch scale with minimal workflow disruption. We aimed to demonstrate that this is a reliable, valid, and feasible method of measuring resident operative autonomy.
Methods:
Before implementation, general surgery residents and faculty underwent frame-of-reference training to the Zwisch scale. Immediately after any operation in which a resident participated, the system automatically sent a text message prompting the attending physician to rate the resident's level of operative autonomy according to the 4-level Zwisch scale. Of these procedures, 8 were videotaped and independently rated by 2 additional surgeons. The Zwisch ratings of the 3 raters were compared using an intraclass correlation coefficient. Videotaped procedures were also scored using 2 alternative operating room (OR) performance assessment instruments (Operative Performance Rating System and Ottawa Surgical Competency OR Evaluation), against which the item correlations were calculated.
Results:
Between December 2012 and June 2013, 27 faculty used the smartphone system to complete 1490 operative performance assessments on 31 residents. During this period, faculty completed evaluations for 92% of all operations performed with general surgery residents. The Zwisch scores were shown to correlate with postgraduate year (PGY) levels based on sequential pairwise chi-squared tests: PGY 1 vs PGY 2 (χ(2) = 106.9, df = 3, p < 0.001); PGY 2 vs PGY 3 (χ(2) = 22.2, df = 3, p < 0.001); and PGY 3 vs PGY 4 (χ(2) = 56.4, df = 3, p < 0.001). Comparison of PGY 4 to PGY 5 scores were not significantly different (χ(2) = 4.5, df = 3, p = 0.21). For the 8 operations reviewed for interrater reliability, the intraclass correlation coefficient was 0.90 (95% CI: 0.72-0.98, p < 0.01). Correlation of Procedural Autonomy and Supervisions System ratings with both Operative Performance Rating System items (each r > 0.90, all p's < 0.01) and Ottawa Surgical Competency OR Evaluation items (each r > 0.86, all p's < 0.01) was high.
Conclusions:
The Zwisch scale can be used to make reliable and valid measurements of faculty guidance and resident autonomy. Our data also suggest that Zwisch ratings may be used to infer resident operative performance. Deployed on an automated smartphone-based system, it can be used to feasibly record evaluations for most operations performed by residents. This information can be used to council individual residents, modify programmatic curricula, and potentially inform national training guidelines.
... 1,2 The main goal of surgical education is to graduate residents who can operate independently; however, providing sufficient opportunity to build skills in a safe and efficient way can sometimes be difficult to achieve. 3,4 Adult learning and assessment scales Several scales were developed to assess resident autonomy for operative procedures and allow faculty to use this assessment information to help build and grant increasing resident autonomy. 3,4 Many of these assessment scales build off of earlier models of adult skills acquisition such as the classic theory described by Fitts and Posner,5 in which learners are characterized as progressing through cognitive, associative, and autonomous phases. ...
... 3,4 Adult learning and assessment scales Several scales were developed to assess resident autonomy for operative procedures and allow faculty to use this assessment information to help build and grant increasing resident autonomy. 3,4 Many of these assessment scales build off of earlier models of adult skills acquisition such as the classic theory described by Fitts and Posner,5 in which learners are characterized as progressing through cognitive, associative, and autonomous phases. 4,5 The cognitive phase is thought to require significant cognitive input and involves identifying the skills goals and the cognitive and motor steps necessary to achieve these goals. ...
... 12 There is existing evidence supporting the construct validity for the use of both of these scales in general surgery and obstetrics and gynecology. 13 Previous studies, for example, have demonstrated high interrater reliability, 3,8 the ability of each assessment scale to differentiate between junior and senior learners, 3,8 and for the Zwisch scale specifically, agreement with existing scales such as the Operative Performance Rating System and the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE). 3 Despite this evidence, there is a debate about the use of these scales, primarily focused on whether the expert level is achievable during residency, 4 and the crux of this debate centers on the concept of aspirational goals in surgical education and the description of the expert level. ...
Drawing upon key principles of adult learning theory, a number of surgical autonomy assessment scales have been developed for use in resident evaluation. These assessment scales allow for graded autonomy in resident surgical education, balancing patient safety with the need for achieving resident competency during training. The main scales utilized, the Zwisch scale and the Dreyfus scale, differ only around the inclusion of an “expert” level, and there is controversy in surgical education as to whether inclusion of this type of aspirational goals is appropriate. This clinical opinion article reviews key aspects of adult learning theory that pertain to surgical skills acquisition as well as use of aspirational goals in education and situates existing surgical autonomy assessment scales within this context. Existing evidence argues for the continued inclusion of aspirational goals in surgical education, but with a concomitant update to the surgical autonomy assessment scales to more closely align with the typical progression of surgical skills during residency. The current process for milestone evaluation put forth by the Accreditation Council for Graduate Medical Education (ACGME) provides an example of a potential framework that could be adapted for use in surgical skills assessment.
... Operative performance, autonomy, and complexity assessments SIMPL allows trainees and attending surgeons to assess trainee performance on five ordinal variables: critical deficiency, inexperienced with procedure, intermediate performance, practice-ready performance, and exceptional performance [14,15]. Resident autonomy is quantified by the Zwisch Scale, consisting of four additional ordinal levels: show and tell, active help, passive help or supervision only [14,16,17]. The Zwisch Scale has been shown to be a valid and reliable way to differentiate faculty guidance levels provided from which to infer resident autonomy [14,16,17]. ...
... Resident autonomy is quantified by the Zwisch Scale, consisting of four additional ordinal levels: show and tell, active help, passive help or supervision only [14,16,17]. The Zwisch Scale has been shown to be a valid and reliable way to differentiate faculty guidance levels provided from which to infer resident autonomy [14,16,17]. Operative cases were assessed by trainee and attending surgeon perception of case complexity relative to similar procedures: easiest third, average complexity, or hardest third [14]. ...
Background
Common bile duct exploration (CBDE) is safe and effective for managing choledocholithiasis, but most US general surgeons have limited experience with CBDE and are uncomfortable performing this procedure in practice. Surgical trainee exposure to CBDE is limited, and their learning curve for achieving autonomous, practice-ready performance has not been previously described. This study tests the hypothesis that receipt of one or more prior CBDE operative performance assessments, combined with formative feedback, is associated with greater resident operative performance and autonomy.
Methods
Resident and attending assessments of resident operative performance and autonomy were obtained for 189 laparoscopic or open CBDEs performed at 28 institutions. Performance and autonomy were graded along validated ordinal scales. Cases in which the resident had one or more prior CBDE case evaluations (n = 48) were compared with cases in which the resident had no prior evaluations (n = 141).
Results
Compared with cases in which the resident had no prior CBDE case evaluations, cases with a prior evaluation had greater proportions of practice-ready or exceptional performance ratings according to both residents (27% vs. 11%, p = .009) and attendings (58% vs. 19%, p < .001) and had greater proportions of passive help or supervision only autonomy ratings according to both residents (17% vs. 4%, p = .009) and attendings (69% vs. 32%, p < .01).
Conclusions
Residents with at least one prior CBDE evaluation and formative feedback demonstrated better operative performance and received greater autonomy than residents without prior evaluations, underscoring the propensity of feedback to help residents achieve autonomous, practice-ready performance for rare operations.
... The program matriculates and graduates two to three residents each year and incorporates graduated levels of responsibility throughout training. To support that goal, we introduced a framework for progressive autonomy (the Zwisch scale) at our training program in 2016 [21]. As part of this initiative, the Zwisch scale was posted on the operating room doors and conversations between faculty and trainees were encouraged. ...
... Beginning in 2021, the Tenwek Hospital general surgery department implemented the use of the Society for Improving Medical Professional Learning (SIMPL™) application, a workplace-based assessment platform that facilitates real-time operative feedback for surgical trainees [24]. This smartphone-based tool includes the use of the Zwisch scale as a measurement for autonomy (Table 1), with "passive help" or "supervision only" having been previously defined to indicate meaningful autonomy [21,25]. The SIMPL™ evaluation system includes three questions, answered by both faculty and trainee within 72 hours of an operation. ...
PurposeOperative autonomy during training is necessary to produce competent surgeons. A lack of autonomy may lessen trainee readiness for practice upon completion of training. Within training programs in Kenya, there is less faculty oversight, but the impact on autonomy has not been assessed. This study evaluates and describes trainee operative autonomy at a teaching hospital in Kenya where there is a deliberate attempt by faculty members to balance oversight and provide appropriate graduated autonomy.Methods
Autonomy was measured with the Zwisch scale utilizing the System for Improving and Measuring Procedural Learning (SIMPL™) application, a workplace-based assessment platform facilitating real-time feedback for surgical trainees. We reviewed the levels of autonomy perceived by trainees and faculty and compared autonomy by post-graduate year (PGY). We explored this relationship using ordinal mixed models with trainee PGY level and case complexity included as fixed effects and procedures as a random effect.ResultsFrom Jan 1, 2021, to Dec 31, 2021, 619 evaluations were completed by all 14 residents and 7 faculty in the general surgery program. By PGY, there was an increase in perceived granted autonomy by both faculty and trainees. On mixed modeling, trainees achieve more autonomy with advancing PGY [coefficient: 1.77 (95% CI 1.37–2.17), p<0.01]. Fourth- and fifth-year trainees achieved meaningful autonomy, defined on the Zwisch scale as “passive help” or “supervision only,” in 79% of operative evaluations (96 of 121).Conclusions
This study shows an increase in autonomy granted to trainees with more advanced training years. Meaningful autonomy can be achieved within surgical training, which should help to prepare surgeons for independent practice in Kenya.
... at easier portions of the case. [14][15][16][17] In this proposed model, the number of index cases that a resident must complete for graduation can be personalized based on the resident's ability to demonstrate complete autonomy and competency. 15,16 Currently, there is no universal way in which surgical educators evaluate resident competency within the operating room. ...
... Many rating scales have been developed to attempt to solve this problem, including the Objective Structured Assessment of Technical Skills (OSATS) 18 and the Zwisch scale. 14,17 The OSATS is an assessment tool for grading technical proficiency in surgery within 4 domains: respect for tissue, time and motion, instrument handling, and flow of operation and forward planning. The Zwisch scale rates resident autonomy level during an entire surgical case. ...
OBJECTIVE
There is no standard way in which physicians teach or evaluate surgical residents intraoperatively, and residents are proving to not be fully competent at core surgical procedures upon graduating. The Surgical Autonomy Program (SAP) is a novel educational model that combines a modified version of the Zwisch scale with Vygotsky’s social learning theory. The objective of this study was to establish preliminary validity evidence that SAP is a reliable measure of autonomy and a useful tool for tracking competency over time.
METHODS
The SAP breaks each surgical case into 4 parts, or zones of proximal development (ZPDs). Residents are evaluated on a 4-tier autonomy scale (TAGS scale) for each ZPD in every surgical case. Attendings were provided with a teaching session about SAP and identified appropriate ZPDs for surgical cases under their area of expertise. All neurosurgery residents at Duke University Hospital from July 2017 to July 2021 participated in this study. Chi-square tests and ordinal logistic regression were used for the analyses.
RESULTS
Between 2017 and 2021, there were 4885 cases logged by 27 residents. There were 30 attendings who evaluated residents using SAP. Faculty completed evaluations on 91% of cases. The ZPD of focus directly correlated with year of residency (postgraduate year) (χ ² = 1221.1, df = 15, p < 0.001). The autonomy level increased with year of residency (χ ² = 3553.5, df = 15, p < 0.001). An ordinal regression analysis showed that for every year increase in postgraduate year, the odds of operating at a higher level of independence was 2.16 times greater (95% CI 2.11–2.21, p < 0.001). The odds of residents performing with greater autonomy was lowest for the most complex portion of the case (ZPD3) (OR 0.18, 95% CI 0.17–0.20, p < 0.001). Residents have less autonomy with increased case complexity (χ ² = 160.28, df = 6, p < 0.001). Compared with average cases, residents were more likely to operate with greater autonomy on easy cases (OR 1.44, 95% CI 1.29–1.61, p < 0.001) and less likely to do so on difficult cases (OR 0.72, 95% CI 0.67–0.77, p < 0.001).
CONCLUSIONS
This study demonstrates preliminary evidence supporting the construct validity of the SAP. This tool successfully tracks resident autonomy and progress over time. The authors’ smartphone application was widely used among surgical faculty and residents, supporting integration into the perioperative workflow. Wide implementation of SAP across multiple surgical centers will aid in the movement toward a competency-based residency education system.
... La evaluación se realizó entre 2015-2016 mediante la escala de Zwisch, un instrumento validado internacionalmente para determinar el nivel de autonomía y supervisión operatoria. La escala permite evaluar cuatro niveles: i) demuestra y explica (el cirujano realiza el procedimiento al residente; ii) ayuda activa (el residente participa activamente en la cirugía sin realizar pasos críticos); iii) ayuda pasiva (el residente realiza el procedimiento y el cirujano lo asiste en partes críticas) y; iv) solo supervisión (el residente realiza el procedimiento con supervisión a distancia del cirujano o con ayuda de un residente menor) 43 . ...
Introducción. El currículo para la formación del cirujano general exige precisión, ajuste al contexto y factibilidad. En 2022, la World Society of Emergency Surgery formuló cinco declaraciones sobre el entrenamiento en cirugía digestiva mínimamente invasiva de emergencia que puede contribuir a estos propósitos. El objetivo del presente artículo fue examinar el alcance de estas declaraciones para la educación quirúrgica en Colombia. Métodos. Se analizó desde una posición crítica y reflexiva el alcance y limitaciones para Colombia de cada una de las declaraciones de la World Society of Emergency Surgery, con base en la evidencia empírica publicada durante las últimas dos décadas en revistas indexadas nacionales e internacionales. Resultados. La evidencia empírica producida en Colombia durante el presente siglo permite identificar que el país cuenta con fundamentos del currículo nacional en cirugía general, formulado por la División de Educación de la Asociación Colombiana de Cirugía en 2021; un sistema de acreditación de la educación superior; un modelo de aseguramiento universal en salud; infraestructura tecnológica y condiciones institucionales que pueden facilitar la adopción exitosa de dichas declaraciones para el entrenamiento de los futuros cirujanos en cirugía digestiva mínimamente invasiva de emergencia. No obstante, su implementación requiere esfuerzos mayores e inversión en materia de simulación quirúrgica, cooperación institucional y fortalecimiento del sistema de recertificación profesional. Conclusión. La educación quirúrgica colombiana está en capacidad de cumplir con las declaraciones de la World Society of Emergency Surgery en materia de entrenamiento en cirugía digestiva mínimamente invasiva de emergencia.
... Case complexity is determined by comparing to the "average" case: lowest 1/3 in complexity, average complexity (middle 1/3), and highest 1/3 in complexity. Case supervision is determined based on the Zwisch scale where 1-show and tell, 2-active help, 3-passive help, 4-supervision only [13]. Case performance is evaluated on a 5-point scale in which 1-critical deficiency, 2-inexperienced with procedure, 3-intermediate performance, 4practice-ready performance, 5-exceptional performance. ...
Background
Studies suggest that there are key differences in operative experience based on a trainee’s gender. A large-scale self-efficacy (SE) survey, distributed to general surgery residents after the American Board of Surgery In-Training Examination in 2020, found that female gender was associated with decreased SE in graduating PGY5 residents for all 4 laparoscopic procedures included on the survey (cholecystectomy, appendectomy, right hemicolectomy, and diagnostic laparoscopy). We sought to determine whether these differences were reflected at the case level when considering operative performance and supervision using an operative assessment tool (SIMPL OR).Methods
Supervision and performance data reported through the SIMPL OR platform for the same 4 laparoscopic procedures included in the SE survey were aggregated for residents who were PGY5s in 2020. Independent t-tests and multiple linear regression were used to determine the relationship between trainee gender and supervision/performance ratings.ResultsFor laparoscopic cases in aggregate (n = 2708), male residents rated their performance higher than females (3.57 vs. 3.26, p < 0.001, 1 = critical deficiency, 5 = exceptional performance) and reported less supervision (3.15 vs. 2.85, p < 0.001, 1 = show and tell, 4 = supervision only); similar findings were seen when looking at attending reports of resident supervision and performance. A multiple linear regression model showed that attending gender did not significantly predict resident-reported supervision or performance levels, while case complexity and trainee gender significantly affected both supervision and performance (p < 0.001).DiscussionFemale residents perceive themselves to be less self-efficacious at core laparoscopic procedures compared to their male colleagues. Comparison to more case-specific data confirm that female residents receive more supervision and lower performance ratings. This may create a domino effect in which female residents receive less operative independence, preventing the opportunity to establish SE. Further research should identify opportunities to break this cycle and consider gender identity beyond the male/female construct.Graphical abstract
... The Zwisch scale [3][4][5] was used for participants' self-evaluation of their operative autonomy for performing advanced pediatric endoscopic surgical procedures. This is a simple scale that rates the competency in performing the procedure according to four levels, as follows: ...
PurposeTo ensure the safe spread of pediatric endoscopic surgery, it is essential to build a training curriculum, and a survey of the current situation in Japan is necessary. The present study assessed an efficient training curriculum by clarifying instructor class pediatric surgeons’ experiences, including autonomy when performing advanced endoscopic surgeries.Methods
An online nationwide questionnaire survey was conducted among pediatric surgeons who had Endoscopic Surgical Skill Qualification (ESSQ) and board-certified instructors who had skills comparable to ESSQ. We assessed participants’ training experience, opinions concerning the ideal training curriculum, and the correlation between surgical experience and the level of autonomy. The Zwisch scale was used to assess autonomy.ResultsFifty-two participants responded to the survey (response rate: 86.7%). Only 57.7% of the respondents felt that they had received sufficient endoscopic surgery training. Most respondents considered an educational curriculum for endoscopic surgery including off-the-job training essential during the training period. Autonomy had been acquired after experiencing two to three cases for most advanced endoscopic surgeries.Conclusion
This first nationwide survey in Japan showed that instructor class pediatric surgeons acquired autonomy after experiencing two to three for most advanced endoscopic surgeries. Our findings suggest that training, especially off-the-job training, has been insufficient.
... The Zwisch scale self-evaluation The Zwisch scale [3,4,5] was used for participants' self-evaluation of their operative autonomy for performing advanced pediatric endoscopic surgical procedures. This is a simple scale that rates the competency in performing the procedure according to four levels, as follows: ...
Purpose: To ensure the safe spread of pediatric endoscopic surgery, it is essential to build a training curriculum, and a survey of the current situation in Japan is necessary. The present study assessed an efficient training curriculum by clarifying instructor class pediatric surgeons’ experiences, including autonomy when performing advanced endoscopic surgeries.
Methods: An online nationwide questionnaire survey was conducted among pediatric surgeons who had Endoscopic Surgical Skill Qualification (ESSQ) and board-certified instructors who had skills comparable to ESSQ. We assessed participants’ training experience, opinions concerning the ideal training curriculum and the correlation between surgical experience and the level of autonomy. The Zwisch scale was used to assess autonomy.
Results: Fifty-two participants responded to the survey (response rate: 86.7%). Only 57.7% of the respondents felt that they had received sufficient endoscopic surgery training. Most respondents considered an educational curriculum for endoscopic surgery including off-the-job training essential during the training period. Autonomy had been acquired after experiencing two to three cases for most advanced endoscopic surgeries.
Conclusion: This first nationwide survey in Japan showed that instructor class pediatric surgeons acquired autonomy after experiencing two to three for most advanced endoscopic surgeries. Our findings suggest that training, especially off-the-job training, has been insufficient.
(Present: 200/ Max. 200 words)
... Given the lack of proven alternatives, rater-based assessment remains the primary mechanism for performance evaluation in orthopedics. [9][10][11][12][13] The flaws of this current technical standard are especially pronounced in the assessment of psychomotor skills. 14,15 One specific core skill, fluoroscopic wire navigation, presents a unique challenge to assessors. ...
Psychomotor skill and decision‐making efficiency in surgical wire navigation can be objectively evaluated by analysis of intraoperative fluoroscopic image sequences. Prior work suggests that such image‐based behavior analysis of operating room (OR) performance can predict performer experience level (R ² = 0.62) and agree with expert opinion (the current standard) on the quality of a final implant construct (R ² = 0.59). However, it is unclear how objective image‐based evaluation compares with expert assessments for entire technical OR performances. This study examines the relationships between three key variables: (1) objective image‐based criteria, (2) expert opinions, and (3) performing surgeon experience level. A paired‐comparison survey of seven experts done based upon eight OR fluoroscopic wire navigation sequences shows that the experts’ preferences are best explained by objective metrics that reflect psychomotor and decision‐making behaviors which are counter‐productive to successful implant placement, like image count (R ² = 0.83) and behavior tally (R ² = 0.74). One such behavior, adjustments away from goal, uniquely correlated well with all three key variables: the fluoroscopic image‐based analysis composite score (R ² = 0.40), expert consensus (R ² = 0.76), and performer experience (R ² = 0.41). These results confirm that experts view less efficient technical behavior as indicative of lesser technical proficiency. However, while expert assessments of technical skill were reliable and consistent, neither individual nor consensus expert opinion appears to correlate with performer experience (R ² = 0.11).
This article is protected by copyright. All rights reserved.
... The System for Improving and Measuring Procedural Learning (SIMPL) is a smartphone-based app that utilizes the Zwisch scale, allowing for timely assessment from teaching faculty of intraoperative performance and autonomy. The Zwisch scale has been validated in the literature and can feasibly be implemented in various surgical specialty residency programs [10][11][12][13][14][15]. The purpose of this study was to identify the utility of the FLS exam in predicting improved operative performance, among general surgery residents using SIMPL evaluations. ...
Background
The American Board of Surgery made the Fundamentals of Laparoscopic Surgery (FLS) exam a prerequisite for board certification in 2009. Some residency programs have questioned the need for a continued FLS testing mandate given limited evidence that supports the impact of FLS on intraoperative skills. The Society for Improving Medical Professional Learning (SIMPL) app is a tool to evaluate resident intraoperative performance. We hypothesized that general surgery resident operative performance would improve immediately after preparing for the FLS exam.
Methods
The national public FLS data registry was matched with SIMPL resident evaluations from 2015 to 2021 and de-identified. SIMPL evaluations are scored in three categories: supervision required (Zwisch scale 1–4, 1 = show and tell and 4 = supervision only), performance (scale 1–5, 1 = exceptional and 5 = unprepared), and case complexity (scale 1–3, 1 = easiest and 3 = hardest). Statistical analyses compared pre and post-FLS exam resident average operative evaluation scores.
Results
There were a total of 76 general surgery residents, and 573 resident SIMPL evaluations included in this study. Residents required more supervision in laparoscopic cases performed before compared to after the FLS exam (2.84 vs. 3.03, respectively, p = 0.007). Residents performance scores improved from cases before compared to after the FLS exam (2.70 vs. 2.43, respectively, p = 0.001). Case complexity did not differ before versus after the FLS exam (2.13 vs. 2.18, respectively, p = 0.202). PGY level significantly predicted evaluation scores with a moderate correlation. A sub analysis grouped by PGY level revealed a significant improvement after the FLS exam in supervision among PGY-2 residents (2.33 vs. 2.58, respectively, p = 0.04) and performance among PGY-4 residents (2.67 vs 2.04, respectively, p < 0.001).
Conclusions
Preparation for, and passing, the FLS exam improves resident intraoperative laparoscopic performance and independence. We recommend taking the exam in the first two years of residency to enhance the laparoscopic experience for the remainder of training.
... 7 Multiple following studies showed reliability of this model for this purpose, mainly in general surgery. 4,5,11 Procedural numbers and operative logs showed to be poor predictor of residents' readiness and operative independence. 6,12 In this study, we showed the practicality and ease of adapting the Zwisch scoring system to the field of plastic surgery with good reliability. ...
Background:
The Zwisch scale is a recognized model for assessing surgical skills competency and autonomy, with paucity of its application in plastic surgery field. We look to assess the validity of utilizing this model for plastic surgery residency programs.
Methods:
This 6-month pilot study was a prospective study with attending surgeons from 3 different teaching hospitals. Two samples were obtained. The first sample contained all procedures performed by a participating faculty member and resident during a specified time period. The second sample was a subset of the first consisting of 15 procedures. The procedures were observed in person by 2 additional surgeons who rated the resident’s performance leveraging the Zwisch scores. A third surgeon utilized the operative skill assessment tool- Ottawa Surgical Competency OR Evaluation (O-SCORE) to provide ratings. All ratings were confidential.
Results:
Six residents and 5 attendings participated with a total number of 126 encounters for 184 procedures. The association between the Q1 (complexity of procedure- easiest, average, hardest) and Q2 (Zwisch level) was estimated with Spearman’s correlation and tested using Chi-square test. ICC and Spearman’s rank correlation coefficient demonstrated validity of the Zwisch scores in plastic surgery procedures with good reliability amongst attending surgeons with resident self-rating decreased score reliability (ICC Without self-rating 0.817, with self-rating 0.752).
Conclusion:
Plastic surgery cases are often more complex and have longer operative times making it difficult to evaluate residents. This research substantiates developing a specifically modified Zwisch scoring system for plastic surgery.
... Question 1 -Zwisch scale: The first question asks raters to use the Zwisch scale to assess the amount of guidance provided by the faculty, where faculty guidance is defined as the inverse of resident autonomy [25]. This scale consists of 4 levels, each representing progressively less faculty guidance: show and tell, active help, passive Help, and supervision only. ...
Purpose of Review
How today’s urology trainees acquire surgical skills has changed dramatically due to multiple forces placing strain on the graduate medical education mission. The development of workplace-based assessments that deliver feedback while capturing performance data has led to a paradigm shift toward individualized learning.
Recent Findings
Delivering feedback that drives surgical skill development requires the educator to provide a meaningful assessment of the learner after an operative experience. Workplace-based assessment involves direct observation of routine clinical practice and has become a central component of competency-based medical education.
Summary
Urology has the chance to fully embrace competency-based medical education, employing robust feedback mechanisms and workplace-based assessments. We must first define what it means to be a proficient urologist and design an assessment system that captures this collective sentiment. This can only be done through effective engagement and collaboration with stakeholders across our specialty.
... Foundational studies outlining the validity evidence for the Zwisch scale as a tool to measure resident operative autonomy incorporated faculty and observer raters but did not include resident self-rating [15]. With the growing evidence highlighting the discrepancy between faculty and resident autonomy scoring, questions can be raised around the interrater reliability of the Zwisch scale and, theoretically, the validity evidence supporting its internal structure. ...
PurposeThe goal of surgical residency is to produce technically competent surgeons who can safely operate autonomously. This study aimed to explore residents’ perception of intraoperative autonomy across post-graduate years (PGY).Methods
General surgery residents at a single academic institution were invited to participate in focus groups divided by PGY to explore their opinions on operative autonomy. Focus groups were audio recorded, transcribed, and analyzed using constant comparative technique. Thematic analysis was performed using an inductive approach.ResultsThirty-nine residents participated in five focus groups. Five themes emerged from the data. Residents distinguished between autonomy of thought representing graduated decision-making and autonomy of action representing technical maneuvers in the operating room. Residents vocalized a respect for the balance of autonomy and patient safety and stated a desire to have clear expectations to minimize the impact of external factors on autonomy.Conclusions
Residents differentiated autonomy as a parallel of autonomy of thought and autonomy of action and were empathetic to the responsibility of faculty to balance patient safety and autonomy. Surgical educators can improve resident autonomy by clearly managing expectations and minimizing external factors negatively affecting intraoperative autonomy.
... In the SIMPL app, operative autonomy is rated on the 4-level Zwisch scale ranging from "Show and Tell" to "Supervision Only" and operative performance is rated on the 5-level SIMPL performance scale ranging from "Unprepared/Critical Deficiency" to "Exceptional Performance." [14,15] Faculty are also given the opportunity to dictate narrative feedback in each evaluation. The resident or faculty may initiate an assessment after completing a procedure which is then sent directly to the other through the app, prompting a response [14,16]. ...
PurposeEngaging surgical residents and faculty with workplace-based assessment (WBA) are challenging. Gamification has helped engage trainees with simulation and online curricula but has not been used to encourage engagement with assessment. We explore the impact of gamification on engagement with WBA using the Society for Improving Medical Professional Learning (SIMPL) app.Methods
General surgery residency programs were divided into two intervention cohorts and a control cohort. The first intervention group received a weekly leaderboard of the most active residents and faculty engaged with SIMPL, while the second intervention group received leaderboards every other week. The control group did not undergo an intervention. For each cohort, resident and faculty engagement with SIMPL was measured, defined as the average number of evaluations submitted per day. Negative binomial mixed models were used to explore the mean number of evaluations completed by residents and faculty in each cohort while adjusting for program-specific factors.ResultsNine programs were included in the weekly intervention, nine programs in the biweekly intervention, and nineteen programs in the control group. Programs receiving a weekly leaderboard increased their average daily evaluations from 5.8 to 8.0 over the course of the intervention (p < 0.01). Average daily evaluations failed to increase for the biweekly intervention (8.2 to 7.3/day, p = 0.14) and control groups (2.7 to 2.5/day, p = 0.48).Conclusions
Sending a weekly leaderboard of resident and faculty activity in SIMPL had a positive impact on engagement with WBA compared to biweekly or no leaderboard interventions. Strategically applying gamification may improve engagement with WBA tools.
... The same authors [22] assessed the correlation of crowd OSATS or GOALS rating with the system for improving procedural learning (SIMPL) Zwisch and Performances scales. SIMPL is a smartphone based mobile application for the evaluation of operative performance and autonomy capturing three metrics, an autonomy metric (Zwisch scale), a difficulty scale and a performance metric [36]. Correlations between crowd-sourced ratings using GOALS and OATS and SIMPL global operative performance ratings tools were weak (GOALS/Zwisch r=-0.40; ...
Background: Crowd-Sourced Assessment of Technical Skills (C-SATS) is a surgical data management and learning platform that leverages the knowledge of large expert surgeon and lay groups to assess the technique and technical skills of surgeons in a highly efficient manner. The aim of this systematic review was to summarize published literature on the performance of C-SATS as compared to expert evaluations and assess its use as a training and validation tool in minimally invasive surgery (MIS).
... 3 Notably, the majority of available literature on surgical education tends to focus on topics such as assessment and feedback as well as evaluation of performance, mentorship, and leadership. [3][4][5][6]8,9 With regards to orthopaedic surgery specifically, the limited available literature tends to focus almost exclusively on fixation planning, leaving many other aspects of operative planning still unguided. 7 In addition, the American Board of Orthopaedic Surgery Knowledge, Skills, and Behavior Program will now be tracking resident assessments from attendings for-among other skills-proficiency with developing a surgical plan. ...
Surgical residents face significant demands to obtain the requisite skills and knowledge for their specialty by the time they graduate. Structured, guided, deliberate learning in a manner that is reproducible is crucial for surgical resident success and for ensuring residents will be able to meet the increasing demands of clinical practice. Use of an organized surgical planning tool, such as the PERFECT (Position, Exposure, Reduction, Fixation, Evaluation, Closure, and Therapy) model described in this article, is fundamental and indispensable to not only resident education, but to ones’ future career and to orthopaedic practice more broadly. This tool helps to emphasize important elements of a case, allows for mental rehearsal, serves as a custom educational resource for the resident, provides a framework for focused and intentional communication between the resident and attending surgeon, helps to encourage a critical appraisal of a procedure by all participants, and provides the resident guidance for future practice. Broadly instituting teaching frameworks such as the PERFECT model has demonstrated the potential to improve the standardization and overall learning experience for orthopaedic surgical residents, who continue to be tasked with learning more efficiently in a changing healthcare and education landscape.
... The learning process can be facilitated by giving instruction in alternative formats, such as online training, and offering additional opportunities to gain experience through simulations [42]. George et al. [44] also describe a four-step model in which instruction begins with "show and tell" and progresses through "active help", "passive help", and finally "supervision only." This echoes the hands-on opportunities we provided for students to watch demonstrations, learn collectively within their groups, and teach their peers under the scrutiny of experts in the field (Figure 8). ...
Wildland fire literacy is the capacity for wildland fire professionals to understand and communicate fundamentals of fuel and fire behavior within the socio-ecological elements of the fire regime. While wildland fire literacy is best developed through education, training, and experience in wildland fire science and management, too often, development among early-career professionals is deficient in one or more aspects of full literacy. We report on a hands-on prescribed fire methods workshop designed to provide training and experience in measuring and conducting prescribed fire, with a focus on grassland ecosystems. The workshop was held in March 2022 at The Nature Conservancy’s Dunn Ranch Prairie in northern Missouri. It consisted of hands-on training and experience in measuring fuels, fire weather, and fire behavior. Prescribed fire operations training facilitated both hands-on learning and vicarious learning by rotating squad roles among several small sub-units on the first day of live fire exercises. Participants then gained experience as crew members for two larger prescribed burns (60 and 200 ha). We report here on the successes and lessons learned from the perspectives of both participants and the instructor cadre for what was widely regarded as a successful workshop.
... [31][32][33] One such tool is the Zwisch scale to assess intraoperative performance. 34 Additionally, feedback is not one-sided. Residents should also receive tools to better utilize feedback and participate in the feedback process. ...
OBJECTIVE
Structured performance evaluations are important for the professional development and personal growth of resident learners. This process is formalized by the Accreditation Council for Graduate Medical Education milestones assessment system. The primary aim of this study was to understand the current feedback delivery mechanism by exploring the culture of feedback, the mechanics of delivery, and the evaluation of the feedback itself.
METHODS
Face-to-face interviews were conducted with 10 neurosurgery residents exploring their perceptions of summative feedback. Coded data were analyzed qualitatively for overriding themes using the matrix framework method. A priori themes of definition of feedback, feedback delivery, and impact of feedback were combined with de novo themes discovered during analysis.
RESULTS
Trainees prioritized formative over summative feedback. Summative and milestone feedback were criticized as being vague, misaligned with practice, and often perceived as erroneous. Barriers to implementation of summative feedback included perceived veracity of feedback, high interrater variability, and the inconstant adoption of a developmental progression model. Gender bias was noted in degree of feedback provided and language used.
CONCLUSIONS
Trainee perception of feedback provided multiple areas of improvement. This paper can serve as a baseline to study improvements in the milestone feedback process and optimize learning.
... The O-score includes an evaluation of 8 steps of the surgical procedure using a 5-point scale, as outlined in Table V. The P-score is a summative evaluation using a 5-level assessment based on the work of Zwischenberger 14,15 , as outlined in Table VI. To compare these 2 tools, the ABOS and AOA/CORD collaborated in a study from January to June 2017, comparing the O-tool and P-tool webbased evaluation assessments as completed by faculty 12 . ...
The American Board of Orthopaedic Surgery (ABOS) is the national organization charged with defining education standards for graduate medical education in orthopaedic surgery. The purpose of this article is to describe initiatives taken by the ABOS to develop assessments of competency of residents to document their progress toward the independent practice of orthopaedic surgery and provide feedback for improved performance during training. These initiatives are called the ABOS Knowledge, Skills, and Behavior Program. Web-based assessment tools have been developed and validated to measure competence. These assessments guide resident progress through residency education and better define the competency level by the end of training. The background and rationale for these initiatives and how they serve as steps toward competency-based education in orthopaedic residency education in the United States will be reviewed with a vision of a hybrid of time and competency-based orthopaedic residency education that will remain 5 years in length, with residents assessed using standardized tools.
General surgery training programs should provide equitable experiences to both categorical and preliminary residents, as up to 60% of non-designated preliminary general surgery residents will go on to complete surgical residencies. However, self-reported data suggest that preliminary general surgery residents perceive inferior operative experiences when compared with their categorical peers. In this study, we used workplace-based data to determine if there are differences in operative experiences between categorical and preliminary general surgery residents.
This study included operative evaluations for all categorical and preliminary PGY-1 and PGY-2 residents in the Society for Improving Medical Professional Learning (SIMPL) data registry from 2015 to 2023. Comparison of proportions and linear mixed models were used to compare frequency of feedback (both numerical and narrative) as well as mean operative autonomy and operative performance ratings between groups (categorical, designated preliminary, non-designated preliminary) within each PGY cohort.
49,737 faculty and resident operative evaluations from 65 institutions were included. Faculty were more likely to complete a SIMPL evaluation for categorical PGY-1’s than for designated or non-designated preliminary PGY-1’s (58 vs. 50% and 58 vs. 48%, respectively; both p < 0.01), and equally as likely to complete a SIMPL evaluation for categorical and preliminary PGY-2’s (54 vs. 52%, p = 0.11). Though performance ratings did not differ by PGY-1 group, both faculty and resident ratings of resident autonomy were lower for non-designated preliminary than categorical PGY-1’s (both p < 0.01). In the PGY-2 cohort, faculty ratings of performance and autonomy were higher for categorical compared to non-designated preliminary residents.
Though preliminary and categorical general surgery PGY-1’s have similar operative performance ratings, preliminary PGY-1’s (specifically, non-designated preliminary PGY-1’s) have less operative autonomy and are less likely to receive operative feedback than their categorical peers. For PGY-2 residents, preliminary residents continue to have lower operative autonomy ratings, though performance ratings are also lower compared to their categorical peers. More equitable operative experiences may accelerate preparation of preliminary residents for future careers in surgery or other procedural fields.
Over the past decade, medical education has shifted from a time-based approach to a competency-based approach for surgical training. This transition presents many new systemic challenges. The Society for Improving Medical Professional Learning (SIMPL) was created to respond to these challenges through coordinated collaboration across an international network of medical educators. The primary goal of the SIMPL network was to implement a workplace-based assessment and feedback platform. To date, SIMPL has developed, implemented, and sustained a platform that represents the earliest and largest effort to support workplace-based assessment at scale. The SIMPL model for collaborative improvement demonstrates a potential approach to addressing other complex systemic problems in medical education.
Assessment of trainees’ operative autonomy is challenging. However, workplace-based assessment (WBA) systems have made it possible to capture longitudinal data on trainees’ operative autonomy. We evaluated the ability of prior WBA operative autonomy ratings to predict future autonomy ratings for pediatric surgery fellows.
WBA data from two pediatric surgery training programs were analyzed using Bayesian mixed effects models to evaluate the relationship between prior cumulative autonomous operative experiences and the probability of being granted meaningful autonomy in a subsequent operation rated using a WBA. Cumulative autonomous experience was modeled as a fixed effect, while procedure, complexity, fellow, and attending surgeon were all modeled as random effects. Marginal predictions were generated and visualized to identify the number of prior autonomous operative experiences needed to achieve a 95% probability of being granted autonomy on a subsequent rating for three procedures: laparoscopic gastrostomy, laparoscopic inguinal hernia repair, and laparoscopic pyloromyotomy.
At site one, 843 of 1111 (76%) evaluations were rated as meaningfully autonomous for 9 fellows. At site two, 201 of 234 evaluations (86%) were rated as meaningfully autonomous for 3 fellows. Both sites identified similar expectations for autonomy based on cumulative autonomous experiences. The number of previously autonomous ratings needed to achieve a 95% probability of being granted meaningful autonomy for a subsequent procedure were 12 (site one and two) for laparoscopic gastrostomy, 10 (site one) and 14 (site two) for laparoscopic inguinal hernia repair, and 9 (site one) and 13 (site two) for laparoscopic pyloromyotomy.
Prior operative autonomy ratings appear effective in predicting the probability of being granted meaningful autonomy in subsequent procedures across multiple faculty and fellow groups. The approach demonstrated in this paper could support establishing minimum case number requirements and monitoring of fellows’ developing entrustability.
This Viewpoint suggests measures to improve surgical resident autonomy and thereby produce capable and resilient surgeons.
Our residents expressed dissatisfaction with operative autonomy and faculty feedback regarding technical skills. They reported variability among faculty regarding allowed operative autonomy. Our goals were to establish a shared mental model among residents and faculty regarding intraoperative performance expectations. We asked faculty to assign a level of expected autonomy (Zwisch scale) for various steps of common procedures according to the resident post-graduate year. Through an iterative process, the maps were standardized across service lines. The resulting "Autonomy Maps" were distributed to the faculty and residents. We held educational sessions and set expectations for use. Selected benchmarks were incorporated into resident end-of-rotation assessment forms. Initial operative case mapping identified variability in faculty expectations for a given post-graduate year and procedure. Residents reported improved satisfaction with understanding expectations regarding operative performance. Establishing autonomy benchmarks facilitated more specific feedback regarding residents' technical skills. Faculty expectations for resident operative autonomy are variable. Autonomy Maps provide structure for a shared mental model between faculty and residents for progressive operative autonomy and serve as a framework for expectations that improve resident satisfaction. Case-specific technical benchmarks are useful tools for assessing residents' technical milestones.
PurposeThere is growing concern over the readiness of orthopedic surgical residents and fellows for independent surgical practice upon completion of their training. This study aims to explore orthopedic surgery (OS) trainees’ experience of accessing operative autonomy by eliciting their perceptions and techniques implemented to gain autonomy.MethodsOS residents and fellows were invited to participate in focus group interviews via a convenience sampling approach. A non-faculty facilitator led the discussions using an interview guide to prompt conversation. All interviews were recorded, de-identified, and then transcribed. Three investigators iteratively analyzed transcripts to identify emerging themes until thematic saturation was achieved. All interviews were performed at Ohio State University Wexner Medical Center, an academic medical institution, in Columbus, Ohio.ResultsA total of 16 residents and 2 fellows participated. Two themes emerged: (1) optimal setting: trainees were allowed more operative autonomy in trauma and on-call cases than elective cases, though they perceived it was their responsibility to earn autonomy; (2) techniques: two techniques promote trainees’ access to autonomy, including trainee-initiated techniques (i.e., building relationship, preoperative planning, knowing attending preferences, and effective communication); and (3) faculty-initiated techniques (i.e., setting expectations, indications conference, and providing graduated autonomy).Conclusions
Our study findings suggest OS trainees tend to access least autonomy in elective OS cases. Although trainees perceived earning autonomy as their responsibility, faculty and resident development is recommended to enhance teaching and learning techniques to increase trainees’ practice readiness.
Objective:
Robotic-assisted surgery is an increasing part of general surgery training, but resident autonomy on the robotic platform can be hard to quantify. Robotic console time (RCT), the percentage of time the resident controls the console, may be an appropriate measure of resident operative autonomy. This study aims to characterize the correlation between objective resident RCT and subjectively scored operative autonomy.
Methods:
Using a validated resident performance evaluation instrument, we collected resident operative autonomy ratings from residents and attendings performing robotic cholecystectomy (RC) and robotic inguinal hernia repair (IH) at a university-based general surgery program between 9/2020-6/2021. We then extracted RCT data from the Intuitive surgical system. Descriptive statistics, t-tests and ANOVA were performed.
Results:
A total of 31 robotic operations (13 RC, 18 IH) performed by 4 attending surgeons and 8 residents (4 junior, 4 senior) were matched and included. 83.9% of cases were scored by both attending and resident. The average RCT per case was 35.6%(95% CI 13.0%,58.3%) for junior residents (PGY 2-3) and 59.7%(CI 51.1%,68.3%) for senior residents (PGY 4-5). The mean autonomy evaluated by residents was 3.29(CI 2.85,3.73) out of a maximum score of 5, while the mean autonomy evaluated by attendings was 4.12(CI 3.68,4.55). RCT significantly correlated with subjective evaluations of resident autonomy (r=0.61, p=0.0003). RCT also moderately correlated with resident training level (r=0.5306, p<0.0001). Neither attending robotic experience nor operation type significantly correlated with RCT or autonomy evaluation scores.
Conclusions:
Our study suggests that resident console time is a valid surrogate for resident operative autonomy in robotic cholecystectomy and inguinal hernia repair. RCT may be a valuable measure in objective assessment of residents' operative autonomy and training efficiency. Future investigation into how RCT correlates with subjective and objective autonomy metrics such as verbal guidance or distinguishing critical operative steps is needed to validate the study findings further.
Background:
Operative meaningful trainee autonomy is an essential component of surgical training. Reduced trainee autonomy is frequently attributed to patient safety concerns, but this has not been examined within Kenya. We aimed to assess whether meaningful trainee autonomy was associated with a change in patient outcomes.
Methods:
We investigated whether meaningful trainee autonomy was associated with a change in severe postoperative complications and all-cause in-hospital mortality in a previously described cohort undergoing emergency gastrointestinal operations. Each operation was reviewed to determine the presence of meaningful autonomy, defined as "supervision only" from faculty. Comparisons were made between faculty-led cases and cases with meaningful trainee autonomy. Multilevel logistic regression models were created for the outcomes of mortality and complications with the exposure of meaningful trainee autonomy, accounting for fixed effects of the Africa Surgical Outcomes Study Risk Score and random effects of discharge diagnoses.
Results:
After excluding laparoscopy (N = 28) and missing data (N = 3), 451 operations were studied, and 343 (76.1%) had meaningful trainee autonomy. Faculty were more involved in operations with older age, cancer, prior complications, and higher risk scores. On unadjusted analysis, meaningful trainee autonomy was associated with mortality odds of 0.32 (95% confidence interval: 0.17-0.58) compared with faculty-led operations. Similarly, the odds of developing complications were 0.52 (95% confidence interval: 0.32-0.84) with meaningful trainee autonomy compared with faculty-led operations. When adjusting for Africa Surgical Outcomes Study Score and clustering discharge diagnoses, the odds of mortality (odds ratio 0.58; 95% confidence interval: 0.27-1.2) and complication (odds ratio 0.83; 95% confidence interval: 0.47-1.5) were not significant.
Conclusion:
Our findings support that increasing trainee autonomy does not change patient outcomes in selected emergency gastrointestinal operations. Further, trainees and faculty appropriately discern patients at higher risk of complications and mortality, and the selective granting of trainee autonomy does not affect patient safety.
Introduction:
As medical education systems increasingly move toward competency-based training, it is important to understand the tools available to assess competency and how these tools are utilized. The Society for Improving Medical Professional Learning (SIMPL) offers a smart phone-based assessment system that supports workplace-based assessment of residents' and fellows' operative autonomy, performance, and case complexity. The purpose of this study was to characterize implementation of the SIMPL app within vascular surgery integrated residency (0+5) and fellowship (5+2) training programs.
Methods:
SIMPL operative ratings recorded between 2018-2022 were collected from all participating vascular surgery training institutions (n=9 institutions with 5+2 and 0+5 programs, n=4 institutions with 5+2 program only). The characteristics of programs, trainees, faculty, and SIMPL operative assessments were evaluated using descriptive statistics.
Results:
Operative assessments were completed for 2,457 cases by 85 attendings and 86 trainees totaling 4,615 unique operative assessment ratings. Attendings included dictated feedback in 52% of assessments. Senior-level residents received more assessments than junior-level residents (PGY 1-3 n=439, PGY4-5 n=551). Performance ratings demonstrated increases from junior to senior trainees for both resident and fellow cohorts with "performance ready" or "exceptional performance" ratings increasing by nearly 2-fold for PGY1 to PGY5 residents (28.1% vs 40.6%), and from first to second year fellows (PGY 6 46.7%, PGY 7 60.3%). Similar gains in autonomy were demonstrated as trainees progressed through training. Senior residents were more frequently granted autonomy with "supervision only" than junior residents (PGY1 8.7%, PGY5 21.6). "Supervision only" autonomy ratings were granted to 21.8% of graduating fellows. Assessment data included a greater proportion of complex cases for senior compared to junior fellows (PGY6 20.9% vs PGY 7 26.5%). Program Directors felt that faculty and trainee buy-in were the main barriers to implementation of the SIMPL assessment app.
Conclusions:
This is the first description of the SIMPL app as an operative assessment tool within vascular surgery which has been successfully implemented in both residency and fellowship programs. The assessment data demonstrates expected progressive gains in trainees' autonomy and performance, as well as increasing case complexity, across PGY years. Given the selection of SIMPL as the assessment platform for required American Board of Surgery and Vascular Surgery Board EPA assessments, understanding facilitators and barriers to implementation of workplace-based assessments using this app is imperative, particularly as we move toward competency-based medical education.
Background:
Trainee autonomy has eroded over time as surgery has become more subspecialized and as attending oversight has increased, causing many trainees to seek additional fellowship training beyond residency. Less clear is whether there are cases that attendings view as "fellowship-level" or "privileged" cases in which resident-level trainees should not have high levels of autonomy due to complexity or high-stakes outcomes.
Objective:
We sought to better understand current attitudes and practices with regards to trainee autonomy in hypospadias repair as it represents a high complexity procedure within pediatric urology.
Study design:
We administered a RedCap survey to the SPU membership, asking respondents to describe the level of autonomy afforded to trainees in various types of hypospadias repair (distal, midshaft, proximal, perineal) as measured by the Zwisch scale. The Zwisch scale describes the role of the attending in the attending-trainee relationship in a low-to-high trainee autonomy fashion: show and tell; active help; passive help; supervision only.
Results:
177 of 761 (23%) unique recipients completed our survey and 174 of 177 (98%) of respondents felt that trainees should not perform hypospadias repair independently in practice without additional fellowship training. Among pediatric urologists who train residents, trainee autonomy as measured by the Zwisch scale decreased as the type of hypospadias repair moved from distal to proximal.
Discussion:
There was near unanimous agreement among respondents that urology trainees should not perform hypospadias repair in practice without additional pediatric urology fellowship training, and that current practice affords little trainee autonomy in hypospadias repair at the resident level. These findings introduce a new wrinkle into the issue of trainee autonomy: cases in which trainees perhaps should not have autonomy. Concurrently, the concern with such findings is that this intentional lack of autonomy may extend to other urologic procedures that one would expect trainees to be able to perform independently.
Conclusion:
Urology trainees are not expected to be able to perform hypospadias in practice without additional training. This raises the question that there may be other such procedures in urology, and if so, should we as instructors, be forthcoming about the limitations of urology residency training to set appropriate trainee expectations?
Introduction:
A 2020 survey of post-graduate year 5 (PGY5) general surgery residents linked to the American Board of Surgery In-Training Examination (ABSITE) revealed significant deficits in self-efficacy (SE), or personal judgment of one's ability to complete a task, for 10 commonly performed operations. Identifying whether this deficit is similarly perceived by program directors (PDs) has not been well established. We hypothesized that PDs would perceive higher levels of operative SE compared to PGY5s.
Methods:
A survey was distributed through the Association of Program Directors in Surgery listserv; PDs were queried about their PGY5 residents' ability to perform the same 10 operations independently and their accuracy of patient assessments and operative plans for components of several core entrustable professional activities (EPAs). Results of this survey were compared to PGY5 residents' perception of their SE and entrustment based on the 2020 post-ABSITE survey. Chi-squared tests were used for statistical analysis.
Results:
108 responses were received, representing ∼32% (108/342) of general surgery programs. Perceptions from PDs of PGY5 residents' operative SE were highly concordant with resident perceptions; no significant differences were observed for 9 of 10 procedures. Both PGY5 residents and PDs perceived adequate levels of entrustment; no significant differences were observed for 6 of 8 EPA components.
Conclusions:
These findings show concordance between PDs and PGY5 residents in their perceptions of operative SE and entrustment. Though both groups perceive adequate levels of entrustment, PDs corroborate the previously described operative SE deficit, illustrating the importance of improved preparation for independent practice.
Background:
Feedback and assessment are difficult to provide in the emergency department (ED) setting despite their critical importance for competency-based education, and traditional end-of-shift evaluations (ESEs) alone may be inadequate. The SIMPL (Society for Improving Medical Professional Learning) mobile application has been successfully implemented and studied in the operative setting for surgical training programs as a point-of-care tool that incorporates three assessment scales in addition to dictated feedback. SIMPL may represent a viable tool for enhancing workplace-based feedback and assessment in emergency medicine (EM).
Methods:
We implemented SIMPL at a 4-year EM residency program during a pilot study from March to June 2021 for observable activities such as medical resuscitations and related procedures. Faculty and residents underwent formal rater training prior to launch and were asked to complete surveys regarding the SIMPL app's content, usability, and future directions at the end of the pilot.
Results:
A total of 36/58 (62%) of faculty completed at least one evaluation, for a total of 190 evaluations and an average of three evaluations per faculty. Faculty initiated 130/190 (68%) and residents initiated 60/190 (32%) evaluations. Ninety-one percent included dictated feedback. A total of 45/54 (83%) residents received at least one evaluation, with an average of 3.5 evaluations per resident. Residents generally agreed that SIMPL increased the quality of feedback received and that they valued dictated feedback. Residents generally did not value the numerical feedback provided from SIMPL. Relative to the residents, faculty overall responded more positively toward SIMPL. The pilot generated several suggestions to inform the optimization of the next version of SIMPL for EM training programs.
Conclusions:
The SIMPL app, originally developed for use in surgical training programs, can be implemented for use in EM residency programs, has positive support from faculty, and may provide important adjunct information beyond current ESEs.
Objectives:. This study tests the null hypotheses that overall sentiment and gendered words in verbal feedback and resident operative autonomy relative to performance are similar for female and male residents.
Background:. Female and male surgical residents may experience training differently, affecting the quality of learning and graduated autonomy.
Methods:. A longitudinal, observational study using a Society for Improving Medical Professional Learning collaborative dataset describing resident and attending evaluations of resident operative performance and autonomy and recordings of verbal feedback from attendings from surgical procedures performed at 54 US general surgery residency training programs from 2016 to 2021. Overall sentiment, adjectives, and gendered words in verbal feedback were quantified by natural language processing. Resident operative autonomy and performance, as evaluated by attendings, were reported on 5-point ordinal scales. Performance-adjusted autonomy was calculated as autonomy minus performance.
Results:. The final dataset included objective assessments and dictated feedback for 2683 surgical procedures. Sentiment scores were higher for female residents (95 [interquartile range (IQR), 4–100] vs 86 [IQR 2–100]; P < 0.001). Gendered words were present in a greater proportion of dictations for female residents (29% vs 25%; P = 0.04) due to male attendings disproportionately using male-associated words in feedback for female residents (28% vs 23%; P = 0.01). Overall, attendings reported that male residents received greater performance-adjusted autonomy compared with female residents (P < 0.001).
Conclusions:. Sentiment and gendered words in verbal feedback and performance-adjusted operative autonomy differed for female and male general surgery residents. These findings suggest a need to ensure that trainees are given appropriate and equitable operative autonomy and feedback.
Background:
Laparoscopic common bile duct exploration is safe and effective for managing choledocholithiasis, but laparoscopic common bile duct exploration is rarely performed, which threatens surgical trainee proficiency. This study tests the hypothesis that prior operative or simulation experience with laparoscopic common bile duct exploration is associated with greater resident operative performance and autonomy without adversely affecting patient outcomes.
Methods:
This longitudinal cohort study included 33 consecutive patients undergoing laparoscopic common bile duct exploration in cases involving postgraduate years 3, 4, and 5 general surgery residents at a single institution during the implementation of a laparoscopic common bile duct exploration simulation curriculum. For each of the 33 cases, resident performance and autonomy were rated by residents and attendings, the resident's prior operative and simulation experience were recorded, and patient outcomes were ascertained from electronic health records for comparison among 3 cohorts: prior operative experience, prior simulation experience, and no prior experience.
Results:
Operative approach was similar among cohorts. Overall morbidity was 6.1% and similar across cohorts. The operative performance scores were higher in prior experience cohorts according to both residents (3.0 [2.8-3.0] vs 2.0 [2.0-3.0]; P = .01) and attendings (3.0 [3.0-4.0]; P < .001). The autonomy scores were higher in prior experience cohorts according to both residents (2.0 [2.0-3.0] vs 2.0 [2.0-2.0]; P = .005) and attendings (2.5 [2.0-3.0] vs 2.0 [1.0-2.0]; P = .001). Prior simulation and prior operative experience had similar associations with performance and autonomy.
Conclusion:
Simulation experience with laparoscopic common bile duct exploration was associated with greater resident operative performance and autonomy, with effects that mimic prior operative experience. This illustrates the potential for simulation-based training to improve resident operative performance and autonomy for laparoscopic common bile duct exploration.
The use of a robotic surgical platform has become common place in thoracic surgery programs throughout the United States. Formal training paradigms need to be reevaluated to allow for effective and efficient training of thoracic surgery residents and fellows. The utilization of video-based coaching and simulation are effective adjuncts in robotics training.
BACKGROUND. Universities are obliged to ensure that dental graduates possess the necessary skills to render safe and effective treatment. Empirical evidence regarding the development of safe and effective independent practice at undergraduate level is unfortunately lacking. OBJECTIVES. To measure final-year students' abilities to correctly perform exodontia (tooth removal/extraction) skills independently, based on the applied postgraduate progressive independence theory. METHODS. Fourteen clinical teachers systematically assessed 13 263 tooth extractions completed by final-year dental students (2014 - 2016). An independence ratio (extractions performed without assistance/total number of extractions) was used as the key performance indicator to provide feedback on the ability to complete procedures independently over time. A customised index was used for controlling the level of difficulty. RESULTS. Final-year students (n=146) achieved independence ratios ranging between 90% and 94% (standard deviation 3.3%) by the end of their clinical training. Logical gradients of increased independence were illustrated with time, as well as variable performance among students. The level of difficulty index scores remained similar within cohorts per year of study. Acceptable assessment differences existed between clinical teachers, which could largely be explained by complex operational circumstances. CONCLUSIONS. As far as we are aware, this is the first study that quantified progressive independence in exodontia for undergraduate students. The measure was sensitive enough to show logical independence gradients and variance among students. Final-year students demonstrated that they could remove >8/10 teeth independently by the time of their graduation. The measure shows promise as a proxy of competence for skills that are often practised. It is recommended that factors that influence these measurements be examined in more detail.
The goal of graduate surgical education is to ensure that the graduate is competent to practice in his or her chosen specialty. Traditionally, surgical learning has been based on an apprenticeship model; that is, the long-term observation and assessment of the trainee over a prolonged period of time. Patient expectations, work hour restrictions, and expectations of increased faculty oversight have led to decreased resident autonomy and independence. Graduates completing surgical training with less surgical autonomy may have lower clinical competence, which may affect patient safety, patient outcomes, and career satisfaction. This will require the modification of current assessment and training methods.
Background
As surgical training shifts toward a competency-based paradigm, deliberate practice for procedures must be a point of focus. The purpose of this study was to assess the impact of an educational time-out intervention on educational experience and operative performance in endocrine surgery.
Methods
For 12 months, third-year general surgery residents used the educational time-out to establish an operative step of focus for thyroidectomy and parathyroidectomy procedures. Data were collected using the System for Improving and Measuring Procedural Learning application and post-rotation surveys. The Zwisch scale was used to classify supervision, with meaningful autonomy defined as passive help or supervision only.
Results
Eight residents and 3 attending surgeons performed the educational time-out for a total of 211 operations (93% completion rate). At the end of each rotation, there was improvement in the frequency of goal setting. There was strong agreement (90%) that the intervention strengthened the educational experience. For most cases (52%), the residents were rated at active help. Residents performed a median of 3/6 thyroidectomy steps at meaningful autonomy and a median of 2/5 parathyroidectomy steps at meaningful autonomy. Review of the qualitative data revealed that optimal feedback was provided in 46% of cases.
Conclusion
The educational time-out strengthened educational experiences. Stepwise procedural data revealed the varying levels of supervision that exist within an operation. Broader implementation of this intervention could facilitate competency-based procedural education.
Congenital heart surgery (CHS) is technically demanding, and its training is extremely complex and challenging. Training of the surgeon's technical skills has relied on a preceptorship format in which the trainees are gradually exposed to patients in the operating room under the close tutelage of senior staff surgeons. Training in the operating room is an inefficient process and the concept of a learning curve is no longer acceptable in terms of patient outcomes. The benefits of surgical simulation in training of congenital heart surgeons are well known and appreciated. However, adequate surgical simulation models and equipment for training have been scarce until the recent development of three-dimensionally (3D) printed models. Using comprehensive 3D printing and silicone-molding techniques, realistic simulation training models for most congenital heart surgical procedures have been produced. Newly developed silicone-molded models allow efficient CHS training in a stress-free environment with instantaneous feedback from the proctors and avoids risk to patients. The time has arrived when all congenital heart surgeons should consider surgical simulation training before progressing to real-life operating in a similar fashion to the aviation industry where all pilots are required to complete simulation training before flying a real aircraft. It is argued here that simulation training is not an option anymore but should be a mandatory component of CHS training.
Objective:
Amidst the current opioid crisis, there is a need for better integration of substance use disorder screening and treatment across specialties. However, there is no consensus regarding how to best instruct OBGYN trainees in the clinical skills related to opioid and other substance use disorders (SUD). Study objectives were (1) to assess the effectiveness a SUD curriculum to improve self-reported competence among OBGYN residents and (2) to explore its effectiveness to improve attending evaluations of residents' clinical skills as well as its feasibility and acceptability from the resident perspective.
Methods:
A pilot 3-session curriculum was developed and adapted to SUD screening and treatment which included readings, didactics, and supervised outpatient clinical experiences for OBGYN post-graduate year 1 (PGY-1) residents rotating through an integrated OBGYN-SUD clinic. Eighteen residents completed pre and post clinical skills self-assessments (SUD screening, counseling, referring, Motivational Interviewing) using an adapted Zwisch Rating Scale (range 1-5). Scores were compared between time points using paired t-tests. Sub-samples also (a) were evaluated by the attending on three relevant Accreditation Council for Graduate Medical Education Milestones (ACGME) milestone sets using the web-based feedback program, myTIPreport (n = 10) and (b) completed a qualitative interview (n = 4).
Results:
All PGY-1s (18/18) across three academic years completed the 3-session SUD curriculum. Clinical skill self-assessments improved significantly in all areas [SUD Screening (2.44 (0.98) vs 3.56 (0.62), p = <0.01); Counseling (1.81 (0.71) vs 3.56 (0.51), p = < .01; Referring (2.03 (0.74) vs 3.17 (0.71), p = < .01; Motivational Interviewing (1.94 (1.06) vs 3.33 (0.69), p = < .01)]. Milestone set levels assigned by attending evaluations (n = 10) also improved. Qualitative data (n = 4) revealed high acceptability; all curriculum components were viewed positively, and feedback was provided (e.g., desire for more patient exposures).
Conclusion:
A pilot SUD curriculum tailored for OBGYN PGY-1 residents that goes beyond opioid prescribing to encompass SUD management is feasible, acceptable and likely effective at improving SUD core clinical skills.
Background:
Competence by design (CBD) residency programs increasingly depend on tools that provide reliable assessments, require minimal rater training, and measure progression through the CBD milestones. To assess intraoperative skills, global rating scales and entrustability ratings are commonly used but may require extensive training. The Competency Continuum (CC) is a CBD framework that may be used as an assessment tool to assess laparoscopic skills. The study aimed to compare the CC to two other assessment tools: the Global Operative Assessment of Laparoscopic Skills (GOALS) and the Zwisch scale.
Methods:
Four expert surgeons rated thirty laparoscopic cholecystectomy videos. Two raters used the GOALS scale while the remaining two raters used both the Zwisch scale and CC. Each rater received scale-specific training. Descriptive statistics, inter-rater reliabilities (IRR), and Pearson's correlations were calculated for each scale.
Results:
Significant positive correlations between GOALS and Zwisch (r = 0.75, p < 0.001), CC and GOALS (r = 0.79, p < 0.001), and CC and Zwisch (r = 0.90, p < 0.001) were found. The CC had an inter-rater reliability of 0.74 whereas the GOALS and Zwisch scales had inter-rater reliabilities of 0.44 and 0.43, respectively. Compared to GOALS and Zwisch scales, the CC had the highest inter-rater reliability and required minimal rater training to achieve reliable scores.
Conclusion:
The CC may be a reliable tool to assess intraoperative laparoscopic skills and provide trainees with formative feedback relevant to the CBD milestones. Further research should collect further validity evidence for the use of the CC as an independent assessment tool.
Introduction
Gender disparities in resident operative experience have been described; however, their etiology is poorly understood, and racial/ethnic disparities have not been explored. This study investigated the relationship between gender, race/ethnicity, and surgery resident case volumes.
Materials and methods
A retrospective analysis of graduating general surgery resident case logs (2010-2020) at an academic medical center was performed. Self-reported gender and race/ethnicity data were collected from program records. Residents were categorized as underrepresented in medicine (URM) (Black, Hispanic, Native American) or non-URM (White, Asian). Associations between gender and URM status and major, chief, and teaching assistant (TA) mean case volumes were analyzed using t-tests.
Results
The cohort included 80 residents: 39 female (48.8%) and 17 URM (21.3%). Compared to male residents, female residents performed fewer TA cases (33 versus 47, P < 0.001). Compared to non-URM residents, URM residents graduated with fewer major (948 versus 1043, P = 0.008) and TA cases (32 versus 42, P = 0.038). Male URM residents performed fewer TA cases than male non-URM residents (32 versus 50, P = 0.031). Subanalysis stratified by graduation year demonstrated that from 2010 to 2015, female residents performed fewer chief (218 versus 248, P = 0.039) and TA cases (29 versus 50, P = 0.001) than male residents. However, from 2016 to 2020, when gender parity was achieved, no significant associations were observed between gender and case volumes.
Conclusions
Female and URM residents perform fewer TA and major cases than male non-URM residents, which may contribute to reduced operative autonomy, confidence, and entrustment. Prioritizing gender and URM parity may help decrease case volume gaps among underrepresented residents.
OBJECTIVE
We hypothesized that a Chief Resident Service educational model provides safe care for patients compared to that received on standard academic services where rotating residents adopt the practices and preferences of their attending.
DESIGN
We retrospectively identified patients undergoing inguinal hernia repairs from July 2016 through June 2019 and matched Chief's service patients to standard academic service patients 1:1 on CPT, sex and age. We compared patient characteristics, recurrence rates, outcomes and complications.
SETTING
Tertiary care center, single institution.
PARTICIPANTS
Overall, 77 patients undergoing inguinal hernia repairs (66% open and 34% laparoscopic) on the Chief's service matched successfully to 77 standard academic service patients during the study period.
RESULTS
Age, BMI and ASA were similar between the services, but Chief's service patients were less likely to be current smokers (1.3% vs. 24.7%) and more likely to be former smokers (59.7% vs. 26.0%) than standard academic service patients (p < 0.01). Patients presenting with incarcerated hernias (5.2% vs. 9.1%), recurrent (10.4% vs. 5.2%) and bilateral hernias (19.5% vs. 10.4%) were similar between the Chief's service and standard academic services, respectively (all p > 0.05). Operative times were longer for the Chief's service for open (123 min vs. 67, p < 0.01) and laparoscopic (112 min vs. 79, p = 0.02) repairs. Recurrence rates (6.5% vs. 3.9%, p = 0.47) and complications including infection, seroma or hematoma requiring evacuation and need for reoperation were similarly low (p > 0.05) between the Chief's and standard academic services, respectively. Despite low complication rates, Chief's service patients were more likely to present to the ED post-op (14.3% vs. 1.3%; p = 0.001), but readmission rates were similarly low (2.6% vs. 0%, p = 0.09).
CONCLUSIONS
Providing general surgery chief residents with a supervised opportunity to direct, plan and provide surgical care in clinic and the operating room, as a transition to independent practice following graduation, is safe for patients presenting with inguinal hernias. Concerns about patient safety should not be a barrier to maximizing entrustment for the evaluation and operative management of select core general surgery diagnoses and operations.
Importance: Concern regarding surgical trainees’ operative autonomy has increased in recent years, emphasizing patient safety and preparation for independent practice. Regarding abdominal wall reconstruction (AWR), long-term outcomes of fellow autonomy have yet to be delineated.
Objectives: To evaluate the long-term outcomes of AWRs performed by fellows and compare them with those of AWRs performed by assistant, associate, and senior-level professors.
Design, Setting, and Participants: This retrospective cohort study included patients who underwent AWR for ventral hernias or repair of tumor resection defects at a 710-bed tertiary cancer center between March 1, 2005, and June 30, 2019. The analysis was conducted between January 2020 and December 2021.
Exposure: Academic rank of primary surgeon.
Main Outcomes and Measures: The primary outcome was hernia recurrence. Secondary outcomes were surgical site occurrence, surgical site infection, length of hospital stay, unplanned return to the operating room, and 30-day readmission. Multivariable hierarchical models were constructed to identify predictive factors.
Results: Of 810 consecutive patients, 720 (mean [SD] age, 59.8 [11.5] years; 375 female [52.1%]) met the inclusion criteria. Mean (SD) body mass index was 31.4 (6.7), and mean (SD) follow-up time was 42 (29) months. Assistant professors performed the most AWRs (276 [38.3%]), followed by associate professors (169 [23.5%]), senior-level professors (157 [21.8%]), and microsurgical fellows (118 [16.4%]). Compared with fellows and more junior surgeons, senior-level professors tended to operate on significantly older patients (mean [SD] age, 59.9 [10.9] years; P = .03), more patients with obesity (103 [65.6%]; P = .003), and patients with larger defects (247.9 [216.0] cm; P < .001), parastomal hernias (27 [17.2%]; P = .001), or rectus muscle violation (53 [33.8%]; P = .03). No significant differences were found for hernia recurrence, surgical site occurrence, surgical site infection, 30-day readmission rates, or length of stay among the fellows and assistant, associate, and senior-level professors in adjusted models. Compared with fellows, assistant professors (OR, 0.22; 95% CI, 0.08-0.64) and senior-level professors (OR, 0.20; 95% CI, 0.06-0.69) had lower rates of unplanned return to the operating room.
Conclusions and Relevance: This cohort study provides evidence-based reassurance that providing fellows with autonomy in performing AWRs does not compromise long-term patient outcomes. These findings may incite efforts to increase appropriate surgical trainee autonomy, thereby empowering future generations of competent, independent surgeons.
Background:
A subset of Entrustable Professional Activities (EPAs) has been developed for general surgery. We aim to contribute validity evidence for EPAs as an assessment framework for general surgery residents, including concurrent validity compared to ACGME milestones, the current gold standard for evaluating competency.
Study design:
This is a cross-sectional study in a general surgery training program within a tertiary academic medical center. EPA assessments were submitted using a mobile app and scored on a numerical scale, mirroring milestones. EPA score distribution was analyzed with respect to post-graduate year (PGY) level and phase of care. Proportional odds logistic regression identified significant predictors. Spearman rank and Wilcoxon rank tests were used for comparisons with milestone ratings.
Results:
From August 2018 to December 2019, 320 assessments were collected. EPA scores increased by PGY level. Operative phase EPA scores were significantly lower than nonoperative phase scores. PGY level, operative phase, and case difficulty significantly influenced entrustment scoring. EPA scores demonstrated strong correlation with nonoperative milestones patient care-1, medical knowledge-1, interpersonal and communication skills-2, interpersonal and communication skills-3, professionalism-1, professionalism-3, and practice-based learning and improvement-2 (ρ > 0.5, p < 0.05) and a weaker correlation with operative milestones patient care-3 and medical knowledge-2 (ρ < 0.5, p < 0.05).
Conclusions:
The influence of PGY level and operative phase on entrustment scoring supports the validity of EPAs as a formative evaluation framework for general surgery resident performance. In addition, evident correlations between EPA scores and respective milestone ratings provide concurrent validity evidence.
Introduction
Leadership is necessary for effective health care teams, particularly for surgeons. Trainees similarly must acquire foundational leadership skills to maximize effectiveness. However, surgical leadership is rarely formally assessed, particularly for junior trainees. We aimed to establish themes of communication, perception and engagement styles, as well as strengths and weaknesses among junior surgical residents at a single institution.
Methods
The Data Dome Inc. (datadome.com) DISC personality assessment was administered in 2018-2021 to junior residents at an academic general surgery training program at a single institution. Resident demographics were recorded, and themes from deidentified reports were analyzed by year (PGY-1 and PGY-2) using JMP 16 Pro Text Explorer.
Results
PGY-1 communication was most frequently described as “accomplished best by well-defined avenues” with “duties and responsibilities of others who will be involved explained” in “friendly terms.” PGY-2 communication involved “deal [ing] with people,” “strong feelings about a particular problem,” and being “good at giving verbal and nonverbal feedback.” In ideal environments, PGY-1s self-perceived as “good listener [s],” “good-natured,” and “team player [s].” However, under stress, PGY-1s were perceived by others as “poor listener [s],” “self-promoter [s],” “detached,” and “insensitive.” In ideal environments, PGY-2s were also “good listener [s],” “good-natured,” and “team player [s].” However, under stress, PGY-2 external perception was “overly confident,” “poor listener [s],” and “self-promoter [s].”
Conclusions
Clear expectations, friendly work environments, and opportunities to succeed are key to effectively train junior surgical residents. In environments where time is often a limited resource, surgical simulation, stress training, and standardized teaching methods from attending surgeons are needed to develop competent trainees.
Introducción:
En el contexto de la educación médica, la educación quirúrgica es un área con características particulares. Múltiples conceptos, modelos y teorías se han desarrollado para caracterizar la educación quirúrgica; sin embargo, la mayoría se han reportado de forma aislada.
Objetivo:
Identificar los principales conceptos y teorías utilizados en educación quirúrgica para abordarlos de manera conjunta en un panorama integrador.
Método:
Como parte de una revisión panorámica de la literatura sobre educación quirúrgica se identificaron conceptos y sus potenciales relaciones. El análisis continuó de la siguiente forma: modelos no tradicionales para educación quirúrgica, estrategias tecnológicas para la educación quirúrgica como simulación, aprendizaje en línea y uso de redes sociales, así como el área de evaluación en educación quirúrgica, incluyendo habilidades técnicas y no técnicas.
Resultados:
En esta segunda parte del artículo, se describen alternativas al modelo tradicional de aprendiz de Osler y Halsted para la educación quirúrgica. El conocimiento e implementación de estos modelos descansa sobre bases teóricas probadas, en algunos contextos. El uso de tecnología para la educación quirúrgica es más factible cuando esta es consistente con los modelos de aprendizaje, existe integración al diseño curricular, y se aprovechan las varias opciones disponibles. Existen múltiples herramientas que permiten conocer el grado de pericia psicomotriz del alumno como parte de una estrategia de evaluación formativa. Asimismo, la evaluación de habilidades no técnicas es un componente cada vez más importante de la educación quirúrgica.
Conclusiones:
Los gestores de programas de educación quirúrgica deben privilegiar un enfoque con sustento teórico, en las etapas de planeación e implementación. Esto permitirá determinar objetivos y estrategias para su logro, en el contexto y formalidad de un programa estructurado y no solo calendarizado.
General surgery residency programs are facing multiple pressures, including attracting and retaining residents. Despite the importance of resident perspectives in designing effective responses to these pressures, understanding of residents' views is limited.
To profile US general surgery residents; characterize resident attitudes, experiences, and expectations regarding training; and examine differences by sex and training year.
Cross-sectional study of all general surgery residents completing a survey in January 2008 following administration of the American Board of Surgery In-Training Examination.
Resident satisfaction; perceived supports, strains and concern; career motivations; and professional expectations.
Of 5345 categorical general surgery residents, 4402 (82.4%) responded, representing 248 of 249 surgical residency programs. Most respondents expressed satisfaction with training (3686 [85.2%]; 95% confidence interval [CI], 84.1%-86.3%) and supportive peer relationships (3433 [84.2%]; 95% CI, 83.1%-85.3%). However, residents also reported unmet needs and apprehensions about training and careers. Worry that they will not feel confident performing procedures independently was reported by 1185 (27.5%; 95% CI, 26.2%-28.8%), while 2681 (63.8%; 95% CI, 62.4%-65.3%) reported that they must complete specialty training to be competitive. Perceptions of program support differ, with men more likely than women to report that their program provides support (2188 [74.5%] vs 895 [65.6%]; P < .001), and that they can turn to faculty when having difficulties (2193 [74.5%] vs 901 [66.4%]; P < .001). Reports of having considered leaving training in the prior year differed significantly across years (P < .001), highest in postgraduate year 2 (19.2%) and lowest in postgraduate year 5 (7.2%).
General surgery residents' attitudes, experiences, and expectations regarding training reflect both high levels of satisfaction and sources of strain. These factors vary by sex and training year.
The practice of general surgery has undergone a marked evolution in the past 20 years, which has been inadequately recognized and minimally addressed. The changes that have occurred have been disruptive to residency training, and to date there has been inadequate compensation for these. Evidence is now emerging of significant issues in the overall performance of recent graduates from at least 3 sources: the evaluation of external agents who incorporate these graduates into their practice or group, the opinions of the residents themselves, and the performance of graduates on the oral examination of the American Board of Surgery during the past 8 years. The environmental and technological causes of the present situation represent improvements in care for patients, and are clearly irreversible. Hence, solutions to the problems must be sought in other areas. To address the issues effectively, greater recognition and engagement are needed by the surgical community so that effective solutions can be crafted. These will need to include improvements in the efficiency of teaching, with the assumption of greater individual resident responsibility for their knowledge, the establishment of more defined standards for knowledge and skills acquisition by level of residency training, with flexible self-assessment available online, greater focus of the curriculum on current rather than historical practice, increased use of structured assessments (including those in a simulated environment), and modifications to the overall structure of the traditional 5-year residency.
The American Board of Surgery has mandated intraoperative assessment of general surgery residents, yet the time required to train faculty to accurately and reliably complete operating room performance evaluation forms is unknown. Outside of surgical education, frame-of-reference (FOR) training has been shown to be an effective training modality to teach raters the specific performance indicators associated with each point on a rating scale. Little is known, however, about what form and duration of FOR training is needed to accomplish reliable ratings among surgical faculty.
Two groups of surgical faculty separately underwent either an accelerated 1-hour (n = 10) or immersive four-hour (n = 34) FOR faculty development program. Both programs included a formal presentation and a facilitated discussion of sample behaviors for each point on the Zwisch operating room performance rating scale (see DaRosa et al.(8)). The immersive group additionally participated in a small group exercise that included additional practice. After training, both groups were tested using 10 video clips of trainees at various levels. Responses were scored against expert consensus ratings. The 2-sided Mann-Whitney U test was used to compare between group means.
All trainees were faculty members in the Department of Surgery of a large midwestern private medical school.
Faculty undergoing the 1-hour FOR training program did not have a statistically different mean correct response rate on the video test when compared with those undergoing the 4-hour training program (88% vs 80%; p = 0.07).
One-hour FOR training sessions are likely sufficient to train surgical faculty to reliably use a simple evaluation instrument for the assessment of intraoperative performance. Additional research is needed to determine how these results generalize to different assessment instruments.
To assess readiness of general surgery graduate trainees entering accredited surgical subspecialty fellowships in North America.
A multidomain, global assessment survey designed by the Fellowship Council research committee was electronically sent to all subspecialty program directors. Respondents spanned minimally invasive surgery, bariatric, colorectal, hepatobiliary, and thoracic specialties. There were 46 quantitative questions distributed across 5 domains and 1 or more reflective qualitative questions/domains.
There was a 63% response rate (n = 91/145). Of respondent program directors, 21% felt that new fellows arrived unprepared for the operating room, 38% demonstrated lack of patient ownership, 30% could not independently perform a laparoscopic cholecystectomy, and 66% were deemed unable to operate for 30 unsupervised minutes of a major procedure. With regard to laparoscopic skills, 30% could not atraumatically manipulate tissue, 26% could not recognize anatomical planes, and 56% could not suture. Furthermore, 28% of fellows were not familiar with therapeutic options and 24% were unable to recognize early signs of complications. Finally, it was felt that the majority of new fellows were unable to conceive, design, and conduct research/academic projects. Thematic clustering of qualitative data revealed deficits in domains of operative autonomy, progressive responsibility, longitudinal follow-up, and scholarly focus after general surgery education.
The operating room (OR) remains primarily a master/apprenticeship-based learning environment for surgical residents. Changes in surgical education and health care systems challenge faculty to efficiently and effectively graduate residents truly competent in operations classified by the Surgical Council on Resident Education as "common essential" and "uncommon essential." Program directors are charged with employing resident evaluation systems that yield useful data, yet feasible enough to fit into a busy surgical faculty member's workflow. This paper proposes a simple model for teaching and assessing residents in the operating room to guide faculty and resident interaction in the OR, and designating a resident's earned level of autonomy for a given procedure. The system as proposed is supported by theories associated with motor skill acquisition and learning.
Purpose:
Most assessment of surgical trainees is based on measures of knowledge, with limited evaluation of their competence to actually perform various surgical procedures. In this study, the authors evaluated a tool they designed to assess a trainee's competence to perform an entire surgical procedure independently, regardless of procedure type or postgraduate year (PGY).
Method:
In phase 1, the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) was piloted in the University of Ottawa's Division of Orthopaedic Surgery. In phase 2, the refined 11-item tool (8 items rated on a 5-point competency scale, 1 item assessing procedural competence, 2 feedback items) was used in the Divisions of Orthopaedic Surgery and General Surgery to assess residents' performance on 11 common procedures. Quantitative and qualitative analyses were conducted.
Results:
In phase 2, 34 orthopaedic and general surgeons assessed the performance of 37 residents in 163 procedures. ANOVA demonstrated an effect of PGY. Post hoc analysis found that total procedure scores for PGYs 1 and 2 were lower than those for PGY 3 (P<.001), and PGY 3 scores were lower than those for PGYs 4 and 5 (P<.02). Analysis of qualitative data indicated that the rating scale was practical and useful for surgeons and residents.
Conclusions:
This novel evaluation tool successfully discriminated between junior and senior residents and identified surgical competency across various PGY levels regardless of procedure type. Multiple sources of evidence support the O-SCORE as a valid tool for the assessment of trainee operative competency.
Threats to the current form of surgical training in the academic medical center include financial pressures from the government and managed care organizations. A diminishing medical student interest in surgical careers has been noted. The constraints of managed care hold the potential to introduce weaknesses in surgical training in the academic medical center.
Resident evaluation traditionally involves global assessments including clinical performance, professional behavior, technical skill, and number of procedures performed. These evaluations lack objective assessment of operative skills. We describe an operative performance rating system (OPRS) designed to provide objective operative performance ratings using a sentinel procedure format.
Ten-item procedure-specific rating instruments were developed. Items included technical skills, operative decision making, and general items. A 1 to 5 (5 = excellent) scale was used for evaluation. Six procedures had sufficient forms returned to allow evaluation. Inter-rater reliability was determined by having faculty evaluators view 2 videotaped operations.
Return rates for the Internet-based form were full-time faculty (92%), volunteer faculty (27%), and overall (67%). Reliability, (average interitem correlation), and total procedures evaluated were excisional biopsy, 0.90, (0.48), 77; open inguinal herniorraphy, 0.94, (0.62), 51; laparoscopic cholecystectomy, 0.95, (0.64), 75; small-bowel and colon resection, 0.92, (0.58), 30; parathyroidectomy, 0.70, (0.19), 30; and lumpectomy, 0.92, (0.51), 38. Years of training accounted for 25% to 57% of the variation in scores. Inter-rater variability was observed; however, the average rater agreement was reliable.
Internet-based management made obtaining the data feasible. The OPRS complements traditional evaluations by providing objective assessment of operative decision-making and technical skills. Interitem correlations indicate the average rating of items provides a reliable indicator of resident performance. The OPRS is useful in tracking resident development throughout postgraduate training and offers a structured means of certifying operative skills.
Competency-based surgical residency training is rapidly becoming the norm across surgical specialties. Ensuring that graduating surgeons are competent to deliver the necessary services and skills to their patients remains a seminal objective of training programs. Defining surgical competence, the measures used to assess and quantify that competence, and the criteria used to judge whether it has been achieved are critical issues. The bar that surgical residency programs have established is, and must continue to be, set very high. Definitions of competency differ across disciplines. In education, two approaches are recognized. According to the behaviorist approach, competence is assessed by precise measures of performance, generally documented by checklists. The integrated (holistic) approach defines competence as a complex combination of personal attributes. Assessments of competence also fall under two categories: the traditional scientific paradigm, emphasizing objectivity and reproducibility, and the judgment paradigm, reflecting the need to assess clinical competence in the final stages of medical training. In surgery, competence is the ability to successfully apply professional knowledge, skills, and attitudes to new situations as well as to familiar tasks. A critical step in assessing surgical competency is developing methodology for competency evaluation and certification. Matching different aspects of surgical competency with the appropriate assessment instruments is the theme of the contemporary evaluation process, with emphasis on a whole-task approach and the assessment of professional judgment. An effective assessment program will incorporate several competency elements, using multiple sources of information to assess competencies on multiple occasions, at various levels, and in different settings.
To identify the most prevalent patterns of technical errors in surgery, and evaluate commonly recommended interventions in light of these patterns.
The majority of surgical adverse events involve technical errors, but little is known about the nature and causes of these events. We examined characteristics of technical errors and common contributing factors among closed surgical malpractice claims.
Surgeon reviewers analyzed 444 randomly sampled surgical malpractice claims from four liability insurers. Among 258 claims in which injuries due to error were detected, 52% (n = 133) involved technical errors. These technical errors were further analyzed with a structured review instrument designed by qualitative content analysis.
Forty-nine percent of the technical errors caused permanent disability; an additional 16% resulted in death. Two-thirds (65%) of the technical errors were linked to manual error, 9% to errors in judgment, and 26% to both manual and judgment error. A minority of technical errors involved advanced procedures requiring special training ("index operations"; 16%), surgeons inexperienced with the task (14%), or poorly supervised residents (9%). The majority involved experienced surgeons (73%), and occurred in routine, rather than index, operations (84%). Patient-related complexities-including emergencies, difficult or unexpected anatomy, and previous surgery-contributed to 61% of technical errors, and technology or systems failures contributed to 21%.
Most technical errors occur in routine operations with experienced surgeons, under conditions of increased patient complexity or systems failure. Commonly recommended interventions, including restricting high-complexity operations to experienced surgeons, additional training for inexperienced surgeons, and stricter supervision of trainees, are likely to address only a minority of technical errors. Surgical safety research should instead focus on improving decision-making and performance in routine operations for complex patients and circumstances.
Attitudes, training experiences, and professional expectations of us general surgery residents: a national survey
- H Yeo
- K Viola
- D Berg
Yeo H, Viola K, Berg D, et al. Attitudes, training
experiences, and professional expectations of us general
surgery residents: a national survey. J Am Med Assoc.
2009;302(12):1301-1308.
Ensuring an Effective Physician Workforce for the US-Summary of a Conference sponsored by the Josiah Macy Jr. Foundation
- M Johns
Johns M. Ensuring an Effective Physician Workforce
for the US-Summary of a Conference sponsored by
the Josiah Macy Jr. Foundation, held in Atlanta, GA,
2011.