Validation of a Global Assessment of Arthroscopic Skills in a Cadaveric Knee Model

Pan Am Clinic, Winnipeg, Manitoba, Canada
Arthroscopy The Journal of Arthroscopic and Related Surgery (Impact Factor: 3.21). 11/2012; 29(1). DOI: 10.1016/j.arthro.2012.07.010
Source: PubMed


PURPOSE: The purpose of this study was to determine whether a global assessment of arthroscopic skills was valid for blinded assessment of cadaveric diagnostic knee arthroscopy. METHODS: A global skills assessment for arthroscopy was created using a published theory of the development of expertise. Faculty surgeons, fellows, and residents were consented and enrolled in this institutional review board-approved validation study. All participants were oriented to the equipment and procedures for diagnostic arthroscopy of the knee. After reviewing the anatomic structures to be visualized, participants were allowed 10 minutes to complete a diagnostic arthroscopy of the knee. The hands and arthroscopic view were recorded during this attempt. Resident participants completed a second filmed diagnostic arthroscopy 1 week after the initial attempt. Five blinded reviewers watched the synchronized videos and assessed arthroscopic skills with a procedure-specific checklist and the newly developed global skills assessment. The agreement between reviewers was determined by intraclass correlation coefficient. Internal consistency was determined with Cronbach's α. Test-retest reliability was measured by correlating repeated arthroscopies by residents. The ability of the global assessment to discriminate skill levels was determined with between-group Mann-Whitney U tests. RESULTS: The agreement between global assessment scores was strong (I.C.C. = 0.80, 95% C.I. 0.68-0.92). The internal consistency of evaluations was excellent (Cronbach's α = 0.97), and the test-retest reliability was strong (r = 0.52). The global assessment score was shown to be able to discriminate between skill levels by an analysis of variance indicating the difference in means among the various levels of training (P < .0001). CONCLUSIONS: The Objective Assessment of Arthroscopic Skills is a useful adjunct to arthroscopic educators and learners and could be used for in-training evaluations. CLINICAL RELEVANCE: The Objective Assessment of Arthroscopic Skills is an instrument that can be employed to measure the impact of skills curricula, including but not limited to simulation.

1 Follower
7 Reads
  • Source
    • "It was able to distinguish between the novice, experienced and expert arthroscopists in the cadaver knee model. The Objective Assessment of Arthroscopic Skills (OAAS) instrument consists of multiple skill domains each rated on an expertise-based scale with 5 skill-level options [57]. When combined with an anatomical checklist, the OAAS discriminated between skills levels of various levels of training with excellent internal consistency and test-retest reliability. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background The current paradigm of arthroscopic training lacks objective evaluation of technical ability and its adequacy is concerning given the accelerating complexity of the field. To combat insufficiencies, emphasis is shifting towards skill acquisition outside the operating room and sophisticated assessment tools. We reviewed (1) the validity of cadaver and surgical simulation in arthroscopic training, (2) the role of psychomotor analysis and arthroscopic technical ability, (3) what validated assessment tools are available to evaluate technical competency, and (4) the quantification of arthroscopic proficiency. Methods The Medline and Embase databases were searched for published articles in the English literature pertaining to arthroscopic competence, arthroscopic assessment and evaluation and objective measures of arthroscopic technical skill. Abstracts were independently evaluated and exclusion criteria included articles outside the scope of knee and shoulder arthroscopy as well as original articles about specific therapies, outcomes and diagnoses leaving 52 articles citied in this review. Results Simulated arthroscopic environments exhibit high levels of internal validity and consistency for simple arthroscopic tasks, however the ability to transfer complex skills to the operating room has not yet been established. Instrument and force trajectory data can discriminate between technical ability for basic arthroscopic parameters and may serve as useful adjuncts to more comprehensive techniques. There is a need for arthroscopic assessment tools for standardized evaluation and objective feedback of technical skills, yet few comprehensive instruments exist, especially for the shoulder. Opinion on the required arthroscopic experience to obtain proficiency remains guarded and few governing bodies specify absolute quantities. Conclusions Further validation is required to demonstrate the transfer of complex arthroscopic skills from simulated environments to the operating room and provide objective parameters to base evaluation. There is a deficiency of validated assessment tools for technical competencies and little consensus of what constitutes a sufficient case volume within the arthroscopy community.
    BMC Medical Education 05/2013; 13(1):61. DOI:10.1186/1472-6920-13-61 · 1.22 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: Examination of arthroscopic skill requires evaluation tools that are valid and reliable with clear criteria for passing. The Arthroscopic Surgery Skill Evaluation Tool was developed as a video-based assessment of technical skill with criteria for passing established by a panel of experts. The purpose of this study was to test the validity and reliability of the Arthroscopic Surgery Skill Evaluation Tool as a pass-fail examination of arthroscopic skill. Twenty-eight residents and two sports medicine faculty members were recorded performing diagnostic knee arthroscopy on a left and right cadaveric specimen in our arthroscopic skills laboratory. Procedure videos were evaluated with use of the Arthroscopic Surgery Skill Evaluation Tool by two raters blind to subject identity. Subjects were considered to pass the Arthroscopic Surgery Skill Evaluation Tool when they attained scores of ≥3 on all eight assessment domains. The raters agreed on a pass-fail rating for fifty-five of sixty videos rated with an interclass correlation coefficient value of 0.83. Ten of thirty participants were assigned passing scores by both raters for both diagnostic arthroscopies performed in the laboratory. Receiver operating characteristic analysis demonstrated that logging more than eighty arthroscopic cases or performing more than thirty-five arthroscopic knee cases was predictive of attaining a passing Arthroscopic Surgery Skill Evaluation Tool score on both procedures performed in the laboratory. The Arthroscopic Surgery Skill Evaluation Tool is valid and reliable as a pass-fail examination of diagnostic arthroscopy of the knee in the simulation laboratory. This study demonstrates that the Arthroscopic Surgery Skill Evaluation Tool may be a useful tool for pass-fail examination of diagnostic arthroscopy of the knee in the simulation laboratory. Further study is necessary to determine whether the Arthroscopic Surgery Skill Evaluation Tool can be used for the assessment of multiple arthroscopic procedures and whether it can be used to evaluate arthroscopic procedures performed in the operating room.
    The Journal of Bone and Joint Surgery 12/2013; 95(23):e1871-6. DOI:10.2106/JBJS.M.00340 · 5.28 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Introduction Traditional orthopaedic training has followed an apprenticeship model whereby trainees enhance their skills by operating under guidance. However the introduction of limitations on training hours and shorter training programmes mean that alternative training strategies are required. Aims To perform a literature review on simulation training in arthroscopy and devise a framework that structures different simulation techniques that could be used in arthroscopic training. Methods A systematic search of Medline, Embase, Google Scholar and the Cochrane Databases were performed. Search terms included “virtual reality OR simulator OR simulation” and “arthroscopy OR arthroscopic”. Results 14 studies evaluating simulators in knee, shoulder and hip arthroplasty were included. The majority of the studies demonstrated construct and transference validity but only one showed concurrent validity. More studies are required to assess its potential as a training and assessment tool, skills transference between simulators and to determine the extent of skills decay from prolonged delays in training. We also devised a “ladder of arthroscopic simulation” that provides a competency-based framework to implement different simulation strategies. Conclusion The incorporation of simulation into an orthopaedic curriculum will depend on a coordinated approach between many bodies. But the successful integration of simulators in other areas of surgery supports a possible role for simulation in advancing orthopaedic education.
Show more