skills in a nonthreatening environment. Multiple measurement
tools, including a task-speciﬁc checklist and a GRS instrument,
were developed to aid in the assessment of participants on the
virtual simulator in the present study. The study also asses sed
and demonstrated the construct validity of a Sawbones simu-
lator. The measurement tools developed for the study were able
to reliably assess surgical ﬁxation of the ulna performed on
both simulators. In addition, the virtual simulator demon-
strated construct validity as the GRS score could be used to
distinguish between experience levels. The procedural mea-
sures used did not correlate between simulators, and concur-
rent validity of the virtual simulator with the Sawbones simulator
was therefore not established. The newly developed virtual sim-
ulator with haptics shows promise for helping surgical trainees
to learn and practice the basic skills involved in ulnar ﬁxation,
but it does not yet attain the same standard as the current sim-
ulators used to achieve optimal surgical performance in residency
An appendix showing the checklist and global rating scale
and the questionnaire distributed after the Sawbones sim-
ulator procedure is available with the online version of this
article as a data supplement at jbjs.org. n
NOTE: The authors thank Drs. Jeremy Tillim, Ganesh Swamy, Beth Pedersen, and Tom Van Raaij,
Synthes (Canada), and all of the residents from the University of Calgary orthopaedic surgery
program who participated in the study. The authors also thank the developers of the virtual ulna
simulator: Manar Din Samad, Aron Su, and Antoine Widmer. The Sawbones simulator and required
equipment for surgical ﬁxation of the ulna were supplied by Synthes (Canada).
Justin LeBlanc, MD, MSc
Carol Hutchison, MD, FRCS(C)
Department of Surgery, Health Science Centre,
Room Orthopaedic Surgery G207,
University of Calgary,
3330 Hospital Drive N.W.,
Calgary, AB T2N 4N1, Canada.
E-mail address for J. LeBlanc: email@example.com
Yaoping Hu, PEng
Department of Electrical and Computer Engineering,
Schulich School of Engineering,
University of Calgary,
2500 University Drive N.W.,
Calgary, AB T2N 1N4, Canada
Tyrone Donnon, PhD
Department of Community Health Sciences,
University of Calgary, 3330 Hospital Drive N.W.,
Calgary, AB T2N 4N1, Canada
1. Berg D, R augi G, Gladstone H, Berkley J, Weghorst S, Ganter M, Turkiyyah G.
Virtual reality simulators for dermatologic surgery: measuring their val idity as a
teaching tool. Dermatol Surg. 2001;27(4):370-4.
2. Cannon WD, Eckhoff DG, Garrett WE Jr, Hunter RE, Sweeney HJ. Report of a group
developing a virtual reality simulator for arthroscopic surgery of the knee joint. Clin
Orthop Relat Res. 2006;442:21-9.
3. Gomoll AH, O’Toole RV, Czarnecki J, Warner JJP. Surgical experience correlates
with performance on a virtual reality simulator for shoulder arthroscopy. Am J Sports
4. Lemole GM Jr, Banerjee PP, Luciano C, Neckrysh S, Charbel FT. Virtual reality in
neurosurgical education: part-task ventriculostomy simulation with dynamic visual
and haptic feedback. Neurosurgery. 2007;61(1):142-8, discussion :148-9.
5. Wang P, Becker AA, Jones IA, Glover AT, Benford SD, Greenhalgh CM, Vloeberghs
M. A virtual reality surgery simulation of cuttin g and retraction in neurosurgery with
force-feedback. Comput Methods Programs Biomed. 2006;84(1):11-8.
6. Schijven MP, Jakimowicz JJ. Validation of virtual reality simulators: Key to the
successful integration of a novel teaching technology into minimal access surgery.
Minim Invasive Ther Allied Technol. 2005;14(4):244-6.
7. Verdaasdonk EGG, Stassen LPS, Monteny LJ, Dankelman J. Validation of a new
basic virtual reality simulator for training of basic endoscopic skills: the SIMENDO.
Surg Endosc. 2006;20(3):511-8.
8. Gallagher AG, Ritter EM, Satava RM. Fundamental principles of validation, and
reliability: rigorous science for the assessment of surgical education and training.
Surg Endosc. 2003;17(10):1525-9.
9. Gay LR, Mills GE, Airasian P. Educational Research: Competencies for analysis
and application. 9 ed. Upper Saddle River, New Jersey: Davis, K. M., 2009:648.
10. Willis AA, Kutsumi K, Zobitz ME, Cooney WP 3rd. Internal ﬁxation of dorsally
displaced fractures of the distal part of the radius. A biomechanical analysis of volar
plate fracture s tability. J Bone Joint Surg Am. 2006;88(11):2411-7.
11. Reznick RK. Teaching and testing technical skills. Am J Surg. 1993;165(3):
12. Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, Brown
M. Objective structured assessment of technical skill (OSATS) for surgical residents.
Br J Surg. 1997;84(2):273-8.
13. Datta V, Chang A, Mackay S, Darzi A. The relationship between motion analysis
and surgical technical assessments. Am J Surg. 2002;184(1):70-3.
14. Goff BA, Lentz GM, Lee D, Houmard B, Mandel LS. Development of an objective
structured assessment of technical skills for obstetric and gynecology residents.
Obstet Gynecol. 2000 Jul;96(1):146-50.
15. Nielsen PE, Foglia LM, Mandel LS, Chow GE. Objective structured assessment of
technical skills for episiotomy repair. Am J Obstet Gynecol. 2003;189(5):1257-60.
16. Regehr G, MacRae H, Reznick RK, Szalay D. Comparin g the psychometric
properties of checklists and global rating scales for assessing performance on an
OSCE-format examination. Acad Med. 1998;73(9):993-7.
17. Reznick R, Regehr G, MacRae H, Martin J, McCulloch W. Testing technical skill
via an innovative ‘‘bench station’’ examination. Am J Surg. 1997;173(3):226-30.
18. Swift SE, Carter JF. Institution and validation of an observed structured as-
sessment of technical skills (OSATS) for obstetrics and gynecology residents and
faculty. Am J Obstet Gynecol. 2006;195(2):617-21, discussion :621-3.
19. VanBlaricom AL, Goff BA, Chinn M, Icasiano MM, Nielsen P, Mandel L. A new
curriculum for hysteroscopy training as demonstrated by an objective structured
assessment of technical skills (OSATS). Am J Obstet Gynecol. 2005;193(5):
20. Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed.
Hillsdale, New Jersey: Lawrence Erlbaum; 1988.
21. Anastakis DJ, Wanzel KR, Brown MH, McIlroy JH, Hamstra SJ, Ali J, Hutchison
CR, Murnaghan J, Reznick RK, Regehr G. Evaluating the effectiveness of a 2-year
curriculum in a surgical skills center. Am J Surg. 2003;185(4):378-85.
22. Ault G, Reznick R, MacRae H, Leadbetter W, DaRosa D, Joehl R, Peters J,
Regehr G. Export ing a technical skills evaluation technology to other sites. Am J
23. Lentz GM, Mandel LS, Goff BA. A six-year study of surgical teaching and skills
evaluation for obstetric/gynecologic residents in porcine and inanimate surgical
models. Am J Obstet Gynecol. 2005;193(6):2056-61.
24. Mackay S, Datta V, Chang A, Shah J, Kneebone R, Darzi A. Multiple Objective
Measures of Skill (MOMS): a new approach to the assessment of technical ability in
surgical trainees. Ann Surg. 2003;238(2):291-300.
25. Szalay D, MacRae H, Regehr G, Reznick R. Using operative outcome to assess
technical skill. Am J Surg. 2000;180(3):234-7.
26. Goff B, Mandel L, Lentz G, Vanblaricom A, Oelschlager AMA, Lee D, Galakatos A,
Davies M, Nielsen P. Assessment of resident surgical skills: is testing feasi b le? Am J
Obstet Gynecol. 2005;192(4):1331-8, discussion :1338-40.
27. Winckel CP, Reznick RK, Cohen R, Taylor B. Reliability and construct validity of a
structured technical skills assessment form. Am J Surg. 1994;167(4):423-7.
28. Olejnik S, Algina J. Measures of effect size for comparative studies: Applica-
tions, interpretations, and limitations. Contemp Educ Psychol. 2000;25(3):241-86.
29. Wilkinson L. Statistical methods in psychology journals: guidelines and expla-
nations. Am Psychol. 1999 Aug;54(8):594-604.
THE J OURNAL OF BONE &JOINT SURGERY
MAY 1, 2013
ACOMPARISON OF RESIDENT PERFORMANCE ON SURGICAL
FIXATION USING VIRTUAL AND SYNTHETIC MODELS