Article

A Comparison of Orthopaedic Resident Performance on Surgical Fixation of an Ulnar Fracture Using Virtual Reality and Synthetic Models

Department of Surgery, The University of Calgary, Calgary, Alberta, Canada
The Journal of Bone and Joint Surgery (Impact Factor: 5.28). 05/2013; 95(9):e601-6. DOI: 10.2106/JBJS.K.01284
Source: PubMed

ABSTRACT

Surgical trainees develop surgical skills using various techniques, with simulators providing a safe learning environment. Fracture fixation is the most common procedure in orthopaedic surgery, and residents may benefit from simulated fracture fixation. The performance of residents on a virtual simulator that allows them to practice the surgical fixation of fractures by providing a sense of touch (haptics) has not yet been compared with their performance using other methods of practicing fracture fixation, such as a Sawbones simulator model. The purpose of this study was to assess whether residents performed similarly on a newly developed virtual simulator compared with a Sawbones simulator fracture fixation model.
A stratified, randomized controlled study involving twenty-two orthopaedic surgery residents was performed. The residents were randomized to first perform surgical fixation of the ulna on either the virtual or the Sawbones simulator, after which they performed the same procedure on the other simulator. Their performance was evaluated by examiners experienced in fracture fixation who completed a task-specific checklist, global rating scale (GRS) form, and time-to-completion record for each participant on each simulator.
Both simulators distinguished between differing experience levels, demonstrating construct validity; for the Sawbones simulator, the Cohen d value (effect size) was >0.90, and for the virtual simulator, d was >1.10 (p < 0.05 for both). The participants achieved significantly better scores on the virtual simulator compared with the Sawbones simulator (p < 0.05) for all measures except time to completion. The GRS scores showed a high level of internal consistency (Cronbach α, >0.80). However, Pearson product-moment correlation analysis showed no significant correlations between the results on the two simulators; therefore, concurrent validity was not achieved.
The newly developed virtual ulnar surgical fixation simulator, which incorporates haptics, shows promise for helping surgical trainees learn and practice basic skills, but it did not attain the same standards as the current standard Sawbones simulator. The procedural measures used to assess resident performance demonstrated good reliability and validity, and both the Sawbones and the virtual simulator showed evidence of construct validity.

Topics in Training
AComparisonofOrthopaedicResidentPerformance
on Surgical Fixation of an Ulnar Fracture Using
Virtual Reality and Synthetic Models
Justin LeBlanc, MD, MSc, Carol Hutchison, MD, FRCS(C), Yaoping Hu, PEng, and Tyrone Donnon, PhD
Investigation per formed at the Departments of Surgery, Electrical and Computer Engineering,
and Community Health Sciences, University of Calgary, Calgar y, Alberta, Canada
Background: Surgical trainees develop surgical skills using various techniques, with simulators providing a safe learning
environment. Fracture fixation is the most common procedure in orthopaedic surgery, and residents may benefit from
simulated fracture fixation. The performance of residents on a virtual simulator that allows them to practice the surgical
fixation of fractures by providing a sense of touch (haptics) has not yet been compared with their performance using other
methods of practicing fracture fixation, such as a Sawbones simulator model. The purpose of this study was to assess
whether residents performed similarly on a newly developed virtual simulator compared with a Sawbones simulator
fracture fixation model.
Methods: A stratified, randomized controlled study involving twenty-two orthopaedic surgery residents was performed.
The residents were randomized to first perform surgical fixation of the ulna on either the virtual or the Sawbones
simulator, after which they performed the same procedure on the other simulator. Their performance was evaluated by
examiners experienced in fracture fixation who completed a task-specific checklist, global rating scale (GRS) form, and
time-to-completion record for each participant on each simulator.
Results: Both simulators distinguished between differing experience levels, demonstrating construct validity; for the
Sawbones simulator, the Cohen d value (effect size) was >0.90, and for the virtual simulator, d was >1.10 (p < 0.05 for
both). The participants achieved significantly better scores on the virtual simulator compared with the Sawbones
simulator (p < 0.05) for all measures except time to completion. The GRS scores showed a high level of internal
consistency (Cronbach a, >0.80). However, Pearson product-moment correlation analysis showed no significant cor-
relations between the results on the two simulators; therefore, concurrent validity was not achieved.
Conclusions: The newly developed virtual ulnar surgical fixation simulator, which incorporates haptics, shows promise
for helping surgical trainees learn and practice basic skills, but it did not attain the same standards as the current
standard Sawbones simulator. The procedural measures used to assess resident performance demonstrated good
reliability and validity, and both the Sawbones and the virtual simulator showed evidence of construct validity.
Disclosure: One or more of the authors received payments or services, either directly or indirectly (i.e., via his or her institution), from a third party in
support of an aspect of this work. None of the authors, or their institution(s), have had any financial relationship, in the thirty-six months prior to s ubmission
of this work, with any entity in the biomedical arena that could be perceived to influence or have the potential to influence what is written in this work. Also,
no author has had any other relationships, or has engaged in any other activities, that could be perceived to influence or have the potential to influence
what is written in this work. The complete Disclosures of Potential Conflicts of Interest submitted by authors are always provided with the online version of
the article.
e60(1)
COPYRIGHT ! 2013 BY THE JOURNAL OF BONE AND JOINT SURGERY,INCORPORATED
J Bone Joint Surg A m. 2013;95:e60(1-6)
d
http://dx.doi.org/10.2106/JBJS.K.01284
Page 1
Surgical resident education is based on a foundation of clinical
experiences, didactic teaching, and acquisition of medical knowl-
edge. Surgical skills are taught through progressively increasing
responsibility in the operating room under the supervision of a
practicing surgeon. This system is not without risks to patients,
as surgical procedures placed in the hands of the inexperienced
trainee may result in injury to the patient. The more practice
that a resident has before performing surgery on a patient, the
more likely it is that he or she will perform the surgical proce-
dure safely and effectively. There has been increasing interest
in practicing with virtual reality simulators
1-5
. Fracture xat ion
is the most co mmon procedure in orthopaedic surgery, and
residents may benefit from simulated fracture fixation. How-
ever, the performance of re sid ents on a virtual simulator that
allows them to practice the surgical fixation of fractures by
providing a sense o f tou ch (haptics) has not yet been com-
pa red with their performance using o ther method s of prac-
ticing f racture fixation, such as a Sawbones simulator model
(Pacific Research Laboratories, Vashon, Washington).
Before a virtual simulator is used in a surgical training
curriculum, its efficacy should be validated. A valid virtual sim-
ulator should be able to imitate the visual and spatial environment
and the real-time characteristics of the simulated procedure ; if
possible, it should also include a haptic (force-feedback) device
6
.
Validit y can be assessed in several different ways. Simulated skills
training is usually assessed on the basis of construct validity, which
refers to the degree to which an instrument teaches the desired
skill
7
. The construct validity of a simulator in surgical education
is described as the ability to distinguish between differing experi-
ence levels; expert surgeons should perform better on the simu-
lator than novices
6,8
.
However, a valid surgical skills simulator should also pro-
vide content and/or criterion validity
6
. Content validity is non-
statistical and is based on the description of the contents of the
simulator and expert judgment regarding the extent to which the
simulator reproduces the real-life procedure
7
. An example of
content validity involves how well a virtual simulator is designed
to develop the skill of drilling bone. Concurrent validity, a type
of criterion-related validity, can be evaluated by comparing a test
score on a simulator with the score on a different, previously
validated simulator that teaches the same skill
9
. Concurrent va-
lidity can be established when participants use both simulators
and their performance is compared. When a new virtual simu-
lator is developed with the intent of recreating a certain concept,
such as surgical fixation of the ulna, it must be compared with
the currently accepted simulators. Sawbones simulators have
been used to test methods of surgical fixation of the radius
10
.
Most evaluations of technical skills of surgical trainees are
subjective in nature, consisting of summative evaluations based
on recall by the attending physician, and demonstrate low reli-
ability
11
. In contrast, the Objective Structured Assessment of
Technical Skill (OSATS) represents an objective measurement
format that is used to assess surgical competency of surgical
trainees
12
. The OSATS measures have gone through extensive va-
lidity and reliability testing
12-19
.Thisformatforevaluatingtechni-
cal skills can be used to aid in the evaluation of residents in a skills
laboratory setting or for the evaluation of new surgical simulators.
The purpose of the present study was to assess the concur-
rent validity of a newly developed virtual simulator, compared
with that of a Sawbones simulator, with use of a modified OSATS
checklist and global rating scale (GRS). The secondary purposes
were to confirm the construct validity of the Sawbones simulator
and to assess the reliability of the procedural measurement tools.
We hypothesized that the participants would perform as well on
the virtual simulator as on the Sawbones simulator and that the
construct validity of the Sawbones simulator would be estab-
lished. We also hypothesized that the virtual simulator would
demonstrate concurrent validity and a level of construct validity
similar to that of the Sawbones simulator.
Materials and Methods
The study was approved by our institutions Research Ethics Board, and in-
formed consent was obtained from all participants. Participants were assigned
Fig. 1
The virtual simulator, consisting of a display (right) and a haptic device (left).
TABLE I Participant Demographics*
Group 1 (no.) Group 2 (no.)
Total 11 11
PGY-5 2 2
PGY-4 0 3
PGY-3 3 0
PGY-2 4 3
PGY-1 2 3
Female 3 5
Age in yr
25-29 7 6
30-34 2 4
35-40 2 1
*Group 1 used the Sawbones simulator first, and Group 2 used
the virtual simulator first.
e60(2)
THE J OURNAL OF BONE &JOINT SURGERY
d
JBJS.ORG
VOLUME 95-A
d
NUMBER 9
d
MAY 1, 2013
ACOMPARISON OF RESIDENT PERFORMANCE ON SURGICAL
FIXATION USING VIRTUAL AND SYNTHETIC MODELS
Page 2
identification numbers when they consented, and these numbers were used on
their evaluation forms to ensure confidentiality.
Simulators
A virtual simulator, including haptics, for fixation of ulnar fractures was devel-
oped in collaboration with our institutions Department of Computer and Elec-
trical Engineering. This simulator consists of a virtual ulna and virtual versions
of all tools required for surgical fixation of the ulna. Tabs displayed around the
top and sides of the screen provide the optional tools to use for this surgical
fixation (Fig. 1). A haptic device (PHANTOM 1.5/6DOF; SensAble Technologies,
Wilmington, Massachusetts) provides realistic force feedback during the proce-
dure, allowing the user to move the tools around the screen and to feel resistance
when the bone is touched with a drill, tap, or screw. This haptic device consists of
a handle attached to a mobile arm that allows six degrees of freedom and a control
box that allows resistance to be felt. Participants using the Sawbones simulator
were provided with the same tools normally found in the operating room and at
procedural skills training courses. Neither the virtual simulator nor the Sawbones
simulator (model 1017) had soft-tissue elements, and neither device simulated
fracture reduction. The only skill tested was internal fixation.
Sample Size
Twenty-two of the twenty-six residents in the Orthopaedic Surgery program at
the University of Calgary volunteered to participate in this study. Two residents
were eliminated because they had participated in the pilot project, one was not
in the country at the time of the study, and one was an investigator for the study.
Examiners
The examiners successfully completed an advanced AO (Arbeitsgemeinschaft f
¨
ur
Osteosynthesefragen) fracture fixation skills course for standardization purposes
prior to this study, and all five were FRCS(C) fellows (three staff surgeons and two
fellows). Checklists and GRS forms were provided to the examiners five days prior
to the examination day. Examiners were also provided with a brief training session
on the examination day, and they were then randomly assigned to a simulator and
a set of participants. For reliability testing, ten participants on the virtual simu-
lator and six participants on the Sawbones simulator were independently evalu-
ated by two examiners at the same time. The remaining participants were each
reviewed by a single examiner on each simulator.
Procedures
Apilotprojectinvolvingtwomedicalstudents,twopostgraduateyear(PGY)-3or-
thopaedic residents, and one orthopaedic staff member was completed prior to the
study . The checklists, GRS, and simulators were all optimized on the basis of the
feedback from these pilot subjects to ensure that the study would proceed without
complications. The participants in the full study were stra tified ac co rding to exp erienc e
level and were then randomized to begin with either the Sawbones simulator (Group
1) or the virtual simulator (Group 2); eleven participants were assigned to each group
with us e of com pute r -ge ner ate d rand om num ber s. Writte n inst ruc tio ns for the pr o-
cedure were available on the bench for each simulator, and those for the virtual
simulator included specific directions about how to select and use the virtual tools.
Participants on each simulator were allowed ten minutes to learn the basic controls
and tools and were then asked to perform an internal fixation of the ulna with use of a
neutralization plate. The participants were told that they had as much time to complete
the task as needed but that the time to completion was being measured. The partic-
ipants then performed the fixation procedur e on the other simulator.
Measurement Tools Used
An independent examiner assessed the performance of each participant during the
procedure with use of an itemized checklist, a GRS form, and the time required to
complete the procedure. The fifteen-task itemized checklist and the six-domain
GRS (see Appendix) were based on the OSATS format and were modified from
those used at a station for the surgical xation of the radius
11
.Thetimetocom-
pletion was calculated on the basis of the start and finish times of the procedure.
Statistical Methods
A blinded assessor analyzed the data. A p value of <0.05 was considered sig-
nificant, and the effect size was calculated. Effect sizes were evaluated with use
TABLE II Comparison of Virtual and Sawbones Simulator Scores
Virtual* Sawbones* P Value Effect Size
Checklist (%) 76 (11.73) 66 (12.15) <0.05 0.83
Global rating scale (%) 83 (9.94) 69 (16.99) <0.05 1.05
Time to completion (sec) 642 (120.19) 574 (112.02) <0.05 0.59
*Values are given as the mean, with the standard deviation in parentheses.Paired-sample t test.Cohen d value.
TABLE III Assessment of Construct Validity of the Simulators
Junior Residents* Senior Residents* P Value Effect Size
Checklist, virtual (%) 71 (13.92) 81 (5.89) 0.054 0.93
Checklist, Sawbones (%) 62 (12.99) 72 (8.34) <0.05 0.98
GRS, virtual (%) 79 (8.45) 89 (8.82) <0.05 1.20
GRS, Sawbones (%) 58 (10.04) 82 (13.35) <0.05 1.98
Time, virtual (sec) 647 (108.96) 636 (137.34) 0.846 0.09
Time, Sawbones (sec) 637 (96.46) 511 (94.08) <0.05 1.32
*Values are given as the mean, with the standard deviation in parentheses.Paired-sample t test.Cohen d value.
e60(3)
THE J OURNAL OF BONE &JOINT SURGERY
d
JBJS.ORG
VOLUME 95-A
d
NUMBER 9
d
MAY 1, 2013
ACOMPARISON OF RESIDENT PERFORMANCE ON SURGICAL
FIXATION USING VIRTUAL AND SYNTHETIC MODELS
Page 3
of the Cohen d value, with d = 0.20 to 0.49 being a small effect, d = 0.50 to 0.79
being a medium effect, and d 0.80 being a large effect
20
.
Paired-sample t tests were used to determine whether differences be-
tween the virtual simulator and Sawbones simulator scores were significant.
Independent-sample t tests were used to evaluate the construct validity of each
simulator by determining whether it could differentiate between experience
levels. Pearson product-moment correlations between simulators were used to
determine the criterion-related validity; this method produces an r value that
measures the strength of the relationship between the two sets of measure-
ments. The internal consistency (reliability) of the checklist and that of the GRS
were assessed with use of the Cronbach a value, with a > 0.70 being considered
an acceptable level. This value measures whether items that purport to measure
the same process produce similar results
9
. The intraclass correlation coefficient
(ICC) was used to assess the inter-rater agreement for the checklist and for the
GRS; an ICC value of 0 indicates no agreement between the independent
evaluators and 1 indicates complete agreement.
Each participant filled out a simulator-specific questionnaire, consisting
of 5-point Likert-style questions, after the completion of each procedure (see
Appendix). The results were used to determine face and content validity.
Source of Funding
This study was partially funded by a Calgary Orthopaedic Research and Edu-
cation Fund (COREF) grant from the Division of Orthopaedic Surgery that
allowed for the design and development of the virtual simulator.
Results
Demographic information on the twenty-two orthopaedic sur-
gery residents is presented in Table I. Participants in PGY-1 and
2 were classified as junior residents and those in PGY-3, 4, and
5 were classified as senior. The sex and age distributions in the
two groups were similar. Residents performed more efficiently
on the virtual simulator compared with the Sawbones simula-
tor according to most measures, as indicated by paired-sample
t tests; the effect sizes were medium to large (Table II).
Construct validity for each simulator was assessed by
comparing the mean checklist score, GRS score, and time to
completion for junior residents with those for senior residents
with use of independent-sample t tests. Senior residents achieved
significantly better performance scores compared with junior
residents on all measures except the virtual simulator checklist
score and the virtual simulator time to completion (Table III).
The effect size for each of the significant differences was large
(d > 0.80).
A correlation analysis between the two simulators revealed
no significant Pearson coefficients for any of the measures.
Face validity was evaluated with a questionnaire completed
after simulator use. Participants thought that both the Sawbones
and virtual simulators would be useful for the introduction of
basic surgical skills (as indicated by scores of 4.6 and 4.2, respec-
tively; p = 0.52). However, they thought that the virtual simulator
needed further development (as indicated by a score of 4.6 com-
pared with 2.2 for the Sawbones, p < 0.05), and participants
would use the Sawbones simulator preferentially if given a choice
of simulator for practicing internal fixation (p < 0.05).
The internal consistency coefficient for the checklist was
a = 0.43 for the Sawbones simulator and a = 0.59 for the
virtual simulator. The corresponding values for the GRS were
a = 0.96 and 0.84. When the checklist was combined with the
GRS, the internal consistency coefficient was a = 0.86 for the
Sawbones and a = 0.82 for the virtual simulator. Both check-
lists demonstrated acceptable inter-rater reliability, with an
ICC of >0.70, and when the checklist and GRS were combined,
the ICC reached >0.90 for both simulators (Fig. 2).
Discussion
We had hypothesized that the participants would perform as
well on the virtual simulator as on the Sawbones simulator and
that construct validity of the Sawbones simulator would be
established. We had also hypothesized that the virtual simulator
would demonstrate concurrent validity and a level of construct
validity similar to that of the Sawbones simulator. The residents
performed as well on the virtual simulator as on the Sawbones
simulator, as demonstrated by the overall checklist and GRS
scores. Construct validity for the Sawbones simulator w as dem-
onstrated for the checklist and GRS scores and the time to
completion. Construct validity for the virtual simula tor was
demonstrated by the GRS score. Concurrent validity of the
Fig. 2
Inter-rater reliability (intraclass correlation coefficient) of the checklist and the global rating scale.
e60(4)
THE J OURNAL OF BONE &JOINT SURGERY
d
JBJS.ORG
VOLUME 95-A
d
NUMBER 9
d
MAY 1, 2013
ACOMPARISON OF RESIDENT PERFORMANCE ON SURGICAL
FIXATION USING VIRTUAL AND SYNTHETIC MODELS
Page 4
virtual simulator with the Sawbones simulator was not estab-
lished. Because the simulator scores did not correlate well, the
virtual simulator may need further development prior to its
implementation for instructional or practice purposes.
The OSATS format is a reliable and validated method to
evaluate the performance of surgical trainees on spe cific surgi-
cal procedures. An itemized checklist and a GRS form were
used to evaluate internal fixation of the ulna on both the newly
developed virtual simulator and the Sawbones ulna model.
These procedural measures were used to assess participant per-
formance and skill level as well as to assess the validity of the
virtual simulator by comparison with the Sawbones standard.
The internal consistency of the modified checklist and that of
the GRS were assessed for each simulator. The internal consis-
tency of the checklist score was modest for both the Sawbones
simulator (a = 0.43) and the virtual simulator (a = 0.59)
compared with the GRS score (a = 0.96 for the Sawbones
and a = 0.84 for the virtual simulator). This result was consis-
tent with those of previous studies, which found that the in-
ternal consistency of che cklists alone ranged from a = 0.33 to
0.79, whereas that of a GRS alone ranged from a = 0.66 to
0.98
11,15,18,21,22
. Other studies have combined the checklist and
GRS scores
17,23-25
. The reliability of the combined checklist and
GRS score for each simulator in the present study was >0.80.
Inter-rater reliability was also assessed because the present
study used a modified version of the OSATS checklist and GRS
form and used these to evaluate a new simulator. Inter-rater
reliability for the checklist for each simulator was acceptable,
with ICC > 0.70. Studies have consistently shown an inter-rater
reliability of between 0.64 and 0.98 for the checklist and between
0.42 and 0.98 for the GRS
11,13,18,26,27
. The combined checklist and
GRS for each simulator had an ICC of >0.90, which was above
an accepted quality benchmark level of >0.7 for inter-rater reli-
ability
8
, suggesting that the checklist and GRS provide a useful
measurement of surgical performance if they are combined.
Multiple methods are needed to measure validity for a
new simulator, and one of these involves a test of construct
validity. The construct validity of a simulator used in surgical
education is described as the ability of the simulator to distin-
guish between differing experience levels. To our knowledge,
the construct validity of the Sawbones simulator for surgical
fixation had not been previously described, and such validit y
needed to be established so that the virtual simulator could be
compared with it. The Sawbones simulator was able to distin-
guish junior residents from senior residents on the basis of the
three different objective measurements. The newly developed
virtual surgical simulator for surgical fixation of the ulna was
able to distingu ish junior residents from senior residents on the
basis of the GRS. When significance was reached, large effect
sizes were noted for both the Sawbones and virtual simulators,
indicating a meaningful difference in participant performance
between experience levels. Effect size helps to assess the mean-
ingfulness of statistically significant results
28
, and the American
Psychological Association (APA) task force on statistical in-
ference has recommended that it be reported for all primary
outcomes
29
. The time to completion of the virtual simulator
procedure did not achieve construct validity, although this may
not be due entirely to the simulator itself. A superior time to
completion may be related to the previous experience of the
participant in using the in struments of the virtual or Sawbones
simulator. An example of this involves the PGY-5 residents,
who had used Sawbones simulators multiple times prior to this
procedure. The residents in this year had the shortest times to
completion of the Sawbones simulator procedure, as they were
at the peak of their training. However, they were using the
virtual simulator for the first time, and they shared some of
the longest times to completion of the procedure on this sim-
ulator. Furthermore, by itself, time to completion of a proce-
dure may not be the best indicator of the construct validity of a
simulator. A surgeon may perform a procedure quickly, but it
may be executed unsafely and with poor results. Outcomes may
actually improve when more time is initially taken to perform a
procedure. On the other hand, it can be argued that time to
completion is an important outcome as participants who have
the knowledge, skill, and comfort level needed to correctly per-
form a procedure will eventually become more proficient and
ultimately be able to perform the task faster. Therefore, time may
be utilized as an outcome measure, but it should not be used in
isolation for the validation of a simulator.
With the confirmation of construct validity for the Saw-
bones simulator, concurrent validity could then be determined
for the virtual simulator. The Pearson product-moment correla-
tion coefficient can be utilized to determine such criterion valid-
ity. No significant correlations between the outcome measures
for the Sawbones and virtual simulators wer e noted in the between-
simulator analysis. This absence of between-simulator corre-
lations indicated that concurrent validity with the Sawbones
simulator was not achieved. This may indicate that the simula-
tors are not measuring the same constructs accurately, and there-
fore the virtual simulator does not meet the same standards for
internal fixation of the ulna as the Sawbones simulator does.
One limitation of the study is the low inter-rater reliability
of the GRS alone, which may be the result of inadequate exam-
iner preparation
18
. This may be a concern as examiners were
permitted only ten minutes of training prior to commencement
of the evaluations. Another possible limitation is the fact that
examiners were not completely blinded to the experience level of
the participants, as they were all from the same orthopaedic
surgery program. This likely did not affect the results as previous
studies have shown an absence of significant differences between
unblinded and blinded examiners, although there was a trend
for unblinded examiners to give participants slightly higher
scores
26
. Another limitation is that the study design did not com-
pare a simulator with a real operati ve experience; however,
Martin et al. found psychometric equivalency between bench-
model simulations and surgical skills involving live animals
12
.The
use of bench-model simulators also allowed standardization of
the procedure, which cannot be guaranteed in live operations.
In conclusion, simulators allow surgical trainees to safely
practice and learn procedures prior to performing them on
patients. Virtual surgical simulators are being developed to
aid surgical trainees in repeatedly and safely practicing surgical
e60(5)
THE J OURNAL OF BONE &JOINT SURGERY
d
JBJS.ORG
VOLUME 95-A
d
NUMBER 9
d
MAY 1, 2013
ACOMPARISON OF RESIDENT PERFORMANCE ON SURGICAL
FIXATION USING VIRTUAL AND SYNTHETIC MODELS
Page 5
skills in a nonthreatening environment. Multiple measurement
tools, including a task-specific checklist and a GRS instrument,
were developed to aid in the assessment of participants on the
virtual simulator in the present study. The study also asses sed
and demonstrated the construct validity of a Sawbones simu-
lator. The measurement tools developed for the study were able
to reliably assess surgical fixation of the ulna performed on
both simulators. In addition, the virtual simulator demon-
strated construct validity as the GRS score could be used to
distinguish between experience levels. The procedural mea-
sures used did not correlate between simulators, and concur-
rent validity of the virtual simulator with the Sawbones simulator
was therefore not established. The newly developed virtual sim-
ulator with haptics shows promise for helping surgical trainees
to learn and practice the basic skills involved in ulnar fixation,
but it does not yet attain the same standard as the current sim-
ulators used to achieve optimal surgical performance in residency
training programs.
Appendix
An appendix showing the checklist and global rating scale
and the questionnaire distributed after the Sawbones sim-
ulator procedure is available with the online version of this
article as a data supplement at jbjs.org. n
NOTE: The authors thank Drs. Jeremy Tillim, Ganesh Swamy, Beth Pedersen, and Tom Van Raaij,
Synthes (Canada), and all of the residents from the University of Calgary orthopaedic surgery
program who participated in the study. The authors also thank the developers of the virtual ulna
simulator: Manar Din Samad, Aron Su, and Antoine Widmer. The Sawbones simulator and required
equipment for surgical fixation of the ulna were supplied by Synthes (Canada).
Justin LeBlanc, MD, MSc
Carol Hutchison, MD, FRCS(C)
Department of Surgery, Health Science Centre,
Room Orthopaedic Surgery G207,
University of Calgary,
3330 Hospital Drive N.W.,
Calgary, AB T2N 4N1, Canada.
E-mail address for J. LeBlanc: leblanc.justin@me.com
Yaoping Hu, PEng
Department of Electrical and Computer Engineering,
Schulich School of Engineering,
University of Calgary,
2500 University Drive N.W.,
Calgary, AB T2N 1N4, Canada
Tyrone Donnon, PhD
Department of Community Health Sciences,
University of Calgary, 3330 Hospital Drive N.W.,
Calgary, AB T2N 4N1, Canada
References
1. Berg D, R augi G, Gladstone H, Berkley J, Weghorst S, Ganter M, Turkiyyah G.
Virtual reality simulators for dermatologic surgery: measuring their val idity as a
teaching tool. Dermatol Surg. 2001;27(4):370-4.
2. Cannon WD, Eckhoff DG, Garrett WE Jr, Hunter RE, Sweeney HJ. Report of a group
developing a virtual reality simulator for arthroscopic surgery of the knee joint. Clin
Orthop Relat Res. 2006;442:21-9.
3. Gomoll AH, O’Toole RV, Czarnecki J, Warner JJP. Surgical experience correlates
with performance on a virtual reality simulator for shoulder arthroscopy. Am J Sports
Med. 2007;35(6):883-8.
4. Lemole GM Jr, Banerjee PP, Luciano C, Neckrysh S, Charbel FT. Virtual reality in
neurosurgical education: part-task ventriculostomy simulation with dynamic visual
and haptic feedback. Neurosurgery. 2007;61(1):142-8, discussion :148-9.
5. Wang P, Becker AA, Jones IA, Glover AT, Benford SD, Greenhalgh CM, Vloeberghs
M. A virtual reality surgery simulation of cuttin g and retraction in neurosurgery with
force-feedback. Comput Methods Programs Biomed. 2006;84(1):11-8.
6. Schijven MP, Jakimowicz JJ. Validation of virtual reality simulators: Key to the
successful integration of a novel teaching technology into minimal access surgery.
Minim Invasive Ther Allied Technol. 2005;14(4):244-6.
7. Verdaasdonk EGG, Stassen LPS, Monteny LJ, Dankelman J. Validation of a new
basic virtual reality simulator for training of basic endoscopic skills: the SIMENDO.
Surg Endosc. 2006;20(3):511-8.
8. Gallagher AG, Ritter EM, Satava RM. Fundamental principles of validation, and
reliability: rigorous science for the assessment of surgical education and training.
Surg Endosc. 2003;17(10):1525-9.
9. Gay LR, Mills GE, Airasian P. Educational Research: Competencies for analysis
and application. 9 ed. Upper Saddle River, New Jersey: Davis, K. M., 2009:648.
10. Willis AA, Kutsumi K, Zobitz ME, Cooney WP 3rd. Internal fixation of dorsally
displaced fractures of the distal part of the radius. A biomechanical analysis of volar
plate fracture s tability. J Bone Joint Surg Am. 2006;88(11):2411-7.
11. Reznick RK. Teaching and testing technical skills. Am J Surg. 1993;165(3):
358-61.
12. Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, Brown
M. Objective structured assessment of technical skill (OSATS) for surgical residents.
Br J Surg. 1997;84(2):273-8.
13. Datta V, Chang A, Mackay S, Darzi A. The relationship between motion analysis
and surgical technical assessments. Am J Surg. 2002;184(1):70-3.
14. Goff BA, Lentz GM, Lee D, Houmard B, Mandel LS. Development of an objective
structured assessment of technical skills for obstetric and gynecology residents.
Obstet Gynecol. 2000 Jul;96(1):146-50.
15. Nielsen PE, Foglia LM, Mandel LS, Chow GE. Objective structured assessment of
technical skills for episiotomy repair. Am J Obstet Gynecol. 2003;189(5):1257-60.
16. Regehr G, MacRae H, Reznick RK, Szalay D. Comparin g the psychometric
properties of checklists and global rating scales for assessing performance on an
OSCE-format examination. Acad Med. 1998;73(9):993-7.
17. Reznick R, Regehr G, MacRae H, Martin J, McCulloch W. Testing technical skill
via an innovative ‘‘bench station’’ examination. Am J Surg. 1997;173(3):226-30.
18. Swift SE, Carter JF. Institution and validation of an observed structured as-
sessment of technical skills (OSATS) for obstetrics and gynecology residents and
faculty. Am J Obstet Gynecol. 2006;195(2):617-21, discussion :621-3.
19. VanBlaricom AL, Goff BA, Chinn M, Icasiano MM, Nielsen P, Mandel L. A new
curriculum for hysteroscopy training as demonstrated by an objective structured
assessment of technical skills (OSATS). Am J Obstet Gynecol. 2005;193(5):
1856-65.
20. Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed.
Hillsdale, New Jersey: Lawrence Erlbaum; 1988.
21. Anastakis DJ, Wanzel KR, Brown MH, McIlroy JH, Hamstra SJ, Ali J, Hutchison
CR, Murnaghan J, Reznick RK, Regehr G. Evaluating the effectiveness of a 2-year
curriculum in a surgical skills center. Am J Surg. 2003;185(4):378-85.
22. Ault G, Reznick R, MacRae H, Leadbetter W, DaRosa D, Joehl R, Peters J,
Regehr G. Export ing a technical skills evaluation technology to other sites. Am J
Surg. 2001;182(3):254-6.
23. Lentz GM, Mandel LS, Goff BA. A six-year study of surgical teaching and skills
evaluation for obstetric/gynecologic residents in porcine and inanimate surgical
models. Am J Obstet Gynecol. 2005;193(6):2056-61.
24. Mackay S, Datta V, Chang A, Shah J, Kneebone R, Darzi A. Multiple Objective
Measures of Skill (MOMS): a new approach to the assessment of technical ability in
surgical trainees. Ann Surg. 2003;238(2):291-300.
25. Szalay D, MacRae H, Regehr G, Reznick R. Using operative outcome to assess
technical skill. Am J Surg. 2000;180(3):234-7.
26. Goff B, Mandel L, Lentz G, Vanblaricom A, Oelschlager AMA, Lee D, Galakatos A,
Davies M, Nielsen P. Assessment of resident surgical skills: is testing feasi b le? Am J
Obstet Gynecol. 2005;192(4):1331-8, discussion :1338-40.
27. Winckel CP, Reznick RK, Cohen R, Taylor B. Reliability and construct validity of a
structured technical skills assessment form. Am J Surg. 1994;167(4):423-7.
28. Olejnik S, Algina J. Measures of effect size for comparative studies: Applica-
tions, interpretations, and limitations. Contemp Educ Psychol. 2000;25(3):241-86.
29. Wilkinson L. Statistical methods in psychology journals: guidelines and expla-
nations. Am Psychol. 1999 Aug;54(8):594-604.
e60(6)
THE J OURNAL OF BONE &JOINT SURGERY
d
JBJS.ORG
VOLUME 95-A
d
NUMBER 9
d
MAY 1, 2013
ACOMPARISON OF RESIDENT PERFORMANCE ON SURGICAL
FIXATION USING VIRTUAL AND SYNTHETIC MODELS
Page 6
  • Source
    • "The authors reported that it could differentiate between expert and novice surgeons. A virtual reality ulna fracture fixation simulator using a Geomagic Premium 1.5 6DOF haptic device was developed and tested [15]. During tests 22 orthopaedic residents were in a stratified, randomized controlled study which showed that VR simulators distinguished between differing experience levels, demonstrating construct validity. "
    [Show abstract] [Hide abstract] ABSTRACT: A summary of 16 recent construct validity studies is provided by this research which assesses evidence demonstrating whether virtual reality (VR) simulators can be used to assess and score the skill levels of orthopaedic surgeons. The findings of this study are that there is overwhelming evidence that VR simulators can reliably assess the skill levels of orthopaedic trainees, which suggests that VR training could potentially be used as part of a residency testing or assessment tool for orthopaedic surgery. This research provides an analysis of virtual reality orthopaedic simulator construct validity.
    Full-text · Conference Paper · Jul 2015
  • Source
    • "This model is not without risk, though. Inexperienced trainees contribute to a higher rate of re-admissions and reoperations (Palm et al. 2007, Leblanc et al. 2013). A review of 609 studies found that virtual-reality simulation training improved operative skills (Cook et al. 2011), and simulation-based training in orthopedic surgery is starting to emerge. "
    [Show abstract] [Hide abstract] ABSTRACT: Background and purpose Internal fixation of hip fractures is a common and important procedure that orthopedic surgeons must master early in their career. Virtual-reality training could improve initial skills, and a simulation-based test would make it possible to ensure basic competency of junior surgeons before they proceed to supervised practice on patients. The aim of this study was to develop a reliable and valid test with credible pass/fail standards. Methods 20 physicians (10 untrained novices and 10 experienced orthopedic surgeons) each performed 3 internal fixation procedures of an undisplaced femoral neck fracture: 2 hook-pins, 2 screws, and a sliding hip screw. All procedures were preformed on a trauma simulator. Performance scores for each procedure were obtained from the predefined metrics of the simulator. The inter-case reliability of the simulator metrics was explored by calculation of intra-class correlation coefficient. Validity was explored by comparison between novices’ and experts’ scores using independent-samples t-test. A pass/fail standard was set by the contrasting-groups method and the consequences were explored. Results The percentage of maximum combined score (PM score) showed an inter-case reliability of 0.83 (95% CI: 0.65–0.93) between the 3 procedures. The mean PM score was 30% (CI: 7–53) for the novices and 76% (CI: 68–83) for the experienced surgeons. The pass/fail standard was set at 58%, resulting in none of the novices passing the test and a single experienced surgeon failing the test. Interpretation The simulation-based test was reliable and valid in our setting, and the pass/fail standard could discriminate between novices and experienced surgeons. Potentially, training and testing of future junior surgeons on a virtual-reality simulator could ensure basic competency before proceeding to supervised practice on patients.
    Full-text · Article · Apr 2014 · Acta Orthopaedica
  • [Show abstract] [Hide abstract] ABSTRACT: Surgical training has followed the master-apprentice model for centuries but is currently undergoing a paradigm shift. The traditional model is inefficient with no guarantee of case mix, quality, or quantity. There is a growing focus on competency-based medical education in response to restrictions on doctors' working hours and the traditional mantra of "see one, do one, teach one" is being increasingly questioned. The medical profession is subject to more scrutiny than ever before and is facing mounting financial, clinical, and political pressures. Simulation may be a means of addressing these challenges. It provides a way for trainees to practice technical tasks in a protected environment without putting patients at risk and helps to shorten the learning curve. The evidence for simulation-based training in orthopedic surgery using synthetic models, cadavers, and virtual reality simulators is constantly developing, though further work is needed to ensure the transfer of skills to the operating theatre.
    No preview · Article · Apr 2014 · Current Reviews in Musculoskeletal Medicine
Show more