Content uploaded by Susanne P. Lajoie
Author content
All content in this area was uploaded by Susanne P. Lajoie on Oct 21, 2015
Content may be subject to copyright.
229
R. Azevedo and V. Aleven (eds.), International Handbook of Metacognition and Learning Technologies,
Springer International Handbooks of Education 2 , DOI 10.1007/978-1-4419-5546-3_16,
© Springer Science+Business Media New York 2013
Abstract
Medical students’ metacognitive and self-regulatory behaviors are exam-
ined as they diagnose patient cases using BioWorld, a technology rich
learning environment. BioWorld offers an authentic problem-based envi-
ronment where students solve clinical cases and receive expert feedback.
We evaluate the effectiveness of key features in BioWorld (the evidence
table and visualization maps) to see whether they promote metacognitive
monitoring and evaluation. Learning outcomes were assessed through
novice/expert comparisons in relation to diagnostic accuracy, con fi dence,
and case summaries. More speci fi cally we examined how diagnostic pro-
cesses and learning outcomes were re fi ned or improved through practice at
solving a series of patient cases. The results suggest that, with practice,
medical students became more expert-like in the processes involved in
making crucial clinical decisions. The implications of these fi ndings for
the design of features embedded within BioWorld that foster key metacog-
nitive and self-regulatory processes are discussed.
S. P. Lajoie (*) • L . Naismith • E. Poitras • Y-. J. Hong
• I. Cruz-Panesso • J. Ranellucci
Advanced Technologies for Learning in Authentic
Settings (ATLAS), Department of Educational and
Counselling Psychology , McGill University ,
Montreal , Canada
e-mail: susanne.lajoie@mcgill.ca
S. Mamane • J. Wiseman
Advanced Technologies for Learning in Authentic
Settings (ATLAS) , Faculty of Medicine,
McGill University , Montreal , Canada
1 6
Technology-Rich Tools to Support
Self-Regulated Learning
and Performance in Medicine
Susanne P. Lajoie , Laura Naismith , Eric Poitras ,
Yuan-Jin Hong , Ilian Cruz-Panesso , John Ranellucci ,
Samuel Mamane , and Jeffrey Wiseman
This chapter explores how metacognition and
self-regulated learning (SRL) are supported in
the context of BioWorld (Lajoie, 2009 ) , a
technology-rich learning environment for pro-
moting clinical reasoning in medical students.
BioWorld was designed using a cognitive appren-
ticeship framework (Collins, Brown, & Newman,
1987 ) , whereby instruction is based on modeling
expert knowledge, coaching skills in the context
of practice, and fading assistance when no longer
needed. The fi rst section of the chapter presents
the theoretical perspectives, followed by a
description that drives the design of BioWorld
features that support metacognition and SRL. We
then describe these speci fi c features followed by
a series of empirical studies that support the claim
that BioWorld supports medical students to
8
230 S.P. Lajoie et al.
self-regulate their cognition, motivation, and
behavior. More speci fi cally, we look at how meta-
cognitive monitoring and evaluation of knowl-
edge is supported in BioWorld as students use the
evidence table and visualization maps.
Theoretical Framework
The term metacognition originates from Flavell
( 1979 ) who described the concept from a devel-
opmental perspective as thinking about one’s
own thinking (Lajoie, 2008 ) . According to
Flavell, one’s metacognitive skills include estab-
lishing goals to attain understanding, the employ-
ment of strategies to achieve such goals, and the
assessment of one’s progress in accomplishing
them. In a general sense, the basis of metacogni-
tion rests within the individual, as it deals with
the individual’s ability to re fl ect on new or exist-
ing cognitive structures (Dinsmore et al., 2008 ) .
One must also consider the interaction between
the person, behavior, and environment where one
component in fl uences the other. Bandura stresses
this reciprocal determinism in describing the
relationship between behavioral, emotional, and
cognitive regulation (Bandura, 1986 ) . He empha-
sized that an individual’s will to learn, or motiva-
tion to learn, was key to maintaining effortful
learning (Bandura, 1997 ) .
Self-regulated learning (SRL) was seen at the
outset as an integrated theory of learning (Corno
& Mandinach,
1983 ) that examined the interac-
tion of cognitive, motivational, and contextual
factors. Whereas metacognition stresses the
development of the learner’s ability, knowledge,
and accomplishments, self-regulation stresses the
reciprocal determinism of the environment on the
individual, mediated through behavior. Dinsmore
et al. (
2008 ) distinguish between those studying
SRL and metacognition, suggesting that the for-
mer focus on how the environment stimulates the
individual’s awareness and regulatory response,
whereas the latter researchers emphasize that the
mind of the individual is the trigger for subse-
quent judgments.
Models of SRL
Various SRL models exist (e.g., Azevedo, Moos,
Greene, Winters, & Cromley,
2008 ; Boekaerts,
1997 ; Corno & Mandinach, 1983 ; Pintrich, 2000 ;
Winne, 2001 ; Winne & Hadwin, 1998 ;
Zimmerman, 2000 ) that describe the relationship
between various components and elements of
learning. Most models integrate elements of both
metacognition and self-regulation, though each
emphasizes a different aspect of the complex
interrelationship between the individual and con-
textual characteristics of self-regulatory skills
(Pintrich & De Groot, 1990 ; Winne, 2001 ;
Zimmerman, 2000 ) . Corno and Mandinach, for
instance, stress the volitional aspects of SRL,
while Winne and Hadwin focus on its cognitive
dimension, and McCaslin and Hickey ( 2001 )
stress the sociocultural aspects of SRL. Azevedo
and colleagues examine SRL as an event, captur-
ing the deployment of SRL processes at different
levels of granularity (e.g., macro- and microlevel)
and distinguishing between their positive and
negative valence (e.g., appropriate vs. inappropri-
ate) as they occur through time (Azevedo, 2009 ;
Azevedo, Moos, Witherspoon, & Chauncey, 2010 ;
Greene & Azevedo, 2010 ) . Alternatively, Winne
and colleagues (Butler & Winne, 1995 ; Winne,
2001 ; Winne & Hadwin, 1998 ; Winne & Perry,
2000 ) describe how self-regulated learning pivots
on metacognitive monitoring and metacognitive
control and emphasize that SRL is progressive.
Despite these differences, SRL researchers
share Pintrich’s ( 2000 ) four basic assumptions:
that learners actively construct their own mean-
ings, goals, and strategies from the information
available in the external environment along with
information in their own minds (the internal
environment); that learners can monitor, control,
and regulate speci fi c aspects of their own cogni-
tion, motivation, and behavior along with certain
environmental features; that there is a standard
with which comparisons are made to reach, mon-
itor their progress, and then adapt and regulate
their cognition, motivation, and behavior to attain
these goals; and that SRL activities mediate
23116 Technology Rich Tools
between personal and contextual traits and actual
achievement or performance.
Furthermore, Pintrich et al. (Pintrich, 2000 ;
Pintrich & De Groot,
1990 ; Pintrich, Wolters,
& Baxter, 2000 ) describe four phases and areas
of regulation. Signi fi cantly, the planning, mon-
itoring, control, and re fl ection phases can be
applied to four areas of regulation: cognition,
motivation, behavior, and context. It follows
that context, which encompasses physical envi-
ronment, social interactions, and task charac-
teristics, can either facilitate or hinder students’
ability to self-regulate.
SRL skills such as self-monitoring follow a
developmental trajectory, from novice to expert
(Chi, Glaser, & Farr, 1988 ; Lajoie, 2003 ) .
Fundamentally, experts are able to identify rele-
vant information, monitor and select appropriate
problem-solving strategies, recognize what they
understand, and identify when they have made
mistakes. In contrast, there is an absence of self-
monitoring skills among novices (Zimmerman &
Schunk, 2001 ) . We investigate the SRL trajectory
in medicine by exploring how BioWorld supports
the development of professional pro fi ciency in
diagnostic reasoning (Lajoie, 2009 ) . Although
BioWorld was not designed exclusively for the
purpose of fostering metacognition and SRL,
decisions were taken to support metacognition
and SRL in the context of developing expertise in
diagnostic reasoning. We describe these deci-
sions below and de fi ne metacognition and SRL
pertaining to diagnostic reasoning in the context
of the BioWorld experience.
Metacognition and Self-Regulated
Learning in Medical Problem Solving
with BioWorld
The Context
Problem-based learning approaches that are
incorporated early on in the medical curriculum
provide students with opportunities to apply their
basic science knowledge to clinical practice prob-
lems. Experiential learning through clinical clerk-
ships is accepted as an effective way of gaining
clinical reasoning skills and integrating newly
acquired competencies into managing cases
(Maudsley & Strivens,
2000 ) . However, it tends
to be inef fi cient when students only see a few
clinical problems. Furthermore the effectiveness
of this approach depends on the availability of
experienced medical supervisors who are avail-
able to provide effective teaching and feedback
during clinical practice.
Computer-based learning environments afford
the opportunity for medical students to gain addi-
tional experiential-style learning opportunities in
a condensed time frame in a supported environ-
ment. BioWorld was designed with speci fi c cog-
nitive tools to support learning through practicing
and re fi ning skills in relation to medical diagno-
sis (Lajoie,
2009 ) . Here we explore the effective-
ness of speci fi c tools designed to support
metacognition and self-regulated learning within
BioWorld, in particular the intersection between
how the environment can stimulate individual
awareness and how the mind serves as an initiator
for judgments and evaluations.
Figure 16.1 provides an overview of the
BioWorld interface. Each problem starts with a
patient case history where students formulate
their differential diagnoses. Once students select
their primary diagnosis, they report their
con fi dence in this hypothesis by using the belief
meter (%certainty). Students gather evidence
from the case history in support of a particular
hypothesis using the evidence table that remains
visible throughout the problem-solving activity.
There is an online library where students access
declarative knowledge about the disease they are
researching. Information in the library represents
the symptoms, diagnostic tests, and transmission
routes of a speci fi c disease, as well as a glossary
of medical terminology. In order to solve prob-
lems, students must conduct diagnostic tests to
con fi rm or discon fi rm their diagnoses. They do
so by ordering tests on the patient chart , where
the outcomes of their tests are recorded. This
chart is a procedural knowledge tool since it pro-
vides a way for actions to be conducted in the
context of problem solving. A simulated consul-
tation tool is present and learners can obtain
feedback during the data collection process as
232 S.P. Lajoie et al.
well as from the expert summary provided after
they post their fi nal diagnosis.
BioWorld includes several features that serve
as metacognitive tools (Lajoie & Azevedo, 2006 )
that promote metacognitive monitoring and con-
trol strategies critical to medical diagnostic
reasoning. Below, we describe the role that the
evidence table, the expert summaries, and the
expert solution visualization maps play in foster-
ing metacognition both during problem solving
and after reaching a fi nal diagnosis for a particu-
lar case. These features are described below.
Evidence Table
As students solve cases, they select and post the
evidence they see as relevant to solving the case
using the evidence table (see Fig. 16.1 ). Once the
evidence is posted to the table, it remains visible
throughout the problem-solving activity. In doing
so, the evidence table serves as an external
reminder to students of the data they considered
relevant to the case. Students are encouraged to
engage in metacognitive control processes through
assessing the relevance and implications of the
evidence they gather – from the case description
and lab tests – in relation to their diagnosis. In
doing so, students decide whether the test they
ordered helped verify or eliminate a diagnosis and
whether they need to order a new test, reconsult
the case description, or revise/submit their
hypothesis.
Visualization Map
Students’ retrospective re fl ection about their
diagnostic reasoning processes after each case is
solved is supported in two ways: fi rst as a simple
comparison with the evidence that an expert used
to solve the problem and, second, through a visu-
alization map that documents expert diagnostic
reasoning processes. The visualization maps
Fig. 16.1 Overview of BioWorld
23316 Technology Rich Tools
represent expert physicians’ solution processes
and explanations (Gauthier,
2009 ; Gauthier,
Naismith, Lajoie, & Wiseman,
2008 ) . These
maps are constructed by capturing expert physi-
cians’ problem-solving processes through the use
of a concurrent think aloud protocol (Ericsson &
Simon,
1993 ) augmented with the screen capture
and log fi le data recorded while using BioWorld.
The expert physicians’ solution paths for each
case were merged together to provide evidence of
commonalities and differences in their diagnostic
reasoning towards a particular case. The visual
representations help pinpoint such differences
with regard to the sequence of events that charac-
terize the medical diagnostic process.
These expert models can be used as scaffolds
where learners can compare their own processing
actions with pro fi cient problem solvers. It allows
medical students to re fl ect on their own reasoning
by comparing when and how their solution paths
differed from expert physicians’ as well as con-
sider different reasoning paths that lead to the
same diagnosis. Schoenfeld ( 1983 ) referred to
this type of activity as an abstracted replay where
students can replay or rethink their own actions
by focusing students’ attention on the critical
decisions or actions taken by experts. In the case
of BioWorld, the student had to chose, or control,
these metacognitive skills to compare their diag-
nostic reasoning strategies with that of an expert’s.
Simply demonstrating a model does not mean
that learners are actively engaged with it.
Overview of Empirical Evidence
of BioWorld’s Role in Fostering
Metacognition
In this section, we describe three empirical
studies that examine the role of BioWorld in
fostering metacognitive processes that are crucial
in performing medical diagnoses. First, we
investigate the impact of the expert solution
visualization maps on students’ medical diag-
nostic process (Gauthier et al., 2008 ) . Second,
we investigate the effects of the evidence table on
students’ ability to monitor and assess the
medical diagnostic process (McCurdy et al.,
2010 ) . Third, we replicate the fi ndings obtained
from the second study and expand this design to
determine whether the evidence table assists
students in writing case summaries (Lajoie
et al.,
in prep. ) . In the following sections, we
provide an overview of each study in terms of its
research question, methods, experimental
design, results, and conclusions. We then discuss
the implications of our fi ndings for learning
about medical diagnosis in BioWorld.
Study 1: Do Visual Representations
of Experts’ Solutions Scaffold
Self-Regulation?
In this study, we investigate the effects of provid-
ing expert solution visualization maps to medical
students after they submit their fi nal diagnoses in
BioWorld (Gauthier et al., 2008 ) . We expected
that the expert solution visualization maps would
serve as a metacognitive tool (Lajoie & Azevedo,
2006 ) in terms of assisting medical students to
re fl ect on the diagnostic process, thereby improv-
ing their accuracy and con fi dence in performing
medical diagnoses.
Methods
Eighteen second-year medical students partici-
pated in this study. Students were randomly
assigned to either the treatment or control
condition. The eight participants assigned to the
treatment condition were shown the visualization
map after they solved each case using BioWorld,
while the ten participants assigned to the control
condition used BioWorld without the visualiza-
tion maps. The study took place over a 2-day
period in a computer laboratory. On day 1, stu-
dents were given a guided tour of how to use
BioWorld and then solved the fi rst case. On day
2, students solved the remaining 2 cases.
Experimental Design
Students were examined as they learned to solve
patient cases using BioWorld. The study follows a
mixed factorial design with group as a between-
subjects factor (treatment and control groups), case
as a within-subjects factor ( pheochromocytoma,
234 S.P. Lajoie et al.
type 1 diabetes, and hyperthyroidism), and the accu-
racy of the fi nal diagnosis and self-reported
con fi dence levels as the dependent variables.
Results
There was no signi fi cant difference in average
diagnostic accuracy of medical students across
conditions, t (16) = −1.43, p > 0.05. The medical
students who had the bene fi t of the expert solu-
tion visualization maps were only slightly more
accurate, on average, than those who did not have
the maps ( M = 0.79, SD = 0.25, vs. M = 0.60,
SD = 0.31). Likewise there was no signi fi cant dif-
ference in average diagnostic con fi dence for the
two groups ( t (16) = 0.30, p > 0.05). The medical
students who saw the expert solution visualiza-
tion maps were on average as con fi dent as those
who did not ( M = 78.58, SD = 13.35, vs. M = 80.40,
SD = 12.33).
Accuracy and con fi dence levels seemed to
vary based on the type of case. For example,
diagnosing pheochromocytoma was more
dif fi cult than the other cases as indicated by
incorrect diagnosis (see Table
16.1 ). Students in
the control condition who made an incorrect
diagnosis of the pheochromocytoma case were
more con fi dent at the time they submitted their
fi nal diagnosis ( M = 73.63) than those who had
the correct diagnosis ( M = 57.50). A similar pat-
tern emerged for students assigned to the treat-
ment condition, although the difference between
the means was less pronounced ( M = 67.50 vs.
M = 62.00). In contrast to these fi ndings, diagnos-
ing cases of type 1 diabetes and hyperthyroidism
were less dif fi cult. Furthermore, those who
obtained a correct diagnosis were more con fi dent
in their fi nal diagnosis (range = 85.00–91.44).
Conclusion
Our hypothesis was that the expert solution visu-
alization maps would foster self-re fl ection in rela-
tion to the diagnostic process and that, consequently,
medical students would be more accurate and
con fi dent in reaching their fi nal diagnoses. Our
fi ndings suggest that both groups who used
BioWorld improved with respect to their accu-
racy and self-con fi dence in solving cases and that
the effects of the visualization maps were small.
More research on the effectiveness of these maps
as scaffolds are needed. Students may need more
guidance interpreting the expert visualization
maps. For example, they may need a debrie fi ng
with a human tutor to point out how and why an
expert selected strategies different from their
own. It is also possible that self-regulation during
problem solving is more effective than re fl ection
tasks after problem solving. In the next section,
we use a mixed methods approach to delve more
deeply into this data to establish how the evidence
table assists medical students to monitor and
assess the evidence they collect.
Study 2: The Effect of the Evidence
Table on the Medical Diagnosis Process
In this study, we compare how medical students
and expert physicians use the evidence table to
Table 16.1 Proportion of frequencies for accuracy and average con fi dence ratings for fi nal diagnoses
Cases
Accuracy of fi nal diagnoses
Control Treatment
Correct Incorrect Correct Incorrect
Case 1 (pheochromocytoma) 0.20 0.80 0.50 0.50
Case 2 (type 1 diabetes) 0.90 0.10 1.00 0.00
Case 3 (hyperthyroidism) 0.70 0.30 0.87 0.13
Cases
Con fi dence in fi nal diagnoses
Control Treatment
Correct Incorrect Correct Incorrect
Case 1 (pheochromocytoma) 57.50 73.63 62.00 67.50
Case 2 (type 1 diabetes) 91.44 75.00 86.75 –
Case 3 (hyperthyroidism) 85.00 71.67 90.57 40.00
23516 Technology Rich Tools
regulate the medical diagnostic process during
learning with BioWorld (McCurdy et al.,
2010 ) .
The research questions explored are the follow-
ing: do expert physicians differ from medical stu-
dents in the use of the evidence table while
making medical diagnoses in BioWorld, and do
they bene fi t differently from the evidence table?
We expected that all participants would use the
evidence table as a metacognitive tool (Lajoie &
Azevedo, 2006 ) to monitor and assess their own
medical diagnostic processing, through the post-
ing and review of their own evidence selection
pertaining to their fi nal diagnosis. We anticipated
that experts would bene fi t more than novices
since they have more established metacognitive
resources.
Methods
The data used to answer this set of questions
were gathered from past studies with medical
students (see study 1, Gauthier et al., 2008 ) and
expert physicians (see Gauthier, 2009 ) diagnos-
ing the same three cases using BioWorld. Data
from a total of 18 second-year medical students
and 5 physicians from the same university sys-
tem were examined. We focused our investiga-
tion on how medical students and expert
physicians selected speci fi c evidence items that
supported their fi nal diagnosis for each case. We
examined the proportion of the total number of
evidence items entered into the evidence table
that was prioritized as being relevant to their
medical diagnoses (i.e., # evidence items priori-
tized/# evidence items selected). We also exam-
ined the proportion of evidence items that
medical students prioritized that matched with
the expert physician’s prioritized list (i.e., #
expert-like evidence items prioritized/# total evi-
dence items selected by students). For example,
a student that had six expert-like evidence items
prioritized out of a total number of 8 item selec-
tions would have more relevant expert-like
moves than a student that had six expert-like evi-
dence items prioritized over 16 item selections.
This metric identi fi es those students who are
more or less focused on the key elements needed
to make an accurate diagnosis.
Experimental Design
The study follows a mixed factorial design with
group as a between-subjects factor (student and
physician groups), case as a within-subjects factor
(pheochromocytoma, type 1 diabetes, and hyper-
thyroidism), and three dependent variables (the
proportion of the total amount of evidence items
that were prioritized as relevant to making the fi nal
diagnosis, the proportion of evidence that was
taken from the problem statement, and the propor-
tion of evidence taken from the diagnostic tests).
Results
The results of the repeated measures ANOVA
show that the proportion of the total amount of
evidence items that were prioritized differs across
cases and groups, F (2, 32) = 7.54, p < 0.05, and
F (1, 16) = 7.96, p < 0.05, respectively. More evi-
dence was prioritized for the hyperthyroidism
case, followed by type 1 diabetes and the pheo-
chromocytoma. In other words, both novices and
experts were more selective in relation to the evi-
dence that they prioritized to reach a diagnosis of
pheochromocytoma ( M = 0.65, SD = 0.32) as
opposed to type 1 diabetes and hyperthyroidism
( M = 0.79, SD = 0.23, and M = 0.84, SD = 0.23).
However, experts were more selective in priori-
tizing their evidence across all of the cases, since
they selected less evidence to support their fi nal
arguments ( M = 0.52, SD = 0.20), respectively,
than medical students ( M = 0.83, SD = 0.25). The
interaction between group and case was not sta-
tistically signi fi cant ( F (2, 32) = 1.38, p > 0.05).
In regard to differences in prioritization of
problem statement items, there was a between-
group difference ( F (1, 16) = 5.91, p < 0.05) but no
case effect ( F (2, 32) = 1.33, p > 0.05) and no inter-
action between cases and groups ( F (2, 32) = 1.25,
p > 0.05). Experts were more selective ( M = 0.59,
SD = 0.17) than novices ( M = 0.83, SD = 0.28) in
their prioritization of evidence found in the
problem statement. The proportion of diagnos-
tic tests prioritized did not differ across cases
( F (2, 32) = 2.03, p > 0.05) nor across groups
( F (1,16) = 0.99, p > 0.05). However, there was
an interaction between group and cases
( F (2, 32) = 6.82, p < 0.05). The novices were less
236 S.P. Lajoie et al.
selective in regard to the evidence selected for
the type 1 diabetes case ( M = 0.91, SD = 0.21) as
opposed to the pheochromocytoma and hyper-
thyroidism cases ( M = 0.65, SD = 0.39, and
M = 0.84, SD = 0.36, respectively). In contrast,
the experts were more selective in regard to the
evidence selected for the type 1 diabetes case
( M = 0.37, SD = 0.28) as opposed to the pheochro-
mocytoma and hyperthyroidism cases ( M = 0.79,
SD = 0.25, and M = 0.82, SD = 0.27, respectively).
Given the group and case differences, a more
detailed analysis was conducted to compare
medical students with experts. Table 16.2 pres-
ents the repeated measures ANOVA on the pro-
portion of expert-like matches for total evidence
selected as well as speci fi c evidence matches
with expert-like problem statement items and
diagnostic tests. Case effects were found for all
three variables. Fisher’s least signi fi cant differ-
ence multiple comparisons test was conducted
to determine which cases were solved in a man-
ner that matched the expert physician’s solution
(see Table
16.3 ). Medical students differed
most from the experts when diagnosing pheo-
chromocytoma. In diagnosing pheochromocy-
toma, only a small proportion of medical
students’ evidence items matched with the ones
the expert prioritized, as opposed to diagnosing
type 1 diabetes and hyperthyroidism ( M = 0.19
vs. M = 0.59 and M = 0.60). We further analyzed
the type of evidence prioritized, in terms of
diagnostic tests and items found in the problem
statement. Group differences across cases were
found in regard to the proportion of expert-like
diagnostic tests that were prioritized by medi-
cal students, F (2, 26) = 10.67, p < 0.001. Once
again case differences were examined using
LSD comparisons test, which showed that there
was a smaller proportion of matches between
medical students and experts on the diagnostic
tests ordered and prioritized while diagnosing
pheochromocytoma ( M = 0.16). The proportion
of problem statement items that were priori-
tized and that matched the experts solution was
also found to differ across cases, F (2,
26) = 14.24, p < 0.001. The Fisher’s LSD results
show that all of the pairwise comparisons were
signi fi cant. Students differed most from the
experts when diagnosing pheochromocytoma
( M = 0.36); however closer matches were found
for hyperthyroidism ( M = 0.63) and type 1 dia-
betes ( M = 0.80).
Table 16.2 ANOVAs performed on the proportion of expert-like total amount of evidence
items, problem statement items, and diagnostic test items
Variables Source df MS F p
Expert-like proportion of total evidence items Cases 2 0.77 21.34 0.001*
Error 26 0.04
Expert-like proportion of problem statement items Cases 2 0.69 14.24 0.001*
Error 26 0.05
Expert-like proportion of diagnostic tests Cases 2 0.70 10.67 0.001*
Error 26 0.07
* p < 0. 001
Table 16.3 Fisher’s least signi fi cant difference multiple comparison tests on the proportion of expert-like total amount
of evidence items, problem statement items, and diagnostic test items
Variables Case 1 pheochromocytoma Case 2 type 1 diabetes Case 3 hyperthyroidism
Expert-like proportion of total
evidence items
0.19 0.59 0.60
Expert-like proportion of
problem statement items
0.36 0.80 0.63
Expert-like proportion of
diagnostic tests
0.16 0.47 0.60
Note: all pairwise comparisons signi fi cant at p < 0.05
23716 Technology Rich Tools
Conclusion
We compared the evidence that was prioritized
by the medical students and expert physicians
based on the assumption that overlapping pat-
terns of behaviors are indicative of better diag-
nostic reasoning processes by the students. The
results suggest that expert physicians engage in
more metacognitive processes than novice
medical students while using the evidence table
to reach a fi nal diagnosis. In contrast to novices,
experts prioritize less evidence from the prob-
lem statement and clinical tests. Expert physi-
cians are also relatively selective and consistent
in regard to the evidence items that they priori-
tize. In contrast, novices were less selective in
regard to the evidence that they prioritize.
However, novices become more expert-like in
the manner in which they solved the cases with
experience using BioWorld. This fi nding
supports the belief that there is a developmental
trajectory for self-regulated monitoring and
control processes. Case analyses revealed that
the greatest expert-novice differences were for
the pheochromocytoma case, with less overlap
between experts and novices on this case in
terms of how evidence was selected and priori-
tized. Given this was the most dif fi cult case,
some discrepancy is to be expected.
These fi ndings show promise, but further
research was needed to verify whether the pos-
itive impact of solving cases with BioWorld
can be attributed to case complexity or to prac-
ticing and re fi ning skills in relation to diagnos-
tic reasoning with BioWorld. The evaluation of
the effectiveness of the evidence table as a
metacognitive tool must take into account the
skill level of the participant, the order of the cases,
as well as the level of complexity of the cases.
These factors impact students’ ability to monitor
and assess their efforts to prioritize the evidence
and solve the cases. Novices may need more
scaffolding in terms of using the evidence table
more ef fi ciently. In the next section, we present
a follow-up study which replicated and elabo-
rated these fi ndings with a different sample.
The order and complexity of the cases were
further examined.
Study 3: The Effects of the Evidence
Table on Prioritizing Evidence
and Writing Case Summaries
We build on the McCurdy et al. ( 2010 ) study with
regard to the in fl uence of the evidence table on the
diagnostic reasoning processes in BioWorld
(Lajoie et al., in prep. ) . In this study, we used a pre-
and posttest evaluation of learning outcomes as
well as a medical student/physician comparison in
terms of the evidence that was prioritized and
summarized. The primary research questions
addressed in this study are the following: does
having the bene fi t of the evidence table assist
medical students in performing medical diagno-
ses in BioWorld, and does the evidence table assist
participants while writing case summaries? We
compare how medical students and physicians use
the evidence table to regulate the medical diag-
nostic process and write case summaries.
Given that the evidence table provides a means
to monitor and assess one’s thinking with respect
to the diagnostic process, we anticipated that it
would serve as a metacognitive tool that would
also help learners when they were writing their
case summaries where they document how and
why they reached a particular diagnosis. We
anticipated that the evidence that was prioritized
and summarized by the medical students would
become more expert-like (overlapping more with
physician responses) as they used BioWorld. We
also expected that accuracy and con fi dence in
their fi nal diagnoses should increase after prac-
ticing and re fi ning their skills using BioWorld.
Methods
Twelve second-year medical students participated
in the study. The study took place over a 2-day
period in a computer laboratory. The methodol-
ogy used in this study was similar to the one used
in study 2 above (McCurdy et al.,
2010 ) , with sev-
eral exceptions. First, we added a pre- and posttest
evaluation of learning outcomes, each one con-
sisting of a case that had to be solved by the stu-
dents in BioWorld. Second, we compared the
medical students’ prioritized evidence and case
summaries to the expert physicians’ solutions by
238 S.P. Lajoie et al.
counting the number of matching idea units (see
Lajoie et al.,
in prep. ) . Third, we included two
questionnaires that were administered during the
posttest. The fi rst questionnaire, based on the
On-line Motivation Questionnaire (OLM)
(Boekaerts, 2002 ) , measures students’ perception
of the usefulness of BioWorld as well as their
motivation to solve cases. The second measure
assessed students’ perceived dif fi culty of each
case. On day 1 students were given a guided tour
of the software and administered questionnaires.
They then solved the type 1 diabetes case. The
more dif fi cult cases, hyperthyroidism and pheo-
chromocytoma, were presented in random order
on day 2 to rule out the effects of case complexity
(one case being more dif fi cult than another) in
evaluating practice effects with BioWorld.
Experimental Design
The study follows a mixed factorial design with
cases as a within-subjects factor (pretest case,
pheochromocytoma, type 1 diabetes, hyperthy-
roidism, and posttest case), and the dependent
variables were the proportion of expert-like evi-
dence items that were prioritized and summa-
rized, the accuracy of the fi nal diagnosis,
self-reported con fi dence levels, as well as their
reactions and motivation towards using
BioWorld. A repeated measures analysis was
performed in which the data from all participants
was compared to those of the expert physi-
cians, in accordance to the methodology fol-
lowed by McCurdy et al. (
2010 ) .
Results
The results obtained from the repeated measures
analysis of variance show a statistically signi fi cant
difference between cases in terms of the propor-
tion of expert-like evidence items that were pri-
oritized, F (2, 20) = 7.40, p < 0.01. We examined
this case effect further by performing post hoc
comparisons using the Fisher’s LSD that indi-
cated a higher overlap between medical students
and experts on prioritizing evidence items on
cases provided on day 2. The proportion of
expert-like prioritized evidence was signi fi cantly
lower on day 1 (type 1 diabetes case) than for the
hyperthyroidism and pheochromocytoma cases
solved on day 2 ( M = 0.62 vs. M = 0.82, and
M = 0.72). However, the results of the RM-ANOVA
showed no signi fi cant differences between the
cases in terms of the proportion of evidence items
that were expert-like that appear in the case sum-
maries, F (2, 12) = 0.76, p = 0.49. We calculated
the proportion of expert-like idea units in the stu-
dents’ case summaries by dividing the number of
idea units that matched an idea unit mentioned in
an expert case summary by the total number of
idea units mentioned. We compared across cases,
excluding the fi ve students who did not write a
summary for each case. The observed power for
this analysis, calculated at a = 0.05, was low
(0.15) due to the missing data.
To assess students’ overall performance in
BioWorld, we examined the accuracy of their
fi nal diagnoses, anticipating an increase in
accuracy from pre- to posttest. Accuracy was
measured with a value of 1 indicating correct
and 0 indicating incorrect. Given the small
sample size ( N = 12), we used the Wilcoxon’s
Matched-Pairs Signed-Ranks Test to assess the
signi fi cance of these differences. Though stu-
dents increased in accuracy from pretest to
posttest, this difference was not signi fi cant,
Z = −1.00, p = 0.32. We also looked at whether
students’ con fi dence in their fi nal diagnoses
increased from pre- to posttest, again using a
Wilcoxon’s Matched-Pairs Signed-Ranks Test.
Con fi dence was interpreted as the belief meter
value at the time a student submitted his or her
fi nal diagnosis. We expected a statistically
signi fi cant increase in con fi dence levels from
pre- to posttest, and this hypothesis was sup-
ported, Z = −2.98, p < 0.01. Table 16.4 displays
Table 16.4 Means and standard deviations of accuracy
and con fi dence in regard to the fi nal diagnosis at pre- and
posttest
Means (standard deviations)
Variables Pretest Posttest
Accuracy 0.75 (0.45) 0.92 (0.29)
Con fi dence 0.79 (0.11) 0.95 (0.06)
On the post-questionnaire, students were asked to rate the
helpfulness of BioWorld on a fi ve-point Likert scale
(1 = not helpful; 5 = very helpful). The mean response was
3.83 ( SD = 0.72), suggesting that students perceived
BioWorld to be a useful learning tool. From the OLM,
students also reported that they put in a lot of effort in
solving each patient case ( M = 2.97, SD = 0.51)
23916 Technology Rich Tools
the means and standard deviations for the two
overall performance variables.
Conclusion
We evaluated the effectiveness of the evidence
table in terms of assisting medical students to
monitor and assess the medical diagnosis pro-
cess and write case summaries. We compared
the evidence that was prioritized and summa-
rized by the medical students and expert physi-
cians based on the assumption that overlapping
patterns of behaviors are indicative of better per-
formance on the part of the students. Overall,
medical students obtained an average of 60–80%
overlap in the amount of evidence that was pri-
oritized and summarized by the expert physi-
cians. The results demonstrate that practicing
medical diagnosis with BioWorld had a positive
effect on students in that they increased from day
1 to day 2 in the proportion of expert-like evi-
dence that was prioritized. Given that students
were given the most complex and challenging
cases on day 2 makes this practice effect more
important. However, there was no increase in the
amount of expert evidence reported in case sum-
maries over time. This fi nding may be due to the
fact that students were not experienced with
writing case summaries and that fewer students
completed the case summaries.
We expected that medical students would
become more con fi dent in their medical diagno-
ses after practicing with BioWorld. Accordingly,
the con fi dence in performing medical diagnoses
increased from pre- to posttest cases. On the one
hand, this provides preliminary evidence in favor
of the bene fi t of practicing medical diagnosis
with BioWorld. On the other hand, the increase in
levels of accuracy showed more variability across
students and was not statistically signi fi cant,
which suggests that medical students may some-
times be overcon fi dent in submitting their fi nal
diagnoses.
Finally, the pre-/post-questionnaire data revealed
that learners put in a lot of effort solving cases and
found BioWorld to be a useful learning tool.
Discussion
The competitive entry requirements for medical
school generally result in cohorts of students that
are of high ability and motivated to succeed. Such
students are likely to be receptive to the introduc-
tion of computer-based tools that provide prac-
tice problems and insight into expert clinical
reasoning. In fact, we found this to be true in
study 3 where we used a modi fi ed version of
Boekaert’s ( 2002 ) motivation questionnaire as a
premeasure that demonstrated that students were
motivated to use BioWorld. BioWorld provides
an authentic experience of clinical reasoning,
complementing the time-constrained learning
within a hospital setting.
For students to bene fi t from the learning
opportunities provided, they must develop appro-
priate domain-speci fi c self-regulated learning
skills. BioWorld is designed to make learners’
medical diagnostic processes more visible, thus
fostering metacognitive skills that are critical to
reaching a fi nal diagnosis. In the following sec-
tion, we summarize the ways in which BioWorld
served as an external regulator of medical stu-
dents’ diagnostic reasoning and how the BioWorld
context played an important role in stimulating
engagement and motivation to learn.
Empirical Support for SRL
with BioWorld
In the fi rst study, expert solution visualization
maps were presented to medical students after
they submitted their fi nal diagnosis for each case.
Our assumption was that the maps would promote
metacognitive processes in that students would
re fl ect on their diagnostic reasoning skills by
monitoring and comparing their own learning
processes with that of an expert. We examined
whether students who had the bene fi t of such
maps would outperform students in terms of diag-
nostic accuracy and also achieve higher levels of
240 S.P. Lajoie et al.
con fi dence. Group differences were nonsigni fi cant
in that both groups improved their performance
accuracy and con fi dence levels as a function of
working with BioWorld cases. The effects of the
maps appear to be small. Further analyses will be
conducted on the think aloud protocol data in
order to determine whether students pinpoint dis-
crepancies between their own performance and
that of experts. Future directions may include
more directed animations of how individuals dif-
fered from the experts or incorporating human
instructors in abstracted replay sessions that
debrief the diagnostic reasoning process.
Our assumption was that the evidence table
would foster self-regulatory processes that are
critical to diagnostic reasoning. In particular, the
table should help learners monitor their reason-
ing and help them be more aware of the implica-
tions of their evidence in relation to the accuracy
of their diagnosis. If using the table correctly,
learners should engage in remedial strategies,
i.e., readjusting their hypotheses when diagnos-
tic test results did not con fi rm their diagnosis.
These self-regulatory processes should result in
medical students eventually prioritizing their
evidence in a more expert-like manner. As such,
we examined both medical students and physi-
cians to see whether they collected and used evi-
dence differently. Not surprisingly, our fi ndings
support the expertise literature (Chi et al.,
1988 ) ,
in that the physicians (experts) engaged in more
metacognitive processing than medical students
(relative novices). The experts in this study were
relatively selective in the type of evidence they
used and prioritized as compared to the novices.
Case dif fi culty was considered a possible con-
founding effect in the learning process, and
hence, study 2 counterbalanced this variable and
found a developmental trajectory in that students
became more expert-like as they practiced medi-
cal diagnosis with BioWorld.
Study 3 expanded on the results from study 2
in that we explored the relationship between the
evidence medical students collected and priori-
tized and used in their case summaries. Once
again, we anticipated improved performance as a
result of practice with BioWorld, as shown in an
increase in the amount of expert-like evidence
items over time. The results con fi rmed this train-
ing effect since there was a signi fi cant increase
from the fi rst to the second day in the overlap
between medical students and experts in terms of
proportion of evidence prioritized. This fi nding
provides support to our hypothesis that the evi-
dence table served effectively as a metacognitive
tool. Students were more likely to exhibit SRL
monitoring and control behaviors for the later
cases, suggesting that BioWorld stimulated meta-
cognitive awareness in a progressive or develop-
mental manner, which supports Winne and
colleagues’ model of SRL following a develop-
mental trajectory. A similar trend is suggested by
the case summary data, but we were unable to
detect statistically signi fi cant relationships, likely
due to missing data and the resulting low power
of the analysis. Future studies will need to con-
sider longer treatments to promote stronger
effects. Furthermore, we will need to overcome
the small sample size issues. We may need to fi nd
alternative method of recruiting participants in
this specialized area who already have a full med-
ical curriculum that competes for their time.
The third study also revealed that students put
in a great deal of effort solving cases in BioWorld
and that BioWorld successfully engaged student
interest and motivation to learn. Students reported
that BioWorld met their initial expectations, sug-
gesting that they are able to employ metacogni-
tive judgment in terms of assessing the prospective
helpfulness of learning tools. These fi ndings sup-
port Bandura’s notion that an individual’s will to
learn is necessary to maintain effortful learning
and suggests that BioWorld is effective in provid-
ing a context that facilitates students’ ability to
self-regulate (Pintrich, 2000 ) .
Overall, the three studies reveal that medical
students generally increased from pre- to post-
test, in terms of accuracy of their fi nal diagnosis,
although not signi fi cantly. The sample of students
demonstrated relatively high ability in solving
cases. In analyzing the accuracy differences, we
noted that it was possible for students to obtain
the correct diagnosis, but they miss important
aspects of the case, i.e., a possible life-threaten-
ing complication. Evaluating accuracy as an iso-
lated variable may thus be an inappropriate
24116 Technology Rich Tools
measure of students’ clinical reasoning ability.
Measures such as the proportion of prioritized
evidence items that were expert-like provide a
more nuanced measurement of students’ ability
to discriminate between expert-like and non-
expert-like evidence.
Students who used BioWorld demonstrated a
statistically signi fi cant increase in con fi dence
level from pre- to posttest. We anticipated that
con fi dence would increase with better problem-
solving ability. However, we also observed inci-
dents of overcon fi dence, in which students
selected a high value for the belief meter, but did
not document appropriate evidence to support
their diagnoses. The literature suggests that nov-
ice physicians fi nd it particularly dif fi cult to
accurately assess their level of competence when
they are “unskilled and unaware of it” (Hodges,
Regehr, & Martin,
2001 ) . To address this situa-
tion, it may be necessary for BioWorld to incor-
porate additional metacognitive scaffolds to
prompt students to re fl ect on how they arrived at
a particular diagnosis and how often they have
encountered such a disease in their previous stud-
ies and clinical experience.
Implications and Future Research
We have provided evidence that advanced learning
technologies can be designed to support self-regu-
lated learning. Our goal in designing BioWorld was
to help novice medical students become more
expert-like in the processes they take to make clini-
cal decisions. The expertise literature demonstrates
that one dimension of expertise is higher metacog-
nitive skills. Tools in BioWorld such as the evidence
table support participants in their metacognitive
monitoring of the choices they make while trying to
solve a patient case as well as support for decisions
and control of what they see as relevant or irrelevant
to the overall decision-making process. In this
regard, the evidence table was found to be effective
in promoting more expert-like behavior, as a func-
tion of supporting metacognitive skills. However,
the expert solution visualization maps were also
designed to support self-regulation by providing a
post-re fl ection tool to compare one’s own decisions
with that of an expert. However, we did not fi nd
support that this feature was used appropriately.
Future research is needed to determine if further
scaffolding is needed in how to use these maps.
Future studies are needed that are of longer dura-
tion, with more patient cases to solve and with more
students to validate the current research.
References
Azevedo, R. (2009). Theoretical, methodological, and
analytical challenges in the research on metacognition
and self-regulation: A commentary. Metacognition &
Learning, 4 (1), 87–95.
Azevedo, R., Moos, D. C., Greene, J. A., Winters, F. I., &
Cromley, J. G. (2008). Why is externally-facilitated
regulated learning more effective than self-regulated
learning with hypermedia? Educational Technology
Research and Development, 56 , 45–72.
Azevedo, R., Moos, D. C., Witherspoon, A. M., &
Chauncey, A. D. (2010). Measuring cognitive and meta-
cognitive regulatory processes used during hypermedia
learning: Theoretical, conceptual, and methodological
issues. Educational Psychologist, 45 (4), 1–14.
Bandura, A. (1986). Social foundations of thought and
action: A social cognitive theory . Englewood Cliffs,
NJ: Prentice-Hall.
Bandura, A. (1997). Self-ef fi cacy: The exercise of control .
New York: Freeman.
Boekaerts, M. (1997). Self-regulated learning: A new
concept embraced by researchers, policy makers, edu-
cators, teachers, and students. Learning and
Instruction, 7 (2), 161–186.
Boekaerts, M. (2002). The on-line motivation question-
naire: A self-report instrument to assess students’ con-
text sensitivity. In P. Pintrich & M. Maehr (Eds.), New
directions in measures and methods (pp. 77–120).
Oxford, UK: Elsevier.
Butler, D. L., & Winne, P. H. (1995). Feedback and self-
regulated learning: A theoretical synthesis. Review of
Educational Research, 65 (3), 245–281.
Chi, M. T. H., Glaser, R., & Farr, M. J. (Eds.). (1988). The
nature of expertise . Hillsdale, NJ: Erlbaum.
Collins, A., Brown, J. S., & Newman, S. E. (1987).
Cognitive apprenticeship: Teaching the craft of read-
ing, writing, and mathematics. In L. B. Resnick (Ed.),
Knowing, learning, and instruction: Essays in honor
of Robert Glaser (Vol. 5, pp. 453–494). Hillsdale, NJ:
Erlbaum.
Corno, L., & Mandinach, E. B. (1983). The role of cogni-
tive engagement in classroom learning and motivation.
Educational Psychologist, 18 , 88–108.
Dinsmore, D. L., Alexander, P. A., & Loughlin, S. M.
(2008). Focusing the conceptual lens on metacogni-
tion, self-regulation, and self-regulated learning.
Educational Psychology Review, 20 , 391–409.
242 S.P. Lajoie et al.
Ericsson, K. A., & Simon, H. A. (1993). Protocol analy-
sis; Verbal reports as data (revised edition) . Cambridge,
MA: Bradfordbooks/MIT.
Flavell, J. H. (1979). Metacognition and cognitive moni-
toring: A new area of cognitive-developmental inquiry.
American Psychologist, 34 , 906–911.
Gauthier, G. (2009). Capturing and representing the rea-
soning processes of expert clinical teachers for case-
based teaching . Unpublished doctoral dissertation.
McGill University, Montreal, Canada.
Gauthier, G., Naismith, L., Lajoie, S. P., & Wiseman, J.
(2008). Using expert decision maps to promote
re fl ection and self-assessment in medical case-based
instruction. In V. Aleven, K. Ashley, C. Lynch, & N.
Pinkwart (Chairs), Intelligent tutoring systems for ill-
de fi ned domains (pp. 68–80). Workshop conducted at
the 9th International Conference on Intelligent
Tutoring Systems, Montreal, Canada.
Greene, J. A., & Azevedo, R. (2010). The measurement of
learners’ self-regulated cognitive and metacognitive
processes while using computer-based learning envi-
ronments. Educational Psychologist, 45 (4), 203–209.
Hodges, B., Regehr, G., & Martin, D. (2001). Dif fi culties
in recognizing one’s own incompetence: Novice phy-
sicians who are unskilled and unaware of it. Academic
Medicine, 76 (10), S87–S89.
Lajoie, S. P. (2003). Transitions and trajectories for studies
of expertise. Educational Researcher, 32 (8), 21–25.
Lajoie, S. P. (2008). Metacognition, self regulation, and
self-regulated learning: A rose by any other name?
Educational Psychology Review, 20 , 469–475.
Lajoie, S. P. (2009). Developing professional expertise with a
cognitive apprenticeship model: Examples from Avionics
and Medicine. In K. A. Ericsson (Ed.), Development of
professional expertise: Toward measurement of expert
performance and design of optimal learning environ-
ments (pp. 61–83).: Cambridge University Press.
Lajoie, S. P., & Azevedo, R. (2006). Teaching and learn-
ing in technology-rich environments. In P. Alexander
& P. Winne (Eds.), Handbook of educational psychol-
ogy . Mahwah, NJ: Erlbaum.
Lajoie, S. P., Naismith, L., Poitras, E., Hong, Y. J.,
Panesso-Cruz, I., Ranellucci, J., & Wiseman, J.
(in prep.) BioWorld: An advanced learning technology
for authentic inquiry in medical reasoning.
Maudsley, G., & Strivens, J. (2000). Promoting profes-
sional knowledge, experiential learning and critical
thinking for medical students. Medical Education, 34 ,
535–544.
McCaslin, M., & Hickey, D. T. (2001). Educational
psychology, social constructivism, and educational
practice: A case of emergent identity. Educational
Psychologist, 36 (2), 133–140.
McCurdy, N., Naismith, L., & Lajoie, S. P. (2010). Using
metacognitive tools to scaffold medical students devel-
oping clinical reasoning skills. In Cognitive and meta-
cognitive educational systems (Tech. Rep. FS-10-01,
pp. 52–56). Menlo Park, CA: AAAI.
Pintrich, P. R. (2000). The role of goal orientation in self-
regulated learning. In M. Boekaerts, P. R. Pintrich,
& M. Zeidner (Eds.), Handbook of self-regulation
(pp. 451–502). San Diego, CA: Academic.
Pintrich, P. R., & De Groot, E. V. (1990). Motivational and
self-regulated learning components of classroom aca-
demic performance. Journal of Educational
Psychology, 82 (1), 33–40.
Pintrich, P. R., Wolters, C., & Baxter, G. (2000). Assessing
metacognition and self-regulated learning. In G.
Schraw & J. Impara (Eds.), Issues in the measurement
of metacognition (pp. 43–97). Lincoln, NE: Buros
Institute of Mental Measurements.
Schoenfeld, A. H. (1983). Mathematical problem solving .
Orlando, FL: Academic.
Winne, P. H. (2001). Self-regulated learning viewed from
models of information processing. In B. J. Zimmerman
& D. H. Schunk (Eds.), Self-regulated learning and
academic achievement: Theoretical perspectives (2nd
ed., pp. 153–189). Hillsdale, NJ: Erlbaum.
Winne, P. H., & Hadwin, A. F. (1998). Studying as
self-regulated learning. In D. J. Hacker, J. Dunlosky,
& A. C. Graesser (Eds.), Metacognition in educa-
tional theory and practice (pp. 277–304). Mahwah,
NJ: Erlbaum.
Winne, P. H., & Perry, N. E. (2000). Measuring self-
regulated learning. In M. Boekaerts, P. R. Pintrich,
& M. Zeidner (Eds.), Handbook of self-regulation
(pp. 531–566). San Diego, CA: Academic.
Zimmerman, B. J. (2000). Attaining self-regulation: A
social cognitive perspective. In M. Boekaerts, P. R.
Pintrich, & M. Zeidner (Eds.), Handbook of self-
regulation (pp. 13–39). San Diego, CA: Academic.
Zimmerman, B. J., & Schunk, D. H. (Eds.). (2001). Self-
regulated learning and academic achievement: theo-
retical perspectives (2nd ed.). Mahwah, NJ: Erlbaum.
