Conference PaperPDF Available

An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators


Abstract and Figures

This paper elaborates on the evaluation of a Metacognitive Scaffolding Service (MSS), which has been integrated into an already existing and mature medical training simulator. The MSS is envisioned to facilitate self-regulated learning (SRL) through thinking prompts and appropriate learning hints enhancing the use of metacognitive strategies. The MSS is developed in the European ImREAL (Immersive Reflective Experience-based Adaptive Learning) project that aims to augment simulated learning environments throughout services that are decoupled from the simulation itself. Results comparing a baseline evaluation of the 'pure' simulator (N=131) and a first user trial including the MSS (N=143) are presented. The findings indicate a positive effect on learning motivation and perceived performance with consistently good usability. The MSS and simulator are perceived as an entity by medical students involved in the study. Further steps of development are discussed and outlined.
Content may be subject to copyright.
adfa, p. 1, 2011.
© Springer-Verlag Berlin Heidelberg 2011
An Initial Evaluation of Metacognitive Scaffolding for
Experiential Training Simulators
Marcel Berthold1, Adam Moore2, Christina Steiner1, Conor Gaffney3, Declan Dag-
ger3, Dietrich Albert1, Fionn Kelly4, Gary Donohoe4, Gordon Power3, Owen Conlan2
1Knowledge Management Institute, Graz University of Technology,
Rechbauerstr. 12, 8010 Graz, Austria
2Knowledge, Data & Engineering Group, School of Computer Science and Statistics,
Trinity College, Dublin, Ireland
3EmpowerTheUser, Trinity Technology & Enterprise Campus,
The Tower, Pearse Street, Dublin, Ireland
4Department of Psychiatry, School of Medicine, Trinity College, Dublin, Ireland;
Abstract. This paper elaborates on the evaluation of a Metacognitive Scaffold-
ing Service (MSS), which has been integrated into an already existing and ma-
ture medical training simulator. The MSS is envisioned to facilitate self-
regulated learning (SRL) through thinking prompts and appropriate learning
hints enhancing the use of metacognitive strategies. The MSS is developed in
the European ImREAL (Immersive Reflective Experience-based Adaptive
Learning) project that aims to augment simulated learning environments
throughout services that are decoupled from the simulation itself. Results com-
paring a baseline evaluation of the pure simulator (N=131) and a first user
trial including the MSS (N=143) are presented. The findings indicate a positive
effect on learning motivation and perceived performance with consistently good
usability. The MSS and simulator are perceived as an entity by medical students
involved in the study. Further steps of development are discussed and outlined.
Keywords: self-regulated learning, metacognitive scaffolding, training simula-
tor, augmentation
1 Introduction
Self-regulated learning (SRL) and especially metacognition, is currently a prominent
topic in technology-enhanced learning (TEL) research. Many studies provide evi-
dence of the effectiveness of SRL in combination with metacognitive scaffolding (cf.
[1, 2]). Self-regulated learning refers to learning experiences that are directed by the
learner and describes the ways in which individuals regulate their own cognitive and
metacognitive processes in educational settings (e.g. [3, 4]). An important aspect of
self-regulated learning is therefore the learners’ use of different cognitive and meta-
cognitive strategies, in order to control and direct their learning [5]. These strategies
include cognitive learning strategies, self-regulatory strategies to control cognition
(i.e. metacognitive strategies) and resource management strategies. Self-regulated
learning also involves motivational processes and motivational beliefs [4]. It has been
shown that good self-regulated learners perform better and are more motivated to
learn [6] than weak self-regulated learners. TEL environments provide opportunities
to support and facilitate metacognitive skills, but most learners need additional help
and guidance [7] to perform well in such environments.
In the EU project, ImREAL1 (Immersive Reflective Experience-based Adaptive
Learning), intelligent services are being developed to augment and improve experien-
tial simulated learning environments including one to scaffold metacognitive proc-
esses. The development of the scaffolding service focuses on the salient and timely
support of learners in their metacognitive processes and self-regulated learning in the
context of a simulation environment. Herein we report a concrete study examining the
medical training simulator provided by EmpowerTheUser2 augmented with the Im-
REAL Metacognitive Scaffolding Service (MSS). The service will provide prompts
and suggestions adapted to a learner’s needs and traits of metacognition and aiming at
enhancing motivation towards the learning activity in the simulation. While the aspect
of supporting metacognition needs to be integrated in the learning process, the accord-
ing service will be technically decoupled from the specific learning system itself.
Overall, the research presented investigates the effectiveness and appropriateness of
the service and the scaffolding it provides. To allow a more detailed examination of
the issues, we address four sub questions:
1. Is self-regulated learning supported? For the evaluation and analysis of self-
regulated learning we distinguish between the general learning approach (i.e. ap-
plication of cognitive, metacognitive strategies), and the metacognitive and specific
learning processes in the simulation (i.e. cognitive, metacognitive strategies or ac-
tions within simulator context); thereby, learning and metacognitive scaffolding in
the simulation may optimally, and on a long-term basis, influence the general
learning approach of a learner.
2. Does the simulator augmentation through the service lead to better learning !
performance?" The learning performance refers to the (objective or subjec-
tive/perceived) learners’ knowledge/competence acquisition and performance in
the learning situation and to the transfer of acquired knowledge to other situations.
3. Does the simulator augmentation through the service increase motivation? The
aspect of motivation addresses the motivation to learn, i.e. the structures and proc-
esses explaining learning actions and the effects of learning [8].
4. Is the service well integrated in the simulation and learning experience? This
refers to the question whether the scaffolding interventions provided during the
simulation via the MSS are perceived by learners as appropriate and useful in
terms of their content, context and timing.
In order to answer these evaluation questions the paper is organized in the follow-
ing structure. Section 2 gives an overview of the simulator and normal usage. Section
3 gives an overview of the MSS service and outlines related work. In section 4 the
experimental design of the study is introduced and section 5 includes the according
results. These results are discussed in section 5. Section 6 provides a conclusion and
an outlook to further research.
2 Metacognitive Scaffolding – background and technology
Scaffolding is an important part of the educational process, supporting learners in
their acquisition of knowledge and developing their learning skills. Scaffolding has
been a major topic of research since the pioneering work of Vygotsky (e.g. 1978 [9])
and the key work of Bruner and Wood and colleagues (cf. [10]).
Work on the use of scaffolding with the help of computer-based learning environ-
ments has been extensive (cf. [11]). Originally, the emphasis was on cognitive scaf-
folding which has many forms (cf. [12]). In the last ten years there has been a move
towards research in metacognitive scaffolding (e.g. [1316]) as well as in the use of
metacognitive scaffolding in adaptive learning environments (e.g. [1720]).
Other forms of scaffolding have also been explored both in educational and tech-
nology enhanced learning contexts such as affective scaffolding and conative scaf-
folding. Van de Pol et al. [13] sought to develop a framework for the analysis of dif-
ferent forms of scaffolding. In the technology enhanced learning community, Poray-
ska-Pomsta and Pain [21] explored affective and cognitive scaffolding through a form
of face theory (the affective scaffolding also included an element of motivational
scaffolding). Aist et al. [22] examined the notion of emotional scaffolding and found
different kinds of emotional scaffolding had an effect on children's persistence using a
reading tutoring system.
There are different forms of metacognitive scaffolding. Molenaar et al. [2] investi-
gated the distinction between structuring and problematizing forms of metacognitive
scaffolding and found that problematizing scaffolding seemed to have a significant
effect on learning the required content. They used Orientation, Planning, Monitoring,
Evaluation and Reflection as subcategories of metacognitive scaffolding.
Sharma and Hannafin [23] reviewed the area of scaffolding in terms of the implica-
tions for technology enhanced learning systems. They point out the need to balance
metacognitive and “procedural” scaffolds since only receiving one kind can lead to
difficulties with only procedural scaffolding students take a piecemeal approach,
and with only metacognitive scaffolding students tend to fail to complete their work.
They also argue for systems that are sensitive to the needs of individuals. Boyer et al.
[24] examined the balance between motivational and cognitive scaffolding through
tutorial dialogue and found evidence that cognitive scaffolding supported learning
gains while motivational scaffolding supported increase in self-efficacy.
The aim of the ImREAL project is to bring simulators closer to the ‘real world’. As
part of training for a diagnostic interview, in the ‘Real World’ a mentor sits at back
observing and providing occasional input / interventions as necessary. The MSS has
been developed to integrate into the simulator learning experience as an analogue of a
mentor, sitting alongside the simulator to provide scaffolding. The ETU simulator
supports meta-comprehension and open reflection via note taking.
For this trial metacognitive scaffolding was provided using calls to a RESTful [25]
service developed as part of the ImREAL project. The service utilises technology
initially developed for the ETTHOS model [26] and presents Items from the Meta-
cognitive Awareness Inventory [27] according to an underlying cognitive activity
model, matched to Factors in the MAI. In this way the importance of the tasks being
undertaken by the learner is clear scaffolding is developed in order to match a learn-
ers’ cognitive activity to metacognitive support.
The scaffolding service supplements the pre-existing ETU note-taking tool, both of
which are illustrated in Figure 1 below. The text of the thinking prompt item is
phrased in order to elicit a yes/no response. If additional context / rephrasing has been
added by the instructional experts, that is displayed before the open text response
area. A link that activates an explanatory text occurs underneath the text input area, as
well as a “Like button” which can be selected and the submit action.
Fig. 1. a) MSS Interface b) ETU Note-taking tool
3 Overview of Simulator and Normal Usage
For this research the ETU Talent Development Platform was used, with training for
medical interview situations. The user plays the role of a clinical therapist and selects
interview questions from a variety of possible options to ask the patient. When a ques-
tion is selected a video is presented that shows the verbal interaction of the therapist
with the patient (close up of the patient, voice of the therapist) and the verbal and non-
verbal reaction of the patient (close up of the patient). Starting the simulation, users
can choose between two types of scenarios (Depression and Mania), which offer the
same types of subcategories: Introduction and negotiating the agenda, eliciting in-
formation, outlining a management plan and closing the interview.
After a scenario is chosen, the user may simulate the interview as long as they pre-
fer or until the interview is “naturally” finished. Furthermore, the users could have as
many runs of the simulation as they want and could choose a different scenario in the
following attempts. In this study we focused only on the Depression interview sce-
nario. A screenshot of a typical interaction within the ETU system is show below in
Figure 2.
Fig. 2. Screen shot of the EmpowerTheUser Simulator. The scenario of diagnosing a pa-
tient with clinical depression is just beginning!
4 Experimental design
4.1 Cohort
143 medical students participated in the study and performed the simulation as part of
their second year (2011/2012) medical training at Trinity College, Dublin (TCD).
They were on average approximately 22 years old (40% male vs. 60% female, 80%
Irish). In addition, these results are compared, as far as they have been assessed at
both time points, to a baseline evaluation based on using the simulator without Im-
REAL services. In the baseline evaluation, 131 TCD medical students from the previ-
ous year group (2010/2011) participated (cross-section design).
4.2 Measurement Instrument
ETU simulator. Within the simulation learning performance is assessed by tracking
scores for each of the 4 subsections, as well as dialogue scores and notes that were
written in an available note pad for reflections.
Questionnaire on Self-Regulated Learning. Self-regulated learning skills were
measured by the Questionnaire for Self-Regulated Learning (QSRL; [28]). The QSRL
consists of 54 items, which belong to six main scales (Memorizing / Elaboration /
Organization / Planning / Self-monitoring / Time management) and three subscales
(Achievement Motivation / Internal attribution / Effort). In the online version of the
questionnaire, respondents indicate their agreement to an item by moving a slider on
an analogue scale between the positions “strongly disagree” to “strongly agree”. The
possible score range is 0-100 in each case, with higher values indicating a better re-
Survey questions on use of ETU, experience in performing clinical interviews
and relevance of the ETU simulator. In order to control possible influences of prior
experience with the respective simulated learning environment or real world medical
interviews, their experiences were assessed through three survey questions. The fol-
lowing survey question assessed the relevance of the simulator with answer options
ranging from “not relevant at all” to “very relevant”.
Thinking Prompts. Triggers that made calls to the MSS were inserted into the prac-
tice phase of the simulator but not available during ‘live/scored usage’. The triggers
were created using the ETU authoring platform and made a call to the MSS request-
ing a prompt of a particular Factor (Planning, Information Management, Comprehen-
sion, Debugging or Evaluation). As explained above, each Factor consisted of a num-
ber of Items or Thinking Prompts. An item was not redisplayed once a reflection had
been entered with it.
Motivation. Motivation was assessed with four survey questions referring to learning
more about clinical interviews, improve own interview skills, performing a good in-
terview during the simulation and applying what has been learned in a real interview.
Workload. Measures of workload were assessed by six subscales of the NASA-TX
[29, 30] with a score range of 0-100. In this case higher values indicate a higher work-
load. An overall workload score was calculated based on the subscales by computing
a mean of all item contributions. In contrast to the original NASA-TLX, students did
not mark their answers to an analogue scale, but entered digits between 0-100 into a
text field.
Usability and Service specific integration. The Short Usability Scale (SUS, [31])
consists of ten items with answer options of a five-point rating-scale ranging from
“strongly disagree” to strongly agree”. The raw data were computed to an overall
SUS score. The overall SUS score ranged from 0-100 with higher values indicating
higher usability. Additionally to the SUS questions, three service specific usability
questions were administered regarding the relation of the prompts to the rest of the
simulation and obvious differences. The answer options were the same as for the
Learning Experience with MSS. Learning experience with MSS was measured by
10 questions referring to helpfulness and appropriateness of the MSS thinking
prompts within the simulator with answer options on a 5-point-Likert-scale ranging
from “not at all” to “very much”. In addition, a free text comment field was provided.
Procedure. The baseline evaluation, using the pure simulator, was conducted in mid-
February and beginning of March of 2011; the first user trial was carried out in Dub-
lin from mid-February until beginning of March of 2012. The TCD medical students
used the ETU medical training simulator.
Data collection was carried out during the simulation (e.g. ETU scores, MSS data)
and after learning with the ETU simulator (questionnaire data). At first the students
worked on the simulation for as long as they wanted and could choose between two
scenarios; Mania or Depression. After they were finished they were directed to the
online questionnaires. In this stage they filled in the survey questions on relevance
and on motivation, NASA-TLX, SUS, questions on prompts, learning experience and
the QSRL.
After working on the simulation in the TCD course students still had access to the
ETU simulator via the internet for approximately two weeks. It was not mandatory to
use the simulation in the medical course at TCD or to participate in the evaluation.
5 Experimental results
PAWS Statistics, version 18.0 [32] and Microsoft Excel (2010) were used for statisti-
cal analyses and graphical presentations. If not explicitly mentioned, statistical re-
quirements for inference statistical analyses and procedures were fulfilled. For all
analyses the alpha level was α=.05. Due to an unbalanced number of participants in
the samples in regard to comparisons of the first user trial and baseline evaluation
appropriate pre-tests have been performed and the according values are presented.
This section focuses mainly on the first user trial evaluation based on using the
ETU simulator with the integrated MSS ImREAL services.
5.1 Log-Data
ETU Simulator Descriptive data. All students of the first user trial reported that
they have never used the ETU medical training simulator before. Nonetheless, they
were quite experienced in conducting clinical interviews, since 97% reported to have
already performed at least one, but only 15 % had experienced interviewing a psychi-
atric patient.
A comparison of the first user trial and the baseline evaluation showed that dura-
tion time in minutes (Mbase=17.89, SDbase=11.15; M1UT=15.45, SD1UT =6.81) and
scored points (Mbase=31.34, SDbase=6.33; M1UT=27.61, SD1UT =5.91) in the simulation
decreased from baseline evaluation to the first user trial (duration time: t211,49=2.17,
p=.031; score: t272=5.10, p=.000). These results show that students spent on average
less time in the simulator and reached lower scores. This cannot be explained by more
experience of the baseline cohort, because they also had not used the simulator before.
Metacognitive Scaffolding Service (MSS) comments. 10 comments have been col-
lected by MSS learning experience questionnaire free text comment field. The par-
ticipants provided interesting comments, which however referred more to the simula-
tor than to the MSS. This implies that the MSS seems to be perceived as well inte-
grated in the simulation, because students do not seem to differentiate between the
additional service and the simulation itself. The participants pointed to sometimes
inappropriate prompts in combination with the simulator in situations, especially,
when only one answering option was available in the dialogue with the patient and
they were asked to think about their strategy. Nevertheless, one learner recorded that
I am learning a lot actually, it is amazing how much you can miss just by asking a
question in a slightly different way! I keep going back a step and looked through the
other options to see where the scenario goes. Usually I’ve picked the most suitable
one, but not always. Sometimes I am surprised about how much I would have
missed!!which overall demonstrates good engagement and a positive view of the
system. !
Prompts analysis. There have been five different types of prompts presented accord-
ing to the five MAI phases described in section 3. In total 2001 prompts (Planning:
469, Information Management: 752, Monitoring: 425, Debugging: 301 and Reflec-
tion: 54) were provided to 50 students. Every student who used the practice facility in
the simulator was presented with a pop-up suggesting they reflected. Clicking on that
pop-up would move the simulation to the MSS screen (Figure 2a). The relative fre-
quency of the prompts was compared to the expected frequency based on the prob-
ability of available prompts for each phase. The results indicate that the learners were
scaffolded more often in the second phase “Information Management” and were less
scaffolded in the reflection phase as could have been expected (
(4,0.95)= 314.55,
p=.000, Figure 3). On the one hand, learners seem to need more assistance in effec-
tively processing information by hints to use more organizational, elaborative, sum-
marizing or selective learning strategies. On the other hand they are rather confident
in the reflection phase and wave the offer of scaffolds.
Fig. 3. Comparison of the expected and empirical distribution of metacognitive scaffolds for
the five phases of Schraw’s Metacognitive Awareness Inventory!
5.2 Questionnaire Data
QSRL. The quantitative results of the MSS service are a little surprising, because
students estimated the use of cognitive learning strategies, especially elaboration
strategies, relatively high. In general, all SRL scores are located above the center
point of the score range and indicate positive results for all cognitive and metacogni-
tive strategies. However, a stronger use of elaboration strategies is reported (t20 =
3.34, p=.003) in the first user trial. It needs to be explicitly stated that this is not an
unfavorable result as such as elaboration strategies are strategies of deeper learning
[33], which should be further supported by scaffolding services.
A comparison to the baseline study shows no significant increase in any of the us-
age of reported learning strategies.
Motivation. 38 participants filled in the motivation questions. The results show that
the scores were on a high motivation level around 3.16-3.49 on a 4-point-Likert rating
scale. This implies that the students were very motivated to learn about the clinical
interview during the simulation, to improve their interview skills, perform a good
interview during the simulation and to apply what they have just learnt in the simula-
tion in a real world clinical interview context. Furthermore, a comparison of the over-
all motivation scores assessed immediately after the simulation of the first user trail
and immediately after the baseline evaluation reveals significant higher motivation
scores for the MSS trial (M=3.35, SD=.4.14) compared to the baseline (M=2.48,
SD=.73; t118.47=-8.64, p=.000).
Workload. A moderate overall workload could be observed. It has to be noted at this
stage that for a learning environment it should not be aimed at reducing the workload
to a minimum; rather, the challenge should be at an appropriate, medium level of
challenge in an optimal case adapted to the individual learner. Participants reported
the highest, but still moderately pronounced, load for effort. This subscale refers to
the mental and physical resources that had to be mobilized to accomplish the task.
Consequently, the result for effort can be relegated rather to mental than physical
demand. Yet the simulation is a complex program that supports and requires active
learning processes; a reduction of mental demand is somewhat challenging, but could
possibly be realized by improving the MSS and addressing the challenge to reduce
repetitions and provide only appropriate scaffolds. Furthermore, the second highest
score was observed for performance (see Figure 4), showing that the students felt they
successfully accomplished what they were supposed to do. Performance scores (refer-
ring to subjective/perceived learning outcome) even increased for the first user trial
(M=54.83, SD=17.90) compared to the baseline evaluation (M=43.00, SD=23.68)
significantly (t109=-2.63, p=.01). A t-test for independent groups remains insignificant
comparing overall workload scores for the first user trial (M=44.19, SD=10.86) and
baseline evaluation (M=44.81, SD=12.011; t75.52=.27, ns.).
Fig. 4. Comparison of Baseline and first user trial data for workload
Usability and Service specific integration. No differences could be observed be-
tween the rather high usability overall scores for the first user trial (M=62.50,
SD=17.90) and the baseline evaluation (M=62.80, SD=16.08; F57.90=.90, ns.).
With respect to service specific integration with the ETU medical training and MSS
prompts the majority of the students ratings were positive with 21 out of 33 stating
they felt supported during their learning process by the MSS service and that the serv-
ice was well integrated in the system (M=3.26, SD=.40).
Learning Experience with MSS. The learning experience with the MSS was rela-
tively positive. More than 63% of the participants perceived the MSS learning experi-
ence overall as very much helpful and appropriate. The high score for the individual
items were all above the center point of the scale, which underlies this encouraging
6 Discussion
In this paper we examined the effectiveness and appropriateness of the MSS. Results
of the first user trial have been reported, involving the ETU medical training simula-
tor augmented with the MSS service. These results have been compared to a baseline
evaluation where the pure’ simulator was administered without any additional Im-
REAL services. Addressing the evaluation questions stated in the introduction sec-
Is self-regulated learning supported? Even though self-regulated learning and
metacognitive scaffolding are closely connected, because the SRL process heavily
relies on applying cognitive and especially metacognitive learning strategies and
techniques, no changes in SRL profile could be observed comparing the first user trial
to the baseline data. This is because influencing self-regulated learning aspects is
rather a long-term process [34]. This result might also be explained by having a look
at the usage frequency of the simulator. The students were confronted with the simu-
lation only in the TCD course and no one had used the simulation before. Further-
more, duration time of working with the simulator, which was on average less than
half an hour, might not be too short to change a rather stable learning approach.
For future studies the application of a longitudinal approach could be suggested in-
stead of a cross-section evaluation, to better meet the requirements of a longer-term
process. In addition, teachers’ or supervisors’ judgments on SRL performance could
be included to assess their observations on potential changes in learners’ daily learn-
ing behavior. However, the last point might be difficult to realize in an university
setting with more than 140 students in a course.
In general, all SRL self-reports were positive, indicating a higher use of elaboration
strategies compared to memorizing strategies. Elaboration strategies represent strate-
gies of deeper learning [33]. Nevertheless, fostering memorizing/rehearsal strategies
might be taken up as an idea for improving the MSS. Assuming that the participants
in the evaluation trials constitute a representative sample of ETU simulator users,
ImREAL could start from this result and aim at improving users’ rehearsal strategies
through the provision of appropriate scaffolding. Of course, this strategy type should
not be the only one to be supported. Rehearsal strategies help the learner to select and
remember important information, but may not represent very deep levels of cognitive
processing [33]. As a result, ImREAL services should especially try to further support
elaboration as well as organizational strategies. In the ImREAL pedagogical frame-
work learning [35] is seen as a cyclic process of three phases: forethought, learning
and reflection. These individual phases are already represented in the ETU system,
but not covered comprehensively. As described above, medical students do not tend
to use the ETU simulator very often and if they do they undertake the interview sce-
nario only for a short period of time. Therefore, reflection and coverage of the SRL
phases should be further extended and supported by the ImREAL MSS.
Does the simulator augmentation through the service lead to better learning per-
formance? Results concerning the learning performance draw an ambiguous picture.
The actual objective data collected by the ETU simulator demonstrates that overall
scores decreased from the baseline evaluation to the first user trial. Surprisingly, self-
report scores on performance increased.
An increasing self-report score caused by the MSS that encourages learners to
think about one’s own performance and making the aspect of actually performing
well more conscious. This in turn might lead to a) better self-estimations and self-
efficiency, resulting in higher workload performance scores and b) to higher motiva-
tion scores (see evaluation question 3 of this section).
Does the simulator augmentation through the service increase motivation? In line
with the increasing scores on performance assessed by self-report are results on moti-
vation. Motivation scores increased from the baseline evaluation to the first user trial.
In addition to the consideration of motivation as a state characteristic, motivational
beliefs (motivational traits in terms of being more stable and outlasting than state
motivation) can be influenced by positive sounding scaffolds and hints to optimize
learning. If students see the prompts as support of their learning approach a positive
attitude to the whole learning process can be expected and could explain the current
result, because these motivational beliefs are factors influencing the initial motivation
of the learner [36].
Is the service well integrated in the simulation and learning experience? Results
on usability of the whole system (simulator + MSS service) and service specific inte-
gration provide evidence that the MSS service is well integrated in the simulation and
leads to real augmentation. This is not only demonstrated by the positive scores on the
service specific integration questions, but also by user comments, which were overall
quite positive. Such positive results may be attributed to the MSS operating in an
appropriately timely and salient manner, with the pop-up triggers appearing at appo-
site times created by the instructional design experts. Also the RESTful interface al-
lows calls to be made to an ETU simulator-specific interface for the MSS, ensuring
there are no obvious presentational and interactional differences between the hosting
simulator and the MSS.
7 Conclusions & Outlook
The results above demonstrate a clear advantage in providing a MSS to augment an
experiential training simulator, leading to more engaged, motivated learners without
overly burdening them or interrupting the flow of their learning experience. With
respect to the actual learning performance no positive effect could be identified. This
would be desirable to investigate in more detail in future studies. These further studies
should optimally be realized in a longitudinal evaluation approach, as well as an as-
sessment of real-world performance on medical interviews (i.e. learning transfer).
The collecting and monitoring of the development of motivation throughout both
evaluation runs is important, because in the next version of the ImREAL MSS service
there will be a strong focus on extending it by ‘affective scaffolding’, which will
mainly address motivational aspects. As a result, the data from the first user trial
evaluation (with metacognitive scaffolding ‘only’) will serve as benchmark for a
comparison with evaluation outcomes for the affective metacognitive scaffolding,
thus allowing to investigate the additional benefit of the affective part.
The MSS will be integrated within additional experiential simulators to investigate
the service’s capabilities for generalization and integration within different systems
and usage cases and to further evaluate its effect on learning experience.
Acknowledgement. The research leading to these results has received funding from
the European Community's Seventh Framework Program (FP7/2007-2013) under
grant agreement no 257831 (ImREAL project) and could not be realized without the
close collaboration between all ImREAL partners.
1. Veenman, M.V.J.: Learning to Self-Monitor and Self-Regulate. Regulation. (2008)
2. Molenaar, I., Boxtel, C.A.M. Van, Sleegers, P.J.C.: The effects of scaffolding metacogni-
tive activities in small groups. Computers in Human Behavior 26, 1727-1738 (2010)
3. Puustinen, M., Pulkkinen, L.: Models of Self-regulated Learning: A review. Scandinavian
Journal of Educational Research 45, 269-286 (2001)
4. Zimmerman, B.: Becoming a Self-Regulated Learner: An Overview. Theory Into Practice
41, 64-70 (2002)
5. Pintrich, P.R.: The role of motivation in promoting and sustaining self-regulated learning.
International Journal of Educational Research. 31, 459-470 (1999).
6. Veenman, M.V.J.: Learning to Self-Monitor and Self-Regulate. In: R. Mayer, P. Alexander
(eds.) Handbook of Research on Learning and Instruction. pp. 197-218. Routledge, New
York (2011)
7. Manuela, B.: Effects of Reflection Prompts when Learning with Hypermedia. Journal of
Educational Computing Research 35, 359-375 (2006)
8. Krapp, A.:: Die Psychologie der Lernmotivation. Perspektiven der Forschung und Proble-
me ihrer pädagogischen Rezeption [The psychology of motivation to learn. Perspectives of
research and their pedagogical reception]. Zeitschrift für Pädagogik 39, 187-206 (1993)
9. Vygotskiǐ, L.S., Cole, M.:: Mind in society: the development of higher psychological
processes. Harvard University Press, Cambridge, MA (1978)
10. Wood, D., Bruner, J.S., Ross, G.: The role of tutoring in problem solving. Journal Of Child
Psychology And Psychiatry 17, 89-100 (1976)
11. Lajoie, S.P.: Extending the Scaffolding Metaphor. Instructional Science 33, 541-557
12. Clark, A.: Towards a science of the bio-technological mind. International Journal of Cog-
nition and Technology 1, 21-33 (2002)
13. Pol, J., Volman, M., Beishuizen, J.: Scaffolding in TeacherStudent Interaction: A Decade
of Research. Educational Psychology Review 22, 271-296 (2010)
14. Dinsmore, D.L., Alexander, P.A., Loughlin, S.M.: Focusing the Conceptual Lens on Meta-
cognition, Self-regulation, and Self-regulated Learning. Educational Psychology Review
20, 391-409 (2008)
15. Greene, J., Azevedo, R.: The Measurement of Learners’ Self-Regulated Cognitive and
Metacognitive Processes While Using Computer-Based Learning Environments. Educa-
tional Psychologist 45, 203-209 (2010)
16. Azevedo, R.:: Does adaptive scaffolding facilitate students? ability to regulate their learn-
ing with hypermedia?*1. Contemporary Educational Psychology 29, 344-370 (2004)
17. Azevedo, R., Hadwin, A.F.: Scaffolding Self-regulated Learning and Metacognition Im-
plications for the Design of Computer-based Scaffolds. Instructional Science 33, 367-379
18. Luckin, R., Hammerton, L.: Getting to Know Me: Helping Learners Understand Their
Own Learning Needs through Metacognitive Scaffolding. In: S. Cerri, G. Gouardères, F.
Paraguaçu (eds.) System. vol. 2363, pp. 759-771. Springer Berlin / Heidelberg, (2002)
19. Roll, I., Aleven, V., McLaren, B.M., Koedinger, K.R.: Designing for metacognition
applying cognitive tutor principles to the tutoring of help seeking. Metacognition and
Learning 2, 125-140 (2007)
20. Roll, I., Aleven, V., McLaren, B.M., Koedinger, K.R.: Improving students’ help-seeking
skills using metacognitive feedback in an intelligent tutoring system. Learning and Instruc-
tion 21, 267-280 (2011)
21. Porayska-Pomsta, K., Pain, H.: Providing Cognitive and Affective Scaffolding Through
Teaching Strategies: Applying Linguistic Politeness to the Educational Context. Pain
3220, 18-21 (2004)
22. Aist, G. et al.: Experimentally augmenting an intelligent tutoring system with human-
supplied capabilities: adding human-provided emotional scaffolding to an automated read-
ing tutor that listens. Proceedings Fourth IEEE International Conference on Multimodal
Interfaces 483-490 (2002)
23. Sharma, P., Hannafin, M.J.: Scaffolding in technology-enhanced learning environments.
Interactive Learning Environments 15, 27-46 (2007)
24. Boyer, K. et al.: Balancing Cognitive and Motivational Scaffolding in Tutorial Dialogue.
In: Intelligent Tutoring Systems, pp. 239-249. Springer, (2008)
25. Fielding, R.T.: Architectural Styles and the Design of Network-based Software Architec-
tures, PhD-thesis, Citeseer (2000)
26. Macarthur, V., Moore, A., Mulwa, C., Conlan, O.:: Towards a Cognitive Model to Support
Self-Reflection: Emulating Traits and Tasks in Higher Order Schemata. In: EC-TEL 2011
Workshop on Augmenting the Learning Experience with Collaborative Reflection, Pal-
ermo, Sicily, Italy (2011)
27. Schraw, G., Sperling Dennison, R.: Assessing metacognitive awareness. Contemporary
Educational Psychology 19, 460475 (1994)
28. Fill Giordano, R., Litzenberger, M., Berthold, M.:: On the Assessment of strategies in self-
regulated learning (SRL)differences in adolescents of different age group and school
type. pp. Poster. 9. Tagung der Österreichischen Gesellschaft für Psychologie, Salzburg,
Salzburg (2010)
29. Hart, S.G., Staveland, L.E.: Development of NASA-TLX (Task Load Index): Results of
Empirical and Theoretical Research. In: N.M.P.A. Hancock (eds.) Human mental work-
load. pp. 239-250. North Holland Press, Amsterdam (1988)
30. Hart, S.G.: NASA-Task Load Index (NASA-TLX); 20 Years Later. Human Factors and
Ergonomics Society Annual Meeting Proceedings 50, 904-908 (2006)
31. Brooke, J.: SUS: A “quick and dirty” usability scale. In: P.W. Jordan, B. Thomas, B..
Weerdmeester, A.L. McClelland (eds.) Usability Evaluation in Industry. pp. 189-194. Tay-
lor & Francis, London (1996)
32. Inc, S.: PASW Statistics 18 Core System User ’s Guide. SPSS Inc., (2007)
33. Weinstein, C.E., Mayer, R.E.: The teaching of learning strategies. In: M. Wittrock (eds.)
Handbook of research on teaching. pp. 315-327. Macmillan, New York (1986)
34. Pressley, M.: More about the development of self-regulation: Complex, long-term, and
thoroughly social. Educational Psychologist 30, 207212 (1995)
35. Hetzner, S., Steiner, C., Dimitrova, V., Brna, P., Conlan, O.: Adult Self-regulated Learning
through Linking Experience in Simulated and Real World: A Holistic Approach. In:
Kloos, C.D., Gillet, D., Garcia, R.M.C., Wild, F., and Wolpers, M. (eds.) Towards Ubiqui-
tous Learning 6th European Conference on Technology Enhanced Learning, EC-TEL,
2011. pp. 166-180. Springer, Palermo (2011).
36. Vollmeyer, R., Rheinberg, F.: Motivational effects on self-regulated learning with different
tasks. Educational Psychology Review 18, 239-253 (2006)
... The EU-funded ImREAL project 1 is researching how to augment existing simulators to increase their meaningfulness and their relevance by real-world activity modelling, enriched user modelling, and affective metacognitive scaffolding. This paper is a follow-up to earlier work on developing, integrating, and testing a metacognitive scaffolding service (MSS) in a medical interview training simulator [7],[8]. After the initial evaluation of the training simulator (section 2) with integrated ImREAL services showed positive effects on learning motivation and perceived performance, the simulator augmentation has been further expanded: the metacognitive scaffolding service has been refined and also extended by incorporating affective aspects and the user modelling has been enriched by exploiting social digital traces as a basis for adaptation (section 2.1). ...
... @BULLET Relating to the simulator augmentation: Are the services well integrated in the simulator and learning experience? [8] how scaffolding is an important part of the educational process, supporting learners in knowledge acquisition and learning skill development. As with the previously reported trial, metacognitive scaffolding was provided using calls to a RESTful service developed as part of the ImREAL project. ...
... Thus, sample sizes vary throughout this section. Where applicable results of this user trial (UT2) are compared to those of the prior user trial (UT1; 143 students using the simulator with MSS only in 2011/2012) reported in [8]. Comparisons are therefore based on independent samples, however participants in both studies were medical students from Trinity College, Dublin, with no significant differences regarding their sex or age. ...
Conference Paper
The ImREAL project is researching how to meaningfully augment and extend existing experiential training simulators. The services developed support self-regulated, goal-, and application-oriented learning in adult training. We present results from a study evaluating a medical interview training simulator that has been augmented by an affective metacognitive scaffolding service and by user modelling exploiting social digital traces. Data from 152 medical students participating in this user trial were compared to the results of a prior trial on an earlier technology version. Findings show that students perceived the learning simulator positively and that the enhanced simulator led to increased feelings of success, less frustration, higher technical flow, and more reflection on learning. Interestingly, this cohort of users proved reluctant to provide their open social IDs to enrich their user models.
... At this point spare cognitive resources can be directed towards engaging with within-action prompts. Hence, the learning afforded by after-action reflection acts as a precursor and enabler to the development of self-regulatory skill (Berthold et al., 2012;Sitzmann and Ely, 2010). ...
... (continued ) (Berthold et al., 2012;Vogel-Walcutt et al., 2009). There are also researchers advocating the use of scaffolding prompts within cognitive tutoring systems (e.g. ...
Purpose – The purpose of this paper is to develop a set of generic prompting principles and a framework of prompts that have the potential to foster learning and skill acquisition among adult novices when performing complex, ill-structured problems. Design/methodology/approach – Relevant research in the literatures surrounding problem structure, sensemaking, expertise, metacognition, scaffolding, and cognitive load were reviewed and synthesised in order to derive generic prompting principles and guidelines for their implementation. Findings – A framework of generic principles and prompts is proposed. Differentiation between prompts supporting cognition either within, or after an ill-structured problem-solving task was supported. Practical implications – Prompts such as those proposed in the framework developed presently can be designed into technology-enhanced learning environments in order to structure and guide the cognitive processes of novices. In addition, prompts can be combined with other learning support technologies (e.g. research diaries, collaborative discourse) in order to support learning. Empirical testing will be required to quantify the potential benefits (and limitations of) the proposed prompting framework. Originality/value – The prompts developed constitute a framework for structuring and guiding learning efforts in domains where explicit, actionable feedback is often unavailable. The proposed framework offers a method of tailoring the scaffolding of prompts in order to support differing levels of problem structure and may serve as the basis for establishing an internalised and adaptive learning approach that can be transferred to new problems or contexts.
... Another study (Nussbaumer et al., 2014b) employed the Questionnaire for Self-Regulated Learning (QSRL) developed by Fill Giordano et al. (2010). The original QSRL consists of 54 items grouped into six main constructs: memorizing, elaboration, organization, planning, self-monitoring, and time management (Berthold et al., 2012). In Nussbaumer et al. (2014b) study, participants answered only 18 questions; which constructs these questions belonged to were not elaborated. ...
Self-regulated learning (SRL) plays a significant role in promoting academic success in online education. In recent years, attention has focused on using new techniques to promote SRL—one of which is the recommender system. However, there has been little discussion of the actual effects of using recommender systems to facilitate SRL skills among online learners. This paper aims to elucidate the role that recommender systems play in assisting learners to gain self-regulation skills. The main topics addressed in this paper are as follows: (1) SRL strategies that are supported by recommender systems, as well as the techniques used by these recommenders to promote SRL strategies; and (2) evaluations conducted on the use of recommender systems and results. Our analysis of 20 empirical articles indicates that various features of recommender systems were designed to promote SRL strategies in different phases, and students were generally positive about using such systems to help them self-regulate. Five key knowledge gaps related to existing research on SRL recommender systems were identified. The conclusions suggest that future studies could be improved by demonstrating a more comprehensive understanding of the design of recommender systems, as well as by placing more emphasis on the evaluation process.
... Estudios recientes han demostrado que las nuevas tecnologías pueden aumentar la motivación del alumnado (Berthold et al., 2012;Schmitz, Klemke y Specht, 2012;Ku, Chu y Tseng, 2013) y el aprendizaje colaborativo asíncrono ha sido relacionado con un mejor proceso de construcción del conocimiento a nivel colectivo (Schellens y Valcke, 2005;Ling Koh, Herring y Hew, 2010). En general, las nuevas tecnologías permiten una mayor implicación activa del alumno (Escontrela Mao y Stojanovic Casas, 2004;Sánchez Ilabaca, 2004), que se convierte en sujeto capaz de construir su propio conocimiento a través de su interacción con el ambiente, fundamento de la pedagogía constructivista (Coll, 1990;Brooks y Brooks, 1999). ...
Full-text available
The present paper describes a series of learning activities based on the interactive digital board and the Internet, which were carried out in a secondary school with a bilingual program for newcomers in Texas by a Science and Biology teacher. The low English level of the students could impede the transmission of contents and delay language learning. New technology may help students in two ways: first, it provides visual aids that compensate for language difficulties; second, it offers active and playful learning, which strongly motivates them. Additionally, these activities stimulate different learning styles and let students actively participate in the lessons, despite their language troubles. In the description of the activities, detailed directions are provided, which may be useful to teachers working in similar contexts, such as immersion or content and language integrated learning (CLIL) programs.
... Estudios recientes han demostrado que las nuevas tecnologías pueden aumentar la motivación del alumnado (Berthold et al., 2012;Schmitz, Klemke y Specht, 2012;Ku, Chu y Tseng, 2013) y el aprendizaje colaborativo asíncrono ha sido relacionado con un mejor proceso de construcción del conocimiento a nivel colectivo (Schellens y Valcke, 2005;Ling Koh, Herring y Hew, 2010). En general, las nuevas tecnologías permiten una mayor implicación activa del alumno (Escontrela Mao y Stojanovic Casas, 2004;Sánchez Ilabaca, 2004), que se convierte en sujeto capaz de construir su propio conocimiento a través de su interacción con el ambiente, fundamento de la pedagogía constructivista (Coll, 1990;Brooks y Brooks, 1999). ...
... We report on the service integration into a medical interview training simulator, show how the simulator and augmentation have grown and become more mature over several development phases, and present a comprehensive study covering three empirical evaluations at different stages of this maturation process. Whereas in previous papers we focused on individual evaluations conducted during and right after the training in the simulator (Berthold et al., 2012;Wesiak et al., 2013), the present work gives a long-term perspective on the successive maturation of the service and simulator extension, and also reports some tentative results obtained after real-world experiences. The next two sections give a short overview on related work from the fields of SRL (Section 1.1) and scaffolding in TEL (Section 1.2). ...
Experiential training simulators are gaining increasing popularity for job-related training due to their potential to engage and motivate adult learners. They are designed to provide learning experiences that are directly connected to users’ work environments and support self-regulated learning. Nevertheless, learners often fail to transfer the knowledge gained in the simulated environment to real-world contexts. The EU-funded ImREAL project aimed to bridge that gap by developing a suite of intelligent services designed to enhance existing training simulators. This paper presents work that was a subset of this research project, reporting the iterative development and evaluation of a scaffolding service, which was integrated into a simulator for training medical students to perform diagnostic interviews. The study comprises three evaluation phases, comparing the pure simulator to a first version with metacognitive scaffolding and then to a final version with affective metacognitive scaffolding and enriched user modelling. The scaffolding service provides the learner with metacognitive prompts; affective elements are realized by an integrated affect reporting tool and affective prompts. Using a mixed-method approach by analysing questionnaires (N = 106) and log-data (N = 426), the effects of the services were investigated with respect to real-world relevance, self-regulated learning support, learning experience, and integration. Despite some limitations, the outcomes of this study demonstrate the usefulness of affective metacognitive scaffolding in the context of experiential training simulators; significant post-simulation increases in perceived relevance of the simulator, reflective note-taking, overall motivation, and feeling of success could be identified. Perceived usability and flow of the simulation increased, whereas overall workload and frustration decreased. However, low response rates to specific functions of the simulation point to a need to further investigate how to raise users’ awareness and understanding of the provided tools, to encourage interaction with the services, and to better convey the benefits of using them. Thus, future challenges concern not so much technological developments for personalizing learning experiences, but rather new ways to change user attitudes towards an open approach to learning systems that enables them to benefit from all offered features.
... " . It has previously been shown that providing this scaffolding within the ETU platform is beneficial [10]. Alongside the scaffolding thinking prompt is an open text box for collecting reflection notes which is consistently prefaced with a short text: " Reflect now on your learning: Was this last part of the simulation useful for you? ...
Conference Paper
Full-text available
Self-regulated learning (SRL) and metacognition are key in the context of 21 st century education, adult training, and lifelong learning. For instructional strategies to foster metacognition and self-regulation it is crucial to know what are good metacognitive and SRL behaviors. We investigated this question in the context of a training simulator in a curriculum setting with 152 medical students. Learning behavior and personal attributes were examined in relation to metacognitive awareness. The results on characteristics of successful SRL confirm findings from traditional learning settings for a TEL context.
Full-text available
Catherine Bruen,1 Clarence Kreiter,2 Vincent Wade,3 Teresa Pawlikowska1 1Health Professions Education Centre, Faculty of Medicine and Health Sciences, Royal College of Surgeons in Ireland, Dublin, Ireland; 2Department of Family Medicine, Carver College of Medicine, University of Iowa, Iowa City, Iowa, USA; 3School of Computer Science and Statistics, Faculty of Engineering, Mathematics and Science, Trinity College Dublin, Dublin, Ireland Abstract: Experience with simulated patients supports undergraduate learning of medical consultation skills. Adaptive simulations are being introduced into this environment. The authors investigate whether it can underpin valid and reliable assessment by conducting a generalizability analysis using IT data analytics from the interaction of medical students (in psychiatry) with adaptive simulations to explore the feasibility of adaptive simulations for supporting automated learning and assessment. The generalizability (G) study was focused on two clinically relevant variables: clinical decision points and communication skills. While the G study on the communication skills score yielded low levels of true score variance, the results produced by the decision points, indicating clinical decision-making and confirming user knowledge of the process of the Calgary–Cambridge model of consultation, produced reliability levels similar to what might be expected with rater-based scoring. The findings indicate that adaptive simulations have potential as a teaching and assessment tool for medical consultations. Keywords: medical education, simulation technology, competency assessment, generalizability theory
Full-text available
Am Beispiel von fünf Forschungsperspektiven der Motivationspsychologie wird gezeigt, daß neue Theorien jeweils durch kritische Auseinandersetzung mit vorausgegangenen Forschungsansätzen entstanden sind. Dabei wurde häufig die Forschungsperspektive gewechselt, so daß die Ergebnisse der Forschung aus den verschiedenen Perspektiven nicht im Widerspruch zueinander stehen, sondern sich gegenseitig ergänzen. Das wird anhand eines Rahmenmodells zur Differenzierung und Strukturierung pädagogisch relevanter Aspekte der Lernmotivation näher erläutert. Auf Konsequenzen, die sich für die Rezeption psychologischer Forschung in der Pädagogik ergeben, wird hingewiesen.
Conference Paper
Can high school students gain a deep conceptual understanding of conceptually challenging topics such as the circulatory system by using hypermedia environments? Recent cognitive research has shown that providing students with flexible access and a high degree of student control in non-linear, random access hypermedia environments rarely leads to deep conceptual understanding of complex systems (e.g., Azevedo, Cromley, & Seibert, in press). Learning in a hypermedia environment requires a learner to regulate his or her learning; that is, to make decisions about what to learn, how to learn it, how much time to spend on it, how to access other instructional materials, and to determine whether he or she understands the material. Specifically, students need to analyze the learning situation, set meaningful learning goals, determine which strategies to use, assess whether the strategies are effective in meeting the learning goal, evaluate their emerging understanding of the topic, and determine whether the learning strategy is effective for a given learning goal. They need to monitor their understanding and modify their plans, goals, strategies, and effort in relation to contextual conditions (e.g., cognitive, motivational, and task conditions). Further, depending on the learning task, they need to reflect on the learning episode.
We constructed a 52-item inventory to measure adults′ metacognitive awareness. Items were classified into eight subcomponents subsumed under two broader categories, knowledge of cognition and regulation of cognition. Two experiments supported the two-factor model. Factors were reliable (i.e., α = .90) and inter-correlated (r = .54). Experiment 2 reported the knowledge of cognition factor was related to pre-test judgments of monitoring ability and performance on a reading comprehension test, but was unrelated to monitoring accuracy. Implications for educational assessment and future research were discussed.
In this study the assumption was tested experimentally, whether prompting for reflection will enhance hypermedia learning and transfer. Students of the experimental group were prompted at each navigation step in a hypermedia system to say the reasons why they chose this specific information node out loud whereas the students of the control group learned without reflection prompting. University students (N = 46) participated and were counter-balanced according to their prior knowledge, metacognitive knowledge, and verbal intelligence. The students' task was to learn the concepts of operant conditioning within 35 minutes. Students were completely free in navigation and students' individual learning sessions were videotaped. Immediately afterwards learning outcome was obtained. As expected, students learning with reflection prompts showed better transfer performance compared to the control group, especially if their navigation was based on strategic decisions. Even though prompting for reflection was successful, ways of further optimizing this kind of metacognitive intervention are discussed.
Self-regulated learning is more complex than Winne argued, as illustrated by the complex strategies articulated by very skilled readers as they process text. So, too, is transfer of newly learned strategies from one task to another, although paradoxically Winne underestimated the evidence supporting some of the mechanisms of transfer he favored. The development of self-regulation is long-term and much more social than Winne implied. Nonetheless, I think the instructional principles Winne favored are so important that I offer my own summary of them.