adfa, p. 1, 2011.
© Springer-Verlag Berlin Heidelberg 2011
An Initial Evaluation of Metacognitive Scaffolding for
Experiential Training Simulators
Marcel Berthold1, Adam Moore2, Christina Steiner1, Conor Gaffney3, Declan Dag-
ger3, Dietrich Albert1, Fionn Kelly4, Gary Donohoe4, Gordon Power3, Owen Conlan2
1Knowledge Management Institute, Graz University of Technology,
Rechbauerstr. 12, 8010 Graz, Austria
2Knowledge, Data & Engineering Group, School of Computer Science and Statistics,
Trinity College, Dublin, Ireland
3EmpowerTheUser, Trinity Technology & Enterprise Campus,
The Tower, Pearse Street, Dublin, Ireland
4Department of Psychiatry, School of Medicine, Trinity College, Dublin, Ireland
Abstract. This paper elaborates on the evaluation of a Metacognitive Scaffold-
ing Service (MSS), which has been integrated into an already existing and ma-
ture medical training simulator. The MSS is envisioned to facilitate self-
regulated learning (SRL) through thinking prompts and appropriate learning
hints enhancing the use of metacognitive strategies. The MSS is developed in
the European ImREAL (Immersive Reflective Experience-based Adaptive
Learning) project that aims to augment simulated learning environments
throughout services that are decoupled from the simulation itself. Results com-
paring a baseline evaluation of the ‘pure’ simulator (N=131) and a first user
trial including the MSS (N=143) are presented. The findings indicate a positive
effect on learning motivation and perceived performance with consistently good
usability. The MSS and simulator are perceived as an entity by medical students
involved in the study. Further steps of development are discussed and outlined.
Keywords: self-regulated learning, metacognitive scaffolding, training simula-
Self-regulated learning (SRL) and especially metacognition, is currently a prominent
topic in technology-enhanced learning (TEL) research. Many studies provide evi-
dence of the effectiveness of SRL in combination with metacognitive scaffolding (cf.
[1, 2]). Self-regulated learning refers to learning experiences that are directed by the
learner and describes the ways in which individuals regulate their own cognitive and
metacognitive processes in educational settings (e.g. [3, 4]). An important aspect of
self-regulated learning is therefore the learners’ use of different cognitive and meta-
cognitive strategies, in order to control and direct their learning . These strategies
include cognitive learning strategies, self-regulatory strategies to control cognition
(i.e. metacognitive strategies) and resource management strategies. Self-regulated
learning also involves motivational processes and motivational beliefs . It has been
shown that good self-regulated learners perform better and are more motivated to
learn  than weak self-regulated learners. TEL environments provide opportunities
to support and facilitate metacognitive skills, but most learners need additional help
and guidance  to perform well in such environments.
In the EU project, ImREAL1 (Immersive Reflective Experience-based Adaptive
Learning), intelligent services are being developed to augment and improve experien-
tial simulated learning environments – including one to scaffold metacognitive proc-
esses. The development of the scaffolding service focuses on the salient and timely
support of learners in their metacognitive processes and self-regulated learning in the
context of a simulation environment. Herein we report a concrete study examining the
medical training simulator provided by EmpowerTheUser2 augmented with the Im-
REAL Metacognitive Scaffolding Service (MSS). The service will provide prompts
and suggestions adapted to a learner’s needs and traits of metacognition and aiming at
enhancing motivation towards the learning activity in the simulation. While the aspect
of supporting metacognition needs to be integrated in the learning process, the accord-
ing service will be technically decoupled from the specific learning system itself.
Overall, the research presented investigates the effectiveness and appropriateness of
the service and the scaffolding it provides. To allow a more detailed examination of
the issues, we address four sub questions:
1. Is self-regulated learning supported? For the evaluation and analysis of self-
regulated learning we distinguish between the general learning approach (i.e. ap-
plication of cognitive, metacognitive strategies), and the metacognitive and specific
learning processes in the simulation (i.e. cognitive, metacognitive strategies or ac-
tions within simulator context); thereby, learning and metacognitive scaffolding in
the simulation may optimally, and on a long-term basis, influence the general
learning approach of a learner.
2. Does the simulator augmentation through the service lead to better learning !
performance?" The learning performance refers to the (objective or subjec-
tive/perceived) learners’ knowledge/competence acquisition and performance in
the learning situation and to the transfer of acquired knowledge to other situations.
3. Does the simulator augmentation through the service increase motivation? The
aspect of motivation addresses the motivation to learn, i.e. the structures and proc-
esses explaining learning actions and the effects of learning .
4. Is the service well integrated in the simulation and learning experience? This
refers to the question whether the scaffolding interventions provided during the
simulation via the MSS are perceived by learners as appropriate and useful – in
terms of their content, context and timing.
In order to answer these evaluation questions the paper is organized in the follow-
ing structure. Section 2 gives an overview of the simulator and normal usage. Section
3 gives an overview of the MSS service and outlines related work. In section 4 the
experimental design of the study is introduced and section 5 includes the according
results. These results are discussed in section 5. Section 6 provides a conclusion and
an outlook to further research.
2 Metacognitive Scaffolding – background and technology
Scaffolding is an important part of the educational process, supporting learners in
their acquisition of knowledge and developing their learning skills. Scaffolding has
been a major topic of research since the pioneering work of Vygotsky (e.g. 1978 )
and the key work of Bruner and Wood and colleagues (cf. ).
Work on the use of scaffolding with the help of computer-based learning environ-
ments has been extensive (cf. ). Originally, the emphasis was on cognitive scaf-
folding which has many forms (cf. ). In the last ten years there has been a move
towards research in metacognitive scaffolding (e.g. [13–16]) as well as in the use of
metacognitive scaffolding in adaptive learning environments (e.g. [17–20]).
Other forms of scaffolding have also been explored both in educational and tech-
nology enhanced learning contexts – such as affective scaffolding and conative scaf-
folding. Van de Pol et al.  sought to develop a framework for the analysis of dif-
ferent forms of scaffolding. In the technology enhanced learning community, Poray-
ska-Pomsta and Pain  explored affective and cognitive scaffolding through a form
of face theory (the affective scaffolding also included an element of motivational
scaffolding). Aist et al.  examined the notion of emotional scaffolding and found
different kinds of emotional scaffolding had an effect on children's persistence using a
reading tutoring system.
There are different forms of metacognitive scaffolding. Molenaar et al.  investi-
gated the distinction between structuring and problematizing forms of metacognitive
scaffolding and found that problematizing scaffolding seemed to have a significant
effect on learning the required content. They used Orientation, Planning, Monitoring,
Evaluation and Reflection as subcategories of metacognitive scaffolding.
Sharma and Hannafin  reviewed the area of scaffolding in terms of the implica-
tions for technology enhanced learning systems. They point out the need to balance
metacognitive and “procedural” scaffolds since only receiving one kind can lead to
difficulties – with only procedural scaffolding students take a piecemeal approach,
and with only metacognitive scaffolding students tend to fail to complete their work.
They also argue for systems that are sensitive to the needs of individuals. Boyer et al.
 examined the balance between motivational and cognitive scaffolding through
tutorial dialogue and found evidence that cognitive scaffolding supported learning
gains while motivational scaffolding supported increase in self-efficacy.
The aim of the ImREAL project is to bring simulators closer to the ‘real world’. As
part of training for a diagnostic interview, in the ‘Real World’ a mentor sits at back
observing and providing occasional input / interventions as necessary. The MSS has
been developed to integrate into the simulator learning experience as an analogue of a
mentor, sitting alongside the simulator to provide scaffolding. The ETU simulator
supports meta-comprehension and open reflection via note taking.
For this trial metacognitive scaffolding was provided using calls to a RESTful 
service developed as part of the ImREAL project. The service utilises technology
initially developed for the ETTHOS model  and presents Items from the Meta-
cognitive Awareness Inventory  according to an underlying cognitive activity
model, matched to Factors in the MAI. In this way the importance of the tasks being
undertaken by the learner is clear scaffolding is developed in order to match a learn-
ers’ cognitive activity to metacognitive support.
The scaffolding service supplements the pre-existing ETU note-taking tool, both of
which are illustrated in Figure 1 below. The text of the thinking prompt item is
phrased in order to elicit a yes/no response. If additional context / rephrasing has been
added by the instructional experts, that is displayed before the open text response
area. A link that activates an explanatory text occurs underneath the text input area, as
well as a “Like button” which can be selected and the submit action.
Fig. 1. a) MSS Interface b) ETU Note-taking tool
3 Overview of Simulator and Normal Usage
For this research the ETU Talent Development Platform was used, with training for
medical interview situations. The user plays the role of a clinical therapist and selects
interview questions from a variety of possible options to ask the patient. When a ques-
tion is selected a video is presented that shows the verbal interaction of the therapist
with the patient (close up of the patient, voice of the therapist) and the verbal and non-
verbal reaction of the patient (close up of the patient). Starting the simulation, users
can choose between two types of scenarios (Depression and Mania), which offer the
same types of subcategories: Introduction and negotiating the agenda, eliciting in-
formation, outlining a management plan and closing the interview.
After a scenario is chosen, the user may simulate the interview as long as they pre-
fer or until the interview is “naturally” finished. Furthermore, the users could have as
many runs of the simulation as they want and could choose a different scenario in the
following attempts. In this study we focused only on the Depression interview sce-
nario. A screenshot of a typical interaction within the ETU system is show below in
Fig. 2. Screen shot of the EmpowerTheUser Simulator. The scenario of diagnosing a pa-
tient with clinical depression is just beginning!
4 Experimental design
143 medical students participated in the study and performed the simulation as part of
their second year (2011/2012) medical training at Trinity College, Dublin (TCD).
They were on average approximately 22 years old (40% male vs. 60% female, 80%
Irish). In addition, these results are compared, as far as they have been assessed at
both time points, to a baseline evaluation based on using the simulator without Im-
REAL services. In the baseline evaluation, 131 TCD medical students from the previ-
ous year group (2010/2011) participated (cross-section design).
4.2 Measurement Instrument
ETU simulator. Within the simulation learning performance is assessed by tracking
scores for each of the 4 subsections, as well as dialogue scores and notes that were
written in an available note pad for reflections.
Questionnaire on Self-Regulated Learning. Self-regulated learning skills were
measured by the Questionnaire for Self-Regulated Learning (QSRL; ). The QSRL
consists of 54 items, which belong to six main scales (Memorizing / Elaboration /
Organization / Planning / Self-monitoring / Time management) and three subscales
(Achievement Motivation / Internal attribution / Effort). In the online version of the
questionnaire, respondents indicate their agreement to an item by moving a slider on
an analogue scale between the positions “strongly disagree” to “strongly agree”. The
possible score range is 0-100 in each case, with higher values indicating a better re-
Survey questions on use of ETU, experience in performing clinical interviews
and relevance of the ETU simulator. In order to control possible influences of prior
experience with the respective simulated learning environment or real world medical
interviews, their experiences were assessed through three survey questions. The fol-
lowing survey question assessed the relevance of the simulator with answer options
ranging from “not relevant at all” to “very relevant”.
Thinking Prompts. Triggers that made calls to the MSS were inserted into the prac-
tice phase of the simulator but not available during ‘live/scored usage’. The triggers
were created using the ETU authoring platform and made a call to the MSS request-
ing a prompt of a particular Factor (Planning, Information Management, Comprehen-
sion, Debugging or Evaluation). As explained above, each Factor consisted of a num-
ber of Items or Thinking Prompts. An item was not redisplayed once a reflection had
been entered with it.
Motivation. Motivation was assessed with four survey questions referring to learning
more about clinical interviews, improve own interview skills, performing a good in-
terview during the simulation and applying what has been learned in a real interview.
Workload. Measures of workload were assessed by six subscales of the NASA-TX
[29, 30] with a score range of 0-100. In this case higher values indicate a higher work-
load. An overall workload score was calculated based on the subscales by computing
a mean of all item contributions. In contrast to the original NASA-TLX, students did
not mark their answers to an analogue scale, but entered digits between 0-100 into a
Usability and Service specific integration. The Short Usability Scale (SUS, )
consists of ten items with answer options of a five-point rating-scale ranging from
“strongly disagree” to “strongly agree”. The raw data were computed to an overall
SUS score. The overall SUS score ranged from 0-100 with higher values indicating
higher usability. Additionally to the SUS questions, three service specific usability
questions were administered regarding the relation of the prompts to the rest of the
simulation and obvious differences. The answer options were the same as for the
Learning Experience with MSS. Learning experience with MSS was measured by
10 questions referring to helpfulness and appropriateness of the MSS thinking
prompts within the simulator with answer options on a 5-point-Likert-scale ranging
from “not at all” to “very much”. In addition, a free text comment field was provided.
Procedure. The baseline evaluation, using the pure simulator, was conducted in mid-
February and beginning of March of 2011; the first user trial was carried out in Dub-
lin from mid-February until beginning of March of 2012. The TCD medical students
used the ETU medical training simulator.
Data collection was carried out during the simulation (e.g. ETU scores, MSS data)
and after learning with the ETU simulator (questionnaire data). At first the students
worked on the simulation for as long as they wanted and could choose between two
scenarios; Mania or Depression. After they were finished they were directed to the
online questionnaires. In this stage they filled in the survey questions on relevance
and on motivation, NASA-TLX, SUS, questions on prompts, learning experience and
After working on the simulation in the TCD course students still had access to the
ETU simulator via the internet for approximately two weeks. It was not mandatory to
use the simulation in the medical course at TCD or to participate in the evaluation.
5 Experimental results
PAWS Statistics, version 18.0  and Microsoft Excel (2010) were used for statisti-
cal analyses and graphical presentations. If not explicitly mentioned, statistical re-
quirements for inference statistical analyses and procedures were fulfilled. For all
analyses the alpha level was α=.05. Due to an unbalanced number of participants in
the samples in regard to comparisons of the first user trial and baseline evaluation
appropriate pre-tests have been performed and the according values are presented.
This section focuses mainly on the first user trial evaluation based on using the
ETU simulator with the integrated MSS ImREAL services.
ETU Simulator – Descriptive data. All students of the first user trial reported that
they have never used the ETU medical training simulator before. Nonetheless, they
were quite experienced in conducting clinical interviews, since 97% reported to have
already performed at least one, but only 15 % had experienced interviewing a psychi-
A comparison of the first user trial and the baseline evaluation showed that dura-
tion time in minutes (Mbase=17.89, SDbase=11.15; M1UT=15.45, SD1UT =6.81) and
scored points (Mbase=31.34, SDbase=6.33; M1UT=27.61, SD1UT =5.91) in the simulation
decreased from baseline evaluation to the first user trial (duration time: t211,49=2.17,
p=.031; score: t272=5.10, p=.000). These results show that students spent on average
less time in the simulator and reached lower scores. This cannot be explained by more
experience of the baseline cohort, because they also had not used the simulator before.
Metacognitive Scaffolding Service (MSS) comments. 10 comments have been col-
lected by MSS learning experience questionnaire free text comment field. The par-
ticipants provided interesting comments, which however referred more to the simula-
tor than to the MSS. This implies that the MSS seems to be perceived as well inte-
grated in the simulation, because students do not seem to differentiate between the
additional service and the simulation itself. The participants pointed to sometimes
inappropriate prompts in combination with the simulator in situations, especially,
when only one answering option was available in the dialogue with the patient and
they were asked to think about their strategy. Nevertheless, one learner recorded that
“I am learning a lot actually, it is amazing how much you can miss just by asking a
question in a slightly different way! I keep going back a step and looked through the
other options to see where the scenario goes. Usually I’ve picked the most suitable
one, but not always. Sometimes I am surprised about how much I would have
missed!!” which overall demonstrates good engagement and a positive view of the
Prompts analysis. There have been five different types of prompts presented accord-
ing to the five MAI phases described in section 3. In total 2001 prompts (Planning:
469, Information Management: 752, Monitoring: 425, Debugging: 301 and Reflec-
tion: 54) were provided to 50 students. Every student who used the practice facility in
the simulator was presented with a pop-up suggesting they reflected. Clicking on that
pop-up would move the simulation to the MSS screen (Figure 2a). The relative fre-
quency of the prompts was compared to the expected frequency based on the prob-
ability of available prompts for each phase. The results indicate that the learners were
scaffolded more often in the second phase “Information Management” and were less
scaffolded in the reflection phase as could have been expected (
p=.000, Figure 3). On the one hand, learners seem to need more assistance in effec-
tively processing information by hints to use more organizational, elaborative, sum-
marizing or selective learning strategies. On the other hand they are rather confident
in the reflection phase and wave the offer of scaffolds.
Fig. 3. Comparison of the expected and empirical distribution of metacognitive scaffolds for
the five phases of Schraw’s Metacognitive Awareness Inventory!
5.2 Questionnaire Data
QSRL. The quantitative results of the MSS service are a little surprising, because
students estimated the use of cognitive learning strategies, especially elaboration
strategies, relatively high. In general, all SRL scores are located above the center
point of the score range and indicate positive results for all cognitive and metacogni-
tive strategies. However, a stronger use of elaboration strategies is reported (t20 =
3.34, p=.003) in the first user trial. It needs to be explicitly stated that this is not an
unfavorable result as such as elaboration strategies are strategies of deeper learning
, which should be further supported by scaffolding services.
A comparison to the baseline study shows no significant increase in any of the us-
age of reported learning strategies.
Motivation. 38 participants filled in the motivation questions. The results show that
the scores were on a high motivation level around 3.16-3.49 on a 4-point-Likert rating
scale. This implies that the students were very motivated to learn about the clinical
interview during the simulation, to improve their interview skills, perform a good
interview during the simulation and to apply what they have just learnt in the simula-
tion in a real world clinical interview context. Furthermore, a comparison of the over-
all motivation scores assessed immediately after the simulation of the first user trail
and immediately after the baseline evaluation reveals significant higher motivation
scores for the MSS trial (M=3.35, SD=.4.14) compared to the baseline (M=2.48,
SD=.73; t118.47=-8.64, p=.000).
Workload. A moderate overall workload could be observed. It has to be noted at this
stage that for a learning environment it should not be aimed at reducing the workload
to a minimum; rather, the challenge should be at an appropriate, medium level of
challenge – in an optimal case adapted to the individual learner. Participants reported
the highest, but still moderately pronounced, load for effort. This subscale refers to
the mental and physical resources that had to be mobilized to accomplish the task.
Consequently, the result for effort can be relegated rather to mental than physical
demand. Yet the simulation is a complex program that supports and requires active
learning processes; a reduction of mental demand is somewhat challenging, but could
possibly be realized by improving the MSS and addressing the challenge to reduce
repetitions and provide only appropriate scaffolds. Furthermore, the second highest
score was observed for performance (see Figure 4), showing that the students felt they
successfully accomplished what they were supposed to do. Performance scores (refer-
ring to subjective/perceived learning outcome) even increased for the first user trial
(M=54.83, SD=17.90) compared to the baseline evaluation (M=43.00, SD=23.68)
significantly (t109=-2.63, p=.01). A t-test for independent groups remains insignificant
comparing overall workload scores for the first user trial (M=44.19, SD=10.86) and
baseline evaluation (M=44.81, SD=12.011; t75.52=.27, ns.).
Fig. 4. Comparison of Baseline and first user trial data for workload
Usability and Service specific integration. No differences could be observed be-
tween the rather high usability overall scores for the first user trial (M=62.50,
SD=17.90) and the baseline evaluation (M=62.80, SD=16.08; F57.90=.90, ns.).
With respect to service specific integration with the ETU medical training and MSS
prompts the majority of the students ratings were positive with 21 out of 33 stating
they felt supported during their learning process by the MSS service and that the serv-
ice was well integrated in the system (M=3.26, SD=.40).
Learning Experience with MSS. The learning experience with the MSS was rela-
tively positive. More than 63% of the participants perceived the MSS learning experi-
ence overall as very much helpful and appropriate. The high score for the individual
items were all above the center point of the scale, which underlies this encouraging
In this paper we examined the effectiveness and appropriateness of the MSS. Results
of the first user trial have been reported, involving the ETU medical training simula-
tor augmented with the MSS service. These results have been compared to a baseline
evaluation where the ‘pure’ simulator was administered without any additional Im-
REAL services. Addressing the evaluation questions stated in the introduction sec-
Is self-regulated learning supported? Even though self-regulated learning and
metacognitive scaffolding are closely connected, because the SRL process heavily
relies on applying cognitive and especially metacognitive learning strategies and
techniques, no changes in SRL profile could be observed comparing the first user trial
to the baseline data. This is because influencing self-regulated learning aspects is
rather a long-term process . This result might also be explained by having a look
at the usage frequency of the simulator. The students were confronted with the simu-
lation only in the TCD course and no one had used the simulation before. Further-
more, duration time of working with the simulator, which was on average less than
half an hour, might not be too short to change a rather stable learning approach.
For future studies the application of a longitudinal approach could be suggested in-
stead of a cross-section evaluation, to better meet the requirements of a longer-term
process. In addition, teachers’ or supervisors’ judgments on SRL performance could
be included to assess their observations on potential changes in learners’ daily learn-
ing behavior. However, the last point might be difficult to realize in an university
setting with more than 140 students in a course.
In general, all SRL self-reports were positive, indicating a higher use of elaboration
strategies compared to memorizing strategies. Elaboration strategies represent strate-
gies of deeper learning . Nevertheless, fostering memorizing/rehearsal strategies
might be taken up as an idea for improving the MSS. Assuming that the participants
in the evaluation trials constitute a representative sample of ETU simulator users,
ImREAL could start from this result and aim at improving users’ rehearsal strategies
through the provision of appropriate scaffolding. Of course, this strategy type should
not be the only one to be supported. Rehearsal strategies help the learner to select and
remember important information, but may not represent very deep levels of cognitive
processing . As a result, ImREAL services should especially try to further support
elaboration as well as organizational strategies. In the ImREAL pedagogical frame-
work learning  is seen as a cyclic process of three phases: forethought, learning
and reflection. These individual phases are already represented in the ETU system,
but not covered comprehensively. As described above, medical students do not tend
to use the ETU simulator very often and if they do they undertake the interview sce-
nario only for a short period of time. Therefore, reflection and coverage of the SRL
phases should be further extended and supported by the ImREAL MSS.
Does the simulator augmentation through the service lead to better learning per-
formance? Results concerning the learning performance draw an ambiguous picture.
The actual objective data collected by the ETU simulator demonstrates that overall
scores decreased from the baseline evaluation to the first user trial. Surprisingly, self-
report scores on performance increased.
An increasing self-report score caused by the MSS that encourages learners to
think about one’s own performance and making the aspect of actually performing
well more conscious. This in turn might lead to a) better self-estimations and self-
efficiency, resulting in higher workload performance scores and b) to higher motiva-
tion scores (see evaluation question 3 of this section).
Does the simulator augmentation through the service increase motivation? In line
with the increasing scores on performance assessed by self-report are results on moti-
vation. Motivation scores increased from the baseline evaluation to the first user trial.
In addition to the consideration of motivation as a state characteristic, motivational
beliefs (motivational traits in terms of being more stable and outlasting than state
motivation) can be influenced by positive sounding scaffolds and hints to optimize
learning. If students see the prompts as support of their learning approach a positive
attitude to the whole learning process can be expected and could explain the current
result, because these motivational beliefs are factors influencing the initial motivation
of the learner .
Is the service well integrated in the simulation and learning experience? Results
on usability of the whole system (simulator + MSS service) and service specific inte-
gration provide evidence that the MSS service is well integrated in the simulation and
leads to real augmentation. This is not only demonstrated by the positive scores on the
service specific integration questions, but also by user comments, which were overall
quite positive. Such positive results may be attributed to the MSS operating in an
appropriately timely and salient manner, with the pop-up triggers appearing at appo-
site times created by the instructional design experts. Also the RESTful interface al-
lows calls to be made to an ETU simulator-specific interface for the MSS, ensuring
there are no obvious presentational and interactional differences between the hosting
simulator and the MSS.
7 Conclusions & Outlook
The results above demonstrate a clear advantage in providing a MSS to augment an
experiential training simulator, leading to more engaged, motivated learners without
overly burdening them or interrupting the flow of their learning experience. With
respect to the actual learning performance no positive effect could be identified. This
would be desirable to investigate in more detail in future studies. These further studies
should optimally be realized in a longitudinal evaluation approach, as well as an as-
sessment of real-world performance on medical interviews (i.e. learning transfer).
The collecting and monitoring of the development of motivation throughout both
evaluation runs is important, because in the next version of the ImREAL MSS service
there will be a strong focus on extending it by ‘affective scaffolding’, which will
mainly address motivational aspects. As a result, the data from the first user trial
evaluation (with metacognitive scaffolding ‘only’) will serve as benchmark for a
comparison with evaluation outcomes for the affective metacognitive scaffolding,
thus allowing to investigate the additional benefit of the affective part.
The MSS will be integrated within additional experiential simulators to investigate
the service’s capabilities for generalization and integration within different systems
and usage cases and to further evaluate its effect on learning experience.
Acknowledgement. The research leading to these results has received funding from
the European Community's Seventh Framework Program (FP7/2007-2013) under
grant agreement no 257831 (ImREAL project) and could not be realized without the
close collaboration between all ImREAL partners.
1. Veenman, M.V.J.: Learning to Self-Monitor and Self-Regulate. Regulation. (2008)
2. Molenaar, I., Boxtel, C.A.M. Van, Sleegers, P.J.C.: The effects of scaffolding metacogni-
tive activities in small groups. Computers in Human Behavior 26, 1727-1738 (2010)
3. Puustinen, M., Pulkkinen, L.: Models of Self-regulated Learning: A review. Scandinavian
Journal of Educational Research 45, 269-286 (2001)
4. Zimmerman, B.: Becoming a Self-Regulated Learner: An Overview. Theory Into Practice
41, 64-70 (2002)
5. Pintrich, P.R.: The role of motivation in promoting and sustaining self-regulated learning.
International Journal of Educational Research. 31, 459-470 (1999).
6. Veenman, M.V.J.: Learning to Self-Monitor and Self-Regulate. In: R. Mayer, P. Alexander
(eds.) Handbook of Research on Learning and Instruction. pp. 197-218. Routledge, New
7. Manuela, B.: Effects of Reflection Prompts when Learning with Hypermedia. Journal of
Educational Computing Research 35, 359-375 (2006)
8. Krapp, A.:: Die Psychologie der Lernmotivation. Perspektiven der Forschung und Proble-
me ihrer pädagogischen Rezeption [The psychology of motivation to learn. Perspectives of
research and their pedagogical reception]. Zeitschrift für Pädagogik 39, 187-206 (1993)
9. Vygotskiǐ, L.S., Cole, M.:: Mind in society: the development of higher psychological
processes. Harvard University Press, Cambridge, MA (1978)
10. Wood, D., Bruner, J.S., Ross, G.: The role of tutoring in problem solving. Journal Of Child
Psychology And Psychiatry 17, 89-100 (1976)
11. Lajoie, S.P.: Extending the Scaffolding Metaphor. Instructional Science 33, 541-557
12. Clark, A.: Towards a science of the bio-technological mind. International Journal of Cog-
nition and Technology 1, 21-33 (2002)
13. Pol, J., Volman, M., Beishuizen, J.: Scaffolding in Teacher–Student Interaction: A Decade
of Research. Educational Psychology Review 22, 271-296 (2010)
14. Dinsmore, D.L., Alexander, P.A., Loughlin, S.M.: Focusing the Conceptual Lens on Meta-
cognition, Self-regulation, and Self-regulated Learning. Educational Psychology Review
20, 391-409 (2008)
15. Greene, J., Azevedo, R.: The Measurement of Learners’ Self-Regulated Cognitive and
Metacognitive Processes While Using Computer-Based Learning Environments. Educa-
tional Psychologist 45, 203-209 (2010)
16. Azevedo, R.:: Does adaptive scaffolding facilitate students? ability to regulate their learn-
ing with hypermedia?*1. Contemporary Educational Psychology 29, 344-370 (2004)
17. Azevedo, R., Hadwin, A.F.: Scaffolding Self-regulated Learning and Metacognition – Im-
plications for the Design of Computer-based Scaffolds. Instructional Science 33, 367-379
18. Luckin, R., Hammerton, L.: Getting to Know Me: Helping Learners Understand Their
Own Learning Needs through Metacognitive Scaffolding. In: S. Cerri, G. Gouardères, F.
Paraguaçu (eds.) System. vol. 2363, pp. 759-771. Springer Berlin / Heidelberg, (2002)
19. Roll, I., Aleven, V., McLaren, B.M., Koedinger, K.R.: Designing for metacognition—
applying cognitive tutor principles to the tutoring of help seeking. Metacognition and
Learning 2, 125-140 (2007)
20. Roll, I., Aleven, V., McLaren, B.M., Koedinger, K.R.: Improving students’ help-seeking
skills using metacognitive feedback in an intelligent tutoring system. Learning and Instruc-
tion 21, 267-280 (2011)
21. Porayska-Pomsta, K., Pain, H.: Providing Cognitive and Affective Scaffolding Through
Teaching Strategies: Applying Linguistic Politeness to the Educational Context. Pain
3220, 18-21 (2004)
22. Aist, G. et al.: Experimentally augmenting an intelligent tutoring system with human-
supplied capabilities: adding human-provided emotional scaffolding to an automated read-
ing tutor that listens. Proceedings Fourth IEEE International Conference on Multimodal
Interfaces 483-490 (2002)
23. Sharma, P., Hannafin, M.J.: Scaffolding in technology-enhanced learning environments.
Interactive Learning Environments 15, 27-46 (2007)
24. Boyer, K. et al.: Balancing Cognitive and Motivational Scaffolding in Tutorial Dialogue.
In: Intelligent Tutoring Systems, pp. 239-249. Springer, (2008)
25. Fielding, R.T.: Architectural Styles and the Design of Network-based Software Architec-
tures, PhD-thesis, Citeseer (2000)
26. Macarthur, V., Moore, A., Mulwa, C., Conlan, O.:: Towards a Cognitive Model to Support
Self-Reflection: Emulating Traits and Tasks in Higher Order Schemata. In: EC-TEL 2011
Workshop on Augmenting the Learning Experience with Collaborative Reflection, Pal-
ermo, Sicily, Italy (2011)
27. Schraw, G., Sperling Dennison, R.: Assessing metacognitive awareness. Contemporary
Educational Psychology 19, 460–475 (1994)
28. Fill Giordano, R., Litzenberger, M., Berthold, M.:: On the Assessment of strategies in self-
regulated learning (SRL)–differences in adolescents of different age group and school
type. pp. Poster. 9. Tagung der Österreichischen Gesellschaft für Psychologie, Salzburg,
29. Hart, S.G., Staveland, L.E.: Development of NASA-TLX (Task Load Index): Results of
Empirical and Theoretical Research. In: N.M.P.A. Hancock (eds.) Human mental work-
load. pp. 239-250. North Holland Press, Amsterdam (1988)
30. Hart, S.G.: NASA-Task Load Index (NASA-TLX); 20 Years Later. Human Factors and
Ergonomics Society Annual Meeting Proceedings 50, 904-908 (2006)
31. Brooke, J.: SUS: A “quick and dirty” usability scale. In: P.W. Jordan, B. Thomas, B..
Weerdmeester, A.L. McClelland (eds.) Usability Evaluation in Industry. pp. 189-194. Tay-
lor & Francis, London (1996)
32. Inc, S.: PASW Statistics 18 Core System User ’s Guide. SPSS Inc., (2007)
33. Weinstein, C.E., Mayer, R.E.: The teaching of learning strategies. In: M. Wittrock (eds.)
Handbook of research on teaching. pp. 315-327. Macmillan, New York (1986)
34. Pressley, M.: More about the development of self-regulation: Complex, long-term, and
thoroughly social. Educational Psychologist 30, 207–212 (1995)
35. Hetzner, S., Steiner, C., Dimitrova, V., Brna, P., Conlan, O.: Adult Self-regulated Learning
through Linking Experience in Simulated and Real World: A Holistic Approach. In:
Kloos, C.D., Gillet, D., Garcia, R.M.C., Wild, F., and Wolpers, M. (eds.) Towards Ubiqui-
tous Learning 6th European Conference on Technology Enhanced Learning, EC-TEL,
2011. pp. 166-180. Springer, Palermo (2011).
36. Vollmeyer, R., Rheinberg, F.: Motivational effects on self-regulated learning with different
tasks. Educational Psychology Review 18, 239-253 (2006)