Conference PaperPDF Available

Analytics meet patient manikins: challenges in an authentic small-group healthcare simulation classroom


Abstract and Figures

Healthcare simulations are hands-on learning experiences aimed at allowing students to practice essential skills that they may need when working with real patients in clinical workplaces. Some clinical classrooms are equipped with patient manikins that can respond to actions or that can be programmed to deteriorate over time. Students can perform assessments and interventions, and enhance their critical thinking and communication skills. There is an opportunity to exploit the students' digital traces that these manikins can pervasively capture to make key aspects of the learning process visible. The setting can be augmented with sensors to capture traces of group interaction. These multimodal data can be used to generate visualisations or feedback for students or teachers. This paper reports on an authentic classroom study using analytics to integrate multimodal data of students' interactions with the manikins and their peers in simulation scenarios. We report on the challenges encountered in deploying such analytics 'in the wild', using an analysis framework that considers the social, epistemic and physical dimensions of collocated group activity.
Content may be subject to copyright.
Analytics Meet Patient Manikins: Challenges in an Authentic
Small-Group Healthcare Simulation Classroom
Roberto Martinez-Maldonado
Tamara Power
Carolyn Hayes
Adrian Abdiprano
Tony Vo
Carmen Axisa
Simon Buckingham Shum
Connected Intelligence Centre,
Faculty of Health, University of Technology Sydney, Australia
Healthcare simulations are hands-on learning experiences aimed at
allowing students to practice essential skills that they may need
when working with real patients in clinical workplaces. Some
clinical classrooms are equipped with patient manikins that can
respond to actions or that can be programmed to deteriorate over
time. Students can perform assessments and interventions, and
enhance their critical thinking and communication skills. There is
an opportunity to exploit the students’ digital traces that these
manikins can pervasively capture to make key aspects of the
learning process visible. The setting can be augmented with sensors
to capture traces of group interaction. These multimodal data can
be used to generate visualisations or feedback for students or
teachers. This paper reports on an authentic classroom study using
analytics to integrate multimodal data of students’ interactions with
the manikins and their peers in simulation scenarios. We report on
the challenges encountered in deploying such analytics ‘in the
wild’, using an analysis framework that considers the social,
epistemic and physical dimensions of collocated group activity.
CCS Concepts
• Information systems
Information systems applications
Collaborative and social computing systems and tools
Classroom, groupwork, multimodal, awareness, face-to-face
Healthcare clinical simulations are practical learning experiences
designed to expose students to a comprehensive range of complex
or typical scenarios that they may encounter in their future
workplaces or professional situations [6]. Some didactic methods,
such as Problem-Based Learning [1], have been effectively applied
in healthcare education in the form of tasks where students engage
in complex medical scenarios, as well as simulations using
manikins or actors as patients. Simulation technology is often used
to assist healthcare training, ranging from very simple (e.g. using
an orange to practice giving injections) to more sophisticated
computer-based systems (e.g. fully simulated high fidelity
manikins or even completely digital patients [16]). Cooper and
Taqueti [5] have outlined different types of healthcare simulation
technologies. For example, high fidelity patient manikins are often
used to simulate medical scenarios in an environment which is safe
for both learners and patients.
Simulations in health care are not associated with specific
technologies. Rather they are learning techniques that substitute or
magnify real experiences with guided experiences that mimic or
reproduce key aspects of real world situations [8]. For example,
students can practice in order to improve: their communication with
the patient and the healthcare team; their expertise in using specific
medical equipment; or their procedural knowledge and technique,
or protocols to be followed under certain circumstances, etc. In this
way, simulations can equip students with essential practical skills
that may be called upon during clinical placements in healthcare
workplaces. Moreover, simulation through scenarios can be an
effective alternative for the achievement of learning outcomes that
are typically met using a traditional lecture format [10].
From the beginning of their degree, healthcare simulations are
integrated into each semester of the Bachelors of Nursing and
Midwifery at the University of Technology Sydney (UTS). During
class, students commonly work in small teams undertaking specific
roles and responding to hypothetical scenarios. Manikins ranging
from newborn to adult (e.g. see Figure 1) give students the
opportunity to practise skills before implementing them in real life
situations. One of the main challenges that teachers and students
face in these scenarios is the very limited awareness of the multiple
co-occurring learning processes in each small group, and the
limited feedback that the teacher can give to the (20 to 30) students
in the classroom. These are common problems that emerge in
several classroom scenarios [15]. As a result, the learning
experience depends on the quality of reflection in which students
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are not
made or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for components
of this work owned by others than the author(s) must be honored.
Abstracting with credit is permitted. To copy otherwise, or republish, to post
on servers or to redistribute to lists, requires prior specific permission and/or
a fee. Request permissions from
LAK '17, March 13 - 17, 2017, Vancouver, BC, Canada
Copyright is held by the owner/author(s). Publication rights licensed to
ACM 978-1-4503-4870-6/17/03…$15.00
Figure 1. The healthcare simulation classroom. Left: small group of learners performing a role-based simulation task.
Center: the classroom setting. Right: a depth sensor tracking learners’ position around a patient manikin in the classroom.
can engage about their own actions and those of others in their
teams. Typically, however, little evidence is gathered that could
support in-class debriefing and reflection, besides video recordings
of the simulation sessions or annotations.
Our research sits in this space, where clinical simulations and
learning analytics can be operationalised together to provide better
feedback to students, and new insights for teachers. There is a
significant opportunity to use multimodal sensors and analytics to
make the normally ephemeral embodied activity in simulations
visible, replayable and referenceable in reflections and analyses by
students and teachers seeking to improve their practice, and (as in
this paper) by researchers. Ultimately, the goal is of course
intensely practical: to make such practices more readily
improvable. This could be accomplished by enriching simulations
with multimodal sensors in order to capture traces of the face-to-
face interactions [3]. Possible ways to exploit these multimodal
data can include the provision of visualisations, notifications or
feedback to students or the teachers, during or after the simulation
activity (e.g. [14]). The analysis of students’ data can also involve
modelling and analytics techniques including the analysis of
behavioural patterns (e.g. interaction of students with the patient);
task performance; students’ sequential processes (e.g. [12]);
communication among students and roles (e.g. [13]); conversation
patterns (e.g. [2]); space usage (e.g. [16]) and gestures.
This paper presents a synthesis of conclusions drawn from an
authentic classroom study aimed at exploring the mechanisms to
collect and integrate multimodal data about students’ interactions
with their peers and the manikins to complete simulation scenarios.
The contributions of the paper are firstly, a discussion of the
challenges encountered in this initial attempt to bring learning
analytics into a real health simulation classroom (in contrast to an
experimental laboratory). Secondly, we present illustrative,
preliminary examples of the analytics that such sensors make
possible. We offer these findings to guide other researchers and
developers seeking to provide support for enhanced awareness,
provision of feedback or evidence-based reflection in face-to-face
(f2f) classroom scenarios.
The rest of the paper is structured as follows. The next section
presents an overview of related work in the area of learning
analytics applied to classroom and face-to-face settings. Section 3
provides details about our healthcare simulation classroom study.
Section 4 presents a synthesis of conclusions drawn from the study
and illustrative analytics examples. The paper concludes in Section
5 with a discussion of lessons learnt and future work.
The study presented in this paper is relevant for both learning
analytics efforts on tackling multimodal interaction (e.g.
employing video, audio and touch sensors to examine learning in a
realistic, mixed-media learning environment) and those focused on
understanding collocated learning environments (where learning
occurs in a physical environment, such as the classroom, where
students not only interact through learning systems but also face-
to-face). Recent reviews of the state of the art of multimodal
analytics and learning analytics in collocated environments can be
found in [3] and [14] respectively. Both have mostly reported work
conducted in controlled, experimental conditions, thus much work
is still needed to bring learning analytics into authentic classrooms.
Our work addresses this gap by reporting on the challenges
encountered and the potential of utilising mobility, audio and log
data towards providing support in a clinical simulation classroom,
a domain that has not yet been explored by learning analytics
initiatives. Our work is inspired by similar LA works conducted
using interactive tabletops and sensors in the classroom to generate
dashboards for the teacher [14] and produce teaching analytics [15].
In the domain of clinical simulation, tools based on video playback
have been developed to aid in debriefing (e.g. [9]) or self-reflection
(e.g. [4]) after the simulation. Although this can be an effective
means of feedback and reflection [7], lengthy or unrelated video
segments can diminish the discussion of key aspects of the
simulation. As a result, video playback is often not suitable for
being used in a classroom where time is commonly limited. To our
knowledge, this work is the first attempt to integrate data from
simulation experiences, with a number of team activity data
streams. Bringing learning analytics into this design space may
provide alternative ways to support debriefing, generate the means
for providing automated feedback during or after the simulation, or
to find patterns indicative of collaboration profiles or student
3.1 The Learning Situation
This study was run in authentic laboratory classes taught in
semester 2, 2016 as part of the undergraduate unit: “Integrated
Nursing Practice”. This is a final-year subject in the Bachelor of
Nursing at the University of Technology Sydney. The main
objective of this unit is to support students to develop clinical
practice skills and prepare them for entry to the nursing workforce.
There is a strong f2f component for the unit, including 3-hour
weekly laboratory classes where students commonly face simulated
scenarios involving chronic and complex conditions. In total, 580
students attended these classes. Each had from 20 to 27 students.
The study focused on the classes conducted in Week 3 of the
semester. The learning design included a series of tasks to be
completed by small teams of students using the patient manikins.
Students had to collaborate and put into practice their theoretical
knowledge and nursing skills to care for a patient in a clinical
scenario. The tasks included: assessment of chest pain symptoms,
administration of medication, management of adverse drug reaction
and conducting an electrocardiogram (ECG) analysis. Additionally,
each student was asked to play one of 4 possible roles: a Team
leader, the Patient, Nurses, and an Observer. The teacher assumed
the Doctor role.
3.2 Apparatus and Multimodal Data Sources
In this study, we focused on 5 randomly selected laboratory classes.
Only the activity occurring in two of the available five simulated
hospital beds was recorded in each session to allow students to opt
out from the data recording. A total of 56 students and 4 different
teachers were in the observed sessions. The laboratory classrooms
are equipped with 5 medium fidelity manikins in bed spaces (see
Figure 1-centre). These manikins generate physiological data and
can be programmed to improve or deteriorate over time in response
to nursing actions.
See Figure 2 for an overview of the bed spaces enhanced with
sensors, which captured the following data. Some of the physical
actions performed on the manikin were automatically logged (e.g.
checking pulse, blood pressure, etc). The SimPad tablet allowed
students to log the completion of required tasks pre-defined by the
teacher to form a checklist. Students’ location data was captured
automatically using a Kinect depth sensor (e.g. see the depth image
recording overlapped with a snapshot of students’ activity in Figure
1-right). Directional audio was captured using the microphone
array (Dev-Audio-Microcone). This captured 6 channels of audio
input around it. Video of the sessions was also captured using
cameras and microphones built into the ceiling. Thus, the types of
data involved in this case were mobility, interaction, self-reported
activity and communication data.
In order to scaffold the analysis of the data and explore the potential
and challenges in terms of learning analytics, we used the Activity-
Centred Analysis & Design (ACAD) framework [13]. This
framework provides a structure for applying different analytical
tools according to four closely related dimensions of user activity:
the social, the setting, the epistemic and the runtime dimensions. In
the following subsections, we present examples of the potential for
learning analytics and the classroom challenges encountered in this
initial attempt. We additionally discuss emerging issues in terms of
data management, ethics and privacy.
4.1 The social dimension
Social formations and mobility. The potential: a feasible source of
student’s behavioural data is the tracked position of the students
around the manikin, which can provide information about how
group members approach the tasks, the processes they follow
before performing actions on the manikin and behaviour according
to learners’ roles. For example, Figure 3 shows the mobility
information of two groups in the same session represented as
heatmaps of location data captured by the depth sensor. These show
two very distinct approaches to the task. Group A stayed mostly
away from the patient during the first half of the
task (see red ovals in quarters Q1 and Q2) to then
work near the patient only during the third quarter
of the activity (see blue ovals in Q3). Then, they
finished earlier than other groups (no data in Q4).
By contrast, Group B followed a very different
approach by engaging with the patient from the
beginning of the task and maintaining proximity
throughout (see blue ovals in Q1-4). This is a
preliminary example of how proximity and
mobility data, when visualised in intuitive ways,
could provoke productive reflection on the
different strategies followed by students. It is a
pedagogical decision exactly how and for what
purposes one would deploy such visualisations
with students.
Challenges: a number of challenges emerged in
the classroom. For example, solutions to
identifying people around the bed (see Figure 4-
left, where the teacher is interacting with the manikin but for the
depth sensor is just another student), or their roles, are still required
to provide continued tracking even though students’ move away
from the depth sensor range. Also, since the depth sensor uses a
computer vision algorithm, problems related with occlusion (e.g.
students occluding each other or being too close to the sensor) and
changes in the setting (e.g. students closing the curtains around the
manikin to perform procedures that required privacy, see Figure 4-
right, tracked in blue) may arise. The potential value of these
heatmaps of mobility for assessing student’s engagement with the
manikin and patient centred care is a current key area of work in
the host university. Still, more work needs to be done to create
robust and unobtrusive solutions to improve the automated
generation of this kind of data.
Audio tracking. The potential: conversation patterns within a group
can be crucial indicators of effective groups’ strategies, processes
or performance [2]. By using the mic array, we aimed to track when
each group member spoke to detect levels of conversation which
could provide insights at a group level or reveal some of the
collaboration story. For example, Figure 5 illustrates the potential
for learning analytics by integrating audio and mobility data for the
overall activity of Groups A and B. The figure shows the total
effective time of conversation as detected by the 6-channels of the
mic array. It can be observed, for example, that most of the talking
for Group A occurred while students were away from the manikin
(see left and central sectors of the microphone indicating 12, ~13
and ~6 minutes of talking corresponding to the clusters of heatmap
activity further from the patient). By contrast, the talking by Group
B mostly occurred on both sides of the manikin (see how the high
heatmap mobility data corresponds to the sectors of the mic array
that detected more talking: ~17 and ~14 minutes).
Challenges: even though these visualisations can be generated
completely automatically, the data used to generate them may have
included some general classroom noise and voices coming from
other beds. Isolating students’ voices in the classroom is
challenging, since it is a very unpredictable and dynamic
environment. Students move and talk at varied voice levels and
there are also unexpected noises such as those generated by the
manikins or medical equipment. Although analytics of student
formations, mobility and conversation patterns can effectively
inform collocated collaboration processes, there are still a number
of challenges to be addressed to get clean data from a laboratory
classroom environment.
Figure 3. Heatmaps of mobility data during student’s activity (1 hour) divided in
quarters for two groups of the same classroom session: A and B. Coloured ovals
mark clusters of activity near (blue) and further (red) from the patient
with sensors
Figure 5. Tracking directional audio (blue areas) and
mobility data (heatmap) for groups A and B
4.2 The setting dimension
The potential: although blended learning models are widespread,
learning analytics have until now focused on the online settings in
which interaction is mediated by computers, since the data is so
easily logged. The f2f setting providing the focus for this paper
seeks to ensure that the co-located elements are no longer invisible,
but could provide automated feedback in the classroom to improve
awareness and reflection in blended-learning settings.
Challenges: unforeseen events can occur in co-located settings.
Commonly, the teacher adapts the task, social aspects and the
setting on-the-fly or in design time to accommodate for these
events. However, these decisions, if not considered in the design,
may affect the quality of the learning analytics outputs or even limit
the types of analytics that can be delivered. In the school hosting
our study, for example, although some manikins are designed for
students to be able to administer injections and intravenous (IV)
fluids (which can generate logged events), other mid-fidelity
manikins do not provide such functionality (thus not generating
logs). External instruments (such as external IV or ECG devices)
are also sometimes used by the teaching team to enhance student’s
experience or augment some manikins lacking of certain
functionalities. However, they are generally not connected to the
manikin system and/or do not record activity logs. Thus, there can
be a trade-off between the setting needs for classroom orchestration
and learning; and what is feasible to be achieved through learning
analytics innovations.
4.3 Epistemic dimension (the task)
The potential: the possibility of generating visual analytics that
describe the different steps and key milestones of the collaborative
learning process within a group of students can provide an effective
area for discussion, reflection or the provision of more informed
feedback. For example, Figure 6 (see coloured pins on the blue
timeline) shows the logs obtained from two groups in two different
sessions. These correspond to three key actions which are important
for this specific simulation case: 1) when the first set of vital signs
of the patient was obtained (red pins), 2) when a prescription
medication was administered (blue pins) and 3) when an ECG test
was performed after the patient deteriorated (yellow pins). Whilst
other actions can also be visualised, these examples indicate how
the achieved milestones can be helpful for the teacher or the
students to reflect on the process followed by each group or groups
in the same classroom.
Challenges: capturing traces of the task and also making sense of
this information in order to render it actionable can be very
challenging. Another epistemic challenge to overcome is the
impact of the task instructions on the data capture. In our example,
the teacher encouraged students to self-report their actions on a pre-
formatted checklist on the tablet, (which although a common
practice for practicing nurses is usually performed on paper). But
only 3 groups completed the checklist, and one of them did it as
explicitly requested by the teacher after the session.
4.4 The runtime enactment of the task
The potential: there is a growing interest in the learning analytics
community in connecting the learning design with evidence-based
data innovations [11] since activity data makes much more sense
when the activity context is known. However, not much has been
done to provide analytics for comparing the enactment of the tasks
in the classroom with the learning design intentions. Figure 6 also
illustrates the timelines that show how two classroom sessions
unfolded. In this case, the manikin logs and the mobility data can
provide evidence about the duration of the simulation session. In
Figure 6. Design analytics for two classroom sessions: showing three key milestones reached by at least 1 group during the
simulation (obtained from the manikin logs and student’s self-reported events) shown in a timeline that represents the enactment
of the learning design
Figure 4. Tracking challenges. Left: a small group (3 students and the teacher). Right: a very large group formed by latecomers
(tracking affected by occlusion and when
students closed the medical curtain
see blue
shaded area
tagged as ‘
the first class (T1) the teacher took ~45 minutes to provide
instructions and discuss the case and then ~1.5 hours for the
reflection after the simulation. By contrast, the second teacher (T2)
dedicated most of the class to the simulation (almost 2 hours)
allowing students to perform the actions at a different pace (note
the separation between the coloured pins in T1 and T2). This type
of visualisation could be offered to a unit coordinator to fine-tune
the session design, or to be aware of the variation in execution.
Challenges: in our study, each of the four teachers enacted the
learning design quite differently, making an impact on the task and
therefore on the analytics. However, learning analytics can provide
insights about how divergent teachers are in enacting the design.
Overall, unexpected classroom events, besides the ones mentioned
in previous subsections, strongly influenced or limited our first
attempt in collecting students’ data for analytics. For example, in
one of the classes, students arriving late to the class, one by one,
generated large groups that were quite dysfunctional and also hard
to track using the depth sensor (see Figure 4, right).
4.5 Data management and ethics
Challenges: privacy, consent and data management issues need to
be taken into consideration for deploying learning analytics in
authentic educational settings. This is particularly crucial for
situations where student data cannot be de-identified or which
involve video/audio recording (as in our case). For example, in our
study, all students were informed by the lecturer about the study a
week before the recording sessions. Blanket authorisation was
granted by the ethics committee of the institution. Then, during the
study, only two out of the 5 clinical beds were recorded (there were
always unmonitored beds in each class) allowing students to
voluntarily participate. Students in this Faculty are already used to
the occasional recording session being organised as it is a common
practice in this learning space. As this is more of an exception than
a commonality in other classrooms, work needs to be done to
explore sustainable strategies to request consent and clarify data
management issues without stretching the already very limited
classroom time.
Learning analytics innovations deployed in authentic physical
classrooms are still rare. We have presented our initial attempt in
bringing learning analytics into a healthcare simulation classroom
aiming to uncover small group learning, collaboration and
enactment processes. We faced several challenges that still need to
be addressed in order to start gaining insights about student’s processes.
Through our illustrative preliminary examples, we uncovered the
potential that these multimodal analytics can bring to reveal
different aspects of the collaborative process, classroom dynamics
and the enactment of the learning design which are commonly
invisible. Finally, we illustrated the complexity of implementing
learning analytics in the classroom, not only in terms of limitations
in the technology, but also because the classroom is a very complex
and dynamic environment where the epistemic, social and physical
dimensions play a crucial role in making each session unique and
filled with unforeseen events. Although the unpredictable nature of
the classroom may affect learning analytics, it certainly makes a
classroom a rich setting that offers great opportunities for learning.
We envisage that this paper can be useful for other researchers and
developers seeking to provide enhanced support in simulation
laboratories and also in more generic collocated settings. Our future
work is aimed at generating the technical and pedagogical means
for providing automated feedback, and/or supporting awareness
and reflection in f2f classroom scenarios, similarly to what is
currently available in computer-mediated (online) platforms.
[1] Barrows, H. S. 1996. Problem‐based learning in medicine and
beyond: A brief overview. New directions for teaching and
learning, 1996, 68, 3-12.
[2] Bergstrom, T., and Karahalios, K. 2007. Conversation Clock:
Visualizing audio patterns in co-located groups. In Proc. of
40th Annual Hawaii International Conference on System
Sciences (Big Island, Hawaii, USA, Jan 3-6). IEEE Computer
Society, 78-87.
[3] Blikstein, P., and Worsley, M. 2016. Multimodal Learning
Analytics and Education Data Mining: using computational
technologies to measure complex learning tasks. Journal of
Learning Analytics, 3, 2, 220-238.
[4] Bussard, M. E. 2016. Self-Reflection of Video-Recorded
High-Fidelity Simulations and Development of Clinical
Judgment. Journal of Nursing Education, 55, 9, 522-527.
[5] Cooper, J., and Taqueti, V. 2008. A brief history of the
development of mannequin simulators for clinical education
and training. Postgraduate medical journal, 84, 997, 563-570.
[6] Dieckmann, P. 2009. Using simulations for education, training
and research: Pabst Science Publishers.
[7] Fanning, R. M., and Gaba, D. M. 2007. The role of debriefing
in simulation-based learning. Simulation in healthcare, 2, 2,
[8] Gaba, D. M. 2004. The future vision of simulation in health
care. Quality and safety in Health care, 13, suppl 1, i2-i10.
[9] Grant, J. S., Moss, J., Epps, C., and Watts, P. 2010. Using
video-facilitated feedback to improve student performance
following high-fidelity simulation. Clinical Simulation in
Nursing, 6, 5, e177-e184.
[10] LeFlore, J. L., Anderson, M., Zielke, M. A., Nelson, K. A.,
Thomas, P. E., Hardee, G., and John, L. D. 2012. Can a virtual
patient trainer teach student nurses how to save lives—teaching
nursing students about pediatric respiratory diseases.
Simulation in Healthcare, 7, 1, 10-17.
[11] Lockyer, L., Heathcote, E., and Dawson, S. 2013. Informing
pedagogical action: Aligning learning analytics with learning
design. American Behavioral Scientist, 57, 10, 1439-1459.
[12] Martinez-Maldonado, R., Dimitriadis, Y., Martinez-Monés,
A., Kay, J., and Yacef, K. 2013. Capturing and analyzing verbal
and physical collaborative learning interactions at an enriched
interactive tabletop. International Journal of Computer-
Supported Collaborative Learning, 8, 4 (Nov. 2013), 455-485.
[13] Martinez-Maldonado, R., Goodyear, P., Kay, J., Thompson,
K., and Carvalho, L. 2016. An Actionable Approach to
Understand Group Experience in Complex, Multi-surface
Spaces. In Proc. of CHI' 16 (San Jose, USA, May 7-12). NY:
ACM, 2062-2074.
[14] Martinez-Maldonado, R., Schneider, B., Charleer, S., Shum,
S. B., Klerkx, J., and Duval, E. 2016. Interactive Surfaces and
Learning Analytics: Data, Orchestration Aspects, Pedagogical
Uses and Challenges. In Proc. of LAK' 16 (Edinburgh, UK,
April 25-29). NY: ACM, 124-133
[15] Prieto, L. P., Sharma, K., Dillenbourg, P., and Jesús, M.
2016. Teaching analytics: towards automatic extraction of
orchestration graphs using wearable sensors. In Proc. of LAK'
16 (Edinburgh, UK, April 25-29). 148-157.
[16] von Zadow, U., Buron, S., Harms, T., Behringer, F.,
Sostmann, K., and Dachselt, R. 2013. SimMed: combining
simulation and interactive tabletops for medical education. In
Proc. of CHI' 13. NY: ACM, 1469-1478.
... The measurement of online collaboration processes is possible due to the measurement, collection and analysis of the learner data using learning analytics (Siemens, 2011;Greller and Drachsler, 2012). With the ubiquitous usage of sensors lately, a new branch of learning analytics otherwise known as multimodal learning analytics (MMLA) has risen to prominence (Di Mitri et al., 2018a;Martinez-Maldonado et al., 2017a). Moreover, sensor technology has become more scalable , affordable and reliable in the past decade (Starr et al., 2018). ...
... Het meten van online samenwerkingsprocessen is mogelijk door het meten, verzamelen en analyseren van de gegevens van leerlingen met behulp van learning analytics (Siemens, 2011;Greller and Drachsler, 2012). Met het alomtegenwoordige gebruik van sensoren de laatste tijd, is een nieuwe tak van learning analytics, ook wel bekend als multimodale learning analytics (MMLA), op de voorgrond getreden (Di Mitri et al., 2018a;Martinez-Maldonado et al., 2017a). Bovendien is sensortechnologie in het afgelopen decennium schaalbaarder , betaalbaar en betrouwbaar geworden (Starr et al., 2018). ...
... Lubold and Pon-Barry (2014),Nakano et al. (2015),Luz (2013),Grover et al. (2016),Worsley and Blikstein (2015),Martinez-Maldonado et al. (2017a),Oviatt et al. (2015),Scherer et al. (2012),Schneider and Pea (2014a),Spikol et al. (2017b),Echeverria et al. (2017),Ochoa et al. (2013),,Scherr and Hammer (2009),Terken and Sturm (2010),Kim et al. (2008),Bergstrom and Karahalios (2007),Bassiou et al. (2016),Thompson et al. (2014),Viswanathan and VanLehn (2017),Kim et al. (2015),Martinez- Maldonado et al. (2013), Bachour et al. (2010), Spikol et al. (2017b), Praharaj et al. (2018b), Worsley and Blikstein (2018), Davidsen and Ryberg (2017), Emara et al. (2017), Rodríguez et al. (2017), Dornfeld et al. (2017), Fake et al. (2017), McBride et al. (2017), Abdu (2015), Flood et al. (2015), Wise et al. (2015), Wake et al. (2015), Martin et al. (2015), Andrade (2015), Dornfeld and Puntambekar (2015), Hardy and White (2015), Andrade-Lotero et al. (2013), Thompson et al. (2013), Martinez et al. (2011), Wong et al. (2011), Johansson et al. (2011), Noel et al. (2018), Bhattacharya et al. (2018), Stewart et al. (2018), Olsen and Finkelstein (2017), Henning et al. al. (2016), Ochoa et al. (2013), Schneider and Blikstein (2015), Scherr and Hammer (2009), Kim et al. (2008), Cukurova et al. (2018), Cukurova et al. (2017a), Cukurova et al. (2017b), Viswanathan and Van-Lehn (2017), Dich et al. (2018), Davidsen and Ryberg (2017), Stiefelhagen and Zhu (2002), Wise et al. (2017), Bhattacharya et al. (2018), Reilly et al. al. (2016), Worsley and Blikstein (2015), Spikol et al. (2017a), Echeverria et al. (2017), Ochoa et al. (2013), Martinez-Maldonado et al. (2015a), Schneider and Blikstein (2015), Scherr and Hammer (2009), Kim et al. (2008), Anastasiou and Ras (2017), Cukurova et al. (2018), Cukurova et al. (2017a), Cukurova et al. (2017b), Viswanathan and VanLehn (2017), Martinez-Maldonado et al. (2013), Worsley and Blikstein (2018), Davidsen and Ryberg (2017), Wise et al. (2017), Emara et al. (2017), Flood et al. (2015), Wake et al. (2015), Hardy and White (2015), Andrade-Lotero et al. (2013), Martinez et al. (2011), Johansson et al. (2011) Eye gaze camera, eye tracker, eye tracking glass Dierker et al. (2009), Nakano et al. (2015), Li et al. (2010), Grover et al. (2016), Schneider and Pea (2014a), Spikol et al. (2017a), Schneider et al. (2015), Scherr and Hammer (2009), Terken and Sturm (2010), Dich et al. (2018), Andrist et al. (2018), Davidsen and Ryberg (2017), Stiefelhagen and Zhu (2002), Flood et al. (2015), Martinez-Maldonado et al. (2015b), Wake et al. (2015), Andrade (2015), Andrade-Lotero et al. (2013) Spatial Kinect, camera Martinez-Maldonado et al. (2017a), Healion et al. (2017), Schneider and Blikstein (2015), Kim et al. (2008), Martinez-Maldonado et al. (2017b), Spikol et al. (2017b), Spikol et al. (2018b), Wise et al. (2017), Martinez-Maldonado et al. (2015b), Reilly et al. (2018) ...
Full-text available
The main objectives of the thesis are: (1) to define the constituents helping to detect co-located (or face-to-face) collaboration (CC) quality (i.e., indicators, indexes and parameters in this case); (2) to design and develop a sensor-based set up for automatic CC analytics; (3) to move towards quantifying the quality of collaboration by using the CC analytics set up and show the visualizations on a dashboard.
... Martinez-Maldonado et al. (2018) also highlighted that future work should explore the ethical issues that can emerge from MMLA. Similarly, Rodriguez Triana et al. (2017) raised an issue related to privacy and user traceability stating that GDPR requirements might lead future technology to generate only anonymous data, Martinez-Maldonado et al. (2017) recommended that issues related to privacy, consent, and data management should be considered when deploying learning analytics in real educational settings to produce sustainable strategies to handle these issues without stretching the classroom time; and Noel et al. (2018) stated that participant's privacy has been protected by labelling the collected survey with identification codes (the number of the team they were in and the microphone they were using). We were not able to find many studies that discuss the ethical implications of MMLA research from our first search. ...
... The first author gratefully acknowledges Princess Nourah Bint Abdulrahman University and the Saudi Arabian Cultural Bureau in London for funding her PhD study at University College London. Alyuz et al., 2017;Barmaki & Hughes, 2015;Barmaki, 2015;Beardsley et al., 2020;Deshmukh et al., 2018;Correa et al., 2020;Hsieh et al., 2010;Järvenoja et al., 2020;Lai et al., 2013;Lee-Cultura et al., 2020a;Lew & Tang, 2017;Liu et al., 2018;Martinez-Maldonado et al., 2017Noroozi et al., 2019;Prieto et al., 2016;Romano et al., 2019;Sharma et al., 2019;Su et al., 2013;Tamura et al., 2019 ...
There is a growing interest in the research and use of multimodal data in learning analytics. This paper presents a systematic literature review of multimodal learning analytics (MMLA) research to assess (i) the available evidence of impact on learning outcomes in real-world contexts and (ii) explore the extent to which ethical considerations are addressed. A few recent literature reviews argue for the promising value of multimodal data in learning analytics research. However, our understanding of the challenges associated with MMLA research from real-world teaching and learning environments is limited. To address this gap, this paper provides an overview of the evidence of impact and ethical considerations stemming from an analysis of the relevant MMLA research published in the last decade. The search of the literature resulted in 663 papers, of which 100 were included in the final synthesis. The results show that the evidence of real-world impact on learning outcomes is weak, and ethical challenges of MMLA work are rarely addressed. In this paper, we discuss the results through the lenses of two theoretical frameworks (1) evidence of impact types and (2) ethical dimensions of MMLA. We conclude that for MMLA to stay relevant and become part of mainstream education, future research should directly address the gaps identified in this review.
... Audio Kinect, microphones, sociometric badge [41], [42], [43], [5], [44], [45], [46], [47], [48], [38], [49], [50], [51], [33], [21], [6], [52], [53], [54], [28], [55], [56], [19], [38], [35], [57], [34], [58], [59], [60], [61], [62], [63], [64], [65], [66], [67], [68], [69], [29], [30], [70], [71], [72], [73], [74], [75], [76], [77], [78] Posture ...
... Kinect, camera, ceiling mounted time-offlight sensors, sociometric badge [5], [50], [79], [33], [6], [37], [36], [40], [28], [80], [34], [22], [81], [75], [12] Gesture Kinect, camera [5], [44], [82], [49], [50], [83], [79], [33], [6], [39], [37], [36], [40], [28], [56], [57], [34], [81], [58], [64], [66], [29], [30], [71], [73] Eye gaze camera, eye tracker, eye tracking glass [84], [42], [85], [5], [48], [82], [20], [33], [21], [80], [86], [34], [22], [64], [87], [66], [68], [30] Spatial Kinect, camera [45], [88], [79], [6], [89], [38], [90], [81], [87], [12] Content (i.e., ideas, knowledge, or task related log data) tangible-user-interface (TUI), human observer, tablets [11], [6], [70], [91], [92], [93], [49], [50], [83], [79], [94], [33], [39], [95], [54], [28], [56], [96], [60], [61], [97], [62], [63], [64], [87], [98], [65], [67], [29], [99], [71], [72], [77] Writing digital pen [42], [100], [47], [50], [94], [95] Physiological empatica [44], [101], [80], [96], [57], [102], [103], [78], [104], [13], [105] Self-reports online forms, questionnaires [94], [6], [39], [101], [95], [84] a Sensors report which hardware sensors have been used to detect these indicator types in each of these referenced articles. b Indicator types report the cluster of similar indicators. ...
Full-text available
Collaboration is one of the important 21st-century skills. It can take place in remote or co-located settings. Co-located collaboration (CC) is a very complex process that involves subtle human interactions that can be described with indicators like eye gaze, speaking time, pitch, and social skills from different modalities. With the advent of sensors, multimodal learning analytics has gained momentum to detect CC quality. Indicators (or low-level events) can be used to detect CC quality with the help of measurable markers (i.e., indexes composed of one or more indicators) which give the high-level collaboration process definition. However, this understanding is incomplete without considering the scenarios (such as problem solving or meetings) of CC. The scenario of CC affects the set of indicators considered: for instance, in collaborative programming, grabbing the mouse from the partner is an indicator of collaboration; whereas in collaborative meetings, eye gaze, and audio level are indicators of collaboration. This can be a result of the differing goals and fundamental parameters (such as group behavior, interaction, or composition) in each scenario. In this review, we present our work on profiles of indicators on the basis of a scenario-driven prioritization, the parameters in different CC scenarios are mapped onto the indicators and the available indexes. This defines the conceptual model to support the design of a CC quality detection and prediction system.
... Multiple data streams, including video, audio, self-reports, interaction traces, are analyzed to classify emotions [12]. Another system created for healthcare education analyzes multimodal data gathered while students practice medical techniques on manikins [29]. Recently, these efforts have expanded to include interactions among students, for example multimodal patterns during collaborative problem solving [49]. ...
... However, the educator and learners commonly do not have objective evidence to discuss as replaying the video-recordings can be time consuming and unpractical. This has motivated the use of multimodal learning analytics to augment the data capture capabilities of the simulation rooms to identify salient aspects of team activity and make these available during the debrief (Martinez-Maldonado et al., 2017). The data is collected using indoor positioning sensors, microphones, physiological wristbands and an observation console (Martinez-Maldonado, Echeverria, Fernandez Nieto, & Buckingham Shum, 2020). ...
Full-text available
There are emerging concerns about the Fairness, Accountability, Transparency, and Ethics (FATE) of educational interventions supported by the use of Artificial Intelligence (AI) algorithms. One of the emerging methods for increasing trust in AI systems is to use eXplainable AI (XAI), which promotes the use of methods that produce transparent explanations and reasons for decisions AI systems make. Considering the existing literature on XAI, this paper argues that XAI in education has commonalities with the broader use of AI but also has distinctive needs. Accordingly, we first present a framework, referred to as XAI-ED, that considers six key aspects in relation to explainability for studying, designing and developing educational AI tools. These key aspects focus on the stakeholders, benefits, approaches for presenting explanations, widely used classes of AI models, human-centred designs of the AI interfaces and potential pitfalls of providing explanations within education. We then present four comprehensive case studies that illustrate the application of XAI-ED in four different educational AI tools. The paper concludes by discussing opportunities, challenges and future research needs for the effective incorporation of XAI in education.
... Training doctors in performing diagnosis [47] and first responders on proper CPR techniques [48] are also areas where mannequin testing data are widely used. Mannequin torsos and heads are also regularly used in testing audio wave propagation and tuning devices [49,50]. ...
Full-text available
Outdoor fall detection, in the context of accidents, such as falling from heights or in water, is a research area that has not received as much attention as other automated surveillance areas. Gathering sufficient data for developing deep-learning models for such applications has also proven to be not a straight-forward task. Normally, footage of volunteer people falling is used for providing data, but that can be a complicated and dangerous process. In this paper, we propose an application for thermal images of a low-cost rubber doll falling in a harbor, for simulating real emergencies. We achieve thermal signatures similar to a human on different parts of the doll’s body. The change of these thermal signatures over time is measured, and its stability is verified. We demonstrate that, even with the size and weight differences of the doll, the produced videos of falls have a similar motion and appearance to what is expected from real people. We show that the captured thermal doll data can be used for the real-world application of pedestrian detection by running the captured data through a state-of-the-art object detector trained on real people. An average confidence score of 0.730 is achieved, compared to a confidence score of 0.761 when using footage of real people falling. The captured fall sequences using the doll can be used as a substitute to sequences of people.
Full-text available
There is a growing interest in the research and use of multimodal data in learning analytics. This paper presents a systematic literature review of multimodal learning analytics (MMLA) research to assess i) the available evidence of impact on learning outcomes in real-world contexts and ii) explore the extent to which ethical considerations are addressed. A few recent literature reviews argue for the promising value of multimodal data in learning analytics research. However, our understanding of the challenges associated with MMLA research from real-world teaching and learning environments is limited. To address this gap, this paper provides an overview of the evidence of impact and ethical considerations stemming from an analysis of the relevant MMLA research published in the last decade. The search of the literature resulted in 663 papers, of which 100 were included in the final synthesis. The results show that the evidence of real-world impact on learning outcomes is weak, and ethical aspects of MMLA work are rarely addressed. We discuss our results through the lenses of two theoretical frameworks for evidence of impact types and ethical dimensions of MMLA. We conclude that for MMLA to stay relevant and become part of mainstream education, future research should directly address the gaps identified in this review.
Despite the ubiquity of learning in workplace and professional settings, the learning analytics (LA) community has paid significant attention to such settings only recently. This may be due to the focus on researching formal learning, as workplace learning is often informal, hard to grasp and not unequivocally defined. This paper summarizes the state of the art of Workplace Learning Analytics (WPLA), extracted from a two-iteration systematic literature review. Our in-depth analysis of 52 existing proposals not only provides a descriptive view of the field, but also reflects on researcher conceptions of learning and their influence on the design, analytics and technology choices made in this area. We also discuss the characteristics of workplace learning that make WPLA proposals different from LA in formal education contexts and the challenges resulting from this. We found that WPLA is gaining momentum, especially in some fields, like healthcare and education. The focus on theory is generally a positive feature in WPLA, but we encourage a stronger focus on assessing the impact of WPLA in realistic settings.
Full-text available
New high-frequency multimodal data collection technologies and machine learning analysis techniques could offer new insights into learning, especially when students have the opportunity to generate unique, personalized artifacts, such as computer programs, robots, and solutions engineering challenges. To date most of the work on learning analytics and educational data mining has been focused on online courses and cognitive tutors, both of which provide a high degree of structure to the tasks, and are restricted to interactions that occur in front of a computer screen. In this paper, we argue that multimodal learning analytics can offer new insights into students’ learning trajectories in more complex and open-ended learning environments. We present several examples of this work and its educational application.
Full-text available
"Teaching analytics" is the application of learning analytics techniques to understand teaching and learning processes, and eventually enable supportive interventions. However, in the case of (often, half-improvised) teaching in face-to-face classrooms, such interventions would require first an understanding of what the teacher actually did, as the starting point for teacher reflection and inquiry. Currently, such teacher enactment characterization requires costly manual coding by researchers. This paper presents a case study exploring the potential of machine learning techniques to automatically extract teaching actions during classroom enactment, from five data sources collected using wearable sensors (eye-tracking, EEG, accelerometer, audio and video). Our results highlight the feasibility of this approach, with high levels of accuracy in determining the social plane of interaction (90%, k=0.8). The reliable detection of concrete teaching activity (e.g., explanation vs. questioning) accurately still remains challenging (67%, k=0.56), a fact that will prompt further research on multimodal features and models for teaching activity extraction, as well as the collection of a larger multimodal dataset to improve the accuracy and generalizability of these methods.
Conference Paper
Full-text available
The proliferation of varied types of multi-user interactive surfaces (such as digital whiteboards, tabletops and tangible interfaces) is opening a new range of applications in face-to-face (f2f) contexts. They offer unique opportunities for Learning Analytics (LA) by facilitating multi-user sensemaking of automatically captured digital footprints of students' f2f interactions. This paper presents an analysis of current research exploring learning analytics associated with the use of surface devices. We use a framework to analyse our first-hand experiences, and the small number of related deployments according to four dimensions: the orchestration aspects involved; the phases of the pedagogical practice that are supported; the target actors; and the levels of iteration of the LA process. The contribution of the paper is twofold: 1) a synthesis of conclusions that identify the degree of maturity, challenges and pedagogical opportunities of the existing applications of learning analytics and interactive surfaces; and 2) an analysis framework that can be used to characterise the design space of similar areas and LA applications.
Conference Paper
Full-text available
There is a steadily growing interest in the design of spaces in which multiple interactive surfaces are present and, in turn, in understanding their role in group activity. However, authentic activities in these multi-surface spaces can be complex. Groups commonly use digital and non-digital artefacts, tools and resources, in varied ways depending on their specific social and epistemic goals. Thus, designing for collaboration in such spaces can be very challenging. Importantly, there is still a lack of agreement on how to approach the analysis of groups' experiences in these heterogeneous spaces. This paper presents an actionable approach that aims to address the complexity of understanding multi-user multi-surface systems. We provide a structure for applying different analytical tools in terms of four closely related dimensions of user activity: the setting, the tasks, the people and the runtime co-configuration. The applicability of our approach is illustrated with six types of analysis of group activity in a multi-surface design studio.
Full-text available
The aim of this book is to introduce the reader to the use of the HELOS hearing-loss simulator in education, training, and research. The contents are based on materials and procedures that have been developed by the author. Chapters 1 Hearing/vision loss and simulation 2 Listen through HELOS 3 Communicate through HELOS 4 Applications: The CONAN system 5 Student training 6 Applications: Submitted by HELOS users
Full-text available
Interactive tabletops can be used to provide new ways to support face-to-face collaborative learning. A little explored and somewhat hidden potential of these devices is that they can be used to enhance teachers' awareness of students' progress by exploiting captured traces of interaction. These data can make key aspects of collaboration visible and can highlight possible problems. In this paper, we explored the potential of an enriched tabletop to automatically and unobtrusively capture data from collaborative interactions. By analyzing that data, there was the potential to discover trends in students' activity. These can help researchers, and eventually teachers, to become aware of the strategies followed by groups. We explored whether it was possible to differentiate groups, in terms of the extent of collaboration, by identifying the interwoven patterns of students' speech and their physical actions on the interactive surface. The analysis was validated on a sample of 60 students, working in triads in a concept mapping learning activity. The contribution of this paper is an approach for analysing students' interactions around an enriched interactive tabletop that is validated through an empirical study that shows its operationalization to extract frequent patterns of collaborative activity.
Full-text available
This article considers the developing field of learning analytics and argues that to move from small-scale practice to broad scale applicability, there is a need to establish a contextual framework that helps teachers interpret the information that analytics provides. The article presents learning design as a form of documentation of pedagogical intent that can provide the context for making sense of diverse sets of analytic data. We investigate one example of learning design to explore how broad categories of analytics—which we call checkpoint and process analytics—can inform the interpretation of outcomes from a learning design and facilitate pedagogical action.
Full-text available
Background: This pilot study evaluated the effect of videotape-facilitated human patient simulator (HPS) practice and guidance on clinical performance indicators. Method: Nursing and nurse anesthetist students in the treatment group (n = 20) participated in HPS practice and guidance using videotape-facilitated debriefing, and the control group (n = 20) participated in HPS practice and guidance using oral debriefing alone. Results: Students in the intervention group were significantly more likely to demonstrate desirable behaviors concerning patient identification, team communication, and vital signs. The role students played in the simulation significantly impacted their performance. When scores of both the intervention and control groups were combined, team leaders, airway managers, and nurse anesthetists had higher mean total performance scores than crash cart managers, recorders, or medication nurses. Conclusion: Video-facilitated simulation feedback is potentially a useful tool in increasing desirable clinical behaviors in a simulated environment. © 2010 International Nursing Association for Clinical Simulation and Learning.
Background: Nurse educators are increasingly using high-fidelity simulators to improve prelicensure nursing students' ability to develop clinical judgment. Traditionally, oral debriefing sessions have immediately followed the simulation scenarios as a method for students to connect theory to practice and therefore develop clinical judgment. Recently, video recording of the simulation scenarios is being incorporated. Method: This qualitative, interpretive description study was conducted to identify whether self-reflection on video-recorded high-fidelity simulation (HFS) scenarios helped prelicensure nursing students to develop clinical judgment. Tanner's clinical judgment model was the framework for this study. Results: Four themes emerged from this study: Confidence, Communication, Decision Making, and Change in Clinical Practice. Conclusion: This study indicated that self-reflection of video-recorded HFS scenarios is beneficial for prelicensure nursing students to develop clinical judgment. [J Nurs Educ. 2016;55(9):522-527.].
Conference Paper
A large body of work asserts that interactive tabletops are well suited for group work, and numerous studies have examined these devices in educational contexts. However, few of the described systems support simulations for collaborative learning, and none of them explicitly address immersion. We present SimMed, a system allowing medical students to collaboratively diagnose and treat a virtual patient using an interactive tabletop. The hybrid user interface combines elements of virtual reality with multitouch input. The paper delineates the development process of the system and rationale behind a range of interface design decisions. Thereby, the role of realism in gaining procedural knowledge is discussed - in particular, the interplay between realism, immersion and training goals. We implemented several medical test cases and evaluated our approach with a user study that suggests the great potential of the system. Results show a high level of immersion, cooperation and engagement by the students.