Content uploaded by María Jesús Rodríguez-Triana
Author content
All content in this area was uploaded by María Jesús Rodríguez-Triana on Feb 01, 2025
Content may be subject to copyright.
This is the accepted, uncorrected manuscript. Content may change prior to final publication. The Version of Record of this manuscript has been
published and is available in International Journal of Human-Computer Interaction 2017, https://doi.org/10.1080/10447318.2017.1331536
Monitoring for awareness and reflection in ubiquitous learning
environments
Juan A. Muñoz-Cristóbal*
,a, María Jesús Rodríguez-Trianab,c, Vanesa Gallego-Lemad,
Higinio F. Arribas-Cuberod, Juan I. Asensio-Péreza, Alejandra Martínez-Monése
aSchool of Telecommunications Engineering, Universidad de Valladolid, Spain
bREACT Research Group, École Polytechnique Fédérale de Lausanne, EPFL SCI STI DG, Switzerland;
cCentre of Excellence in Educational Innovation School of Digital Technologies, Tallinn University, Estonia;
dFaculty of Education and Social Work, Universidad de Valladolid, Spain
eSchool of Computer Science, Universidad de Valladolid, Spain
Abstract: Despite the educational affordances that ubiquitous learning has shown, it is still
hampered by several orchestration difficulties. One of these difficulties is that teachers lose
awareness of what the students perform across the multiple technologies and spaces involved.
Monitoring can help in such awareness, and it has been highly explored in face-to-face and
blended learning. Nevertheless, in ubiquitous learning environments monitoring has been
usually limited to activities taking place in a specific type of space (e.g., outdoors). In this
paper we propose a monitoring system for ubiquitous learning, which was evaluated in three
authentic studies, supporting the participants in the affordable monitoring of learning
situations involving web, augmented-physical, and 3D virtual world spaces. The work carried
out also helped identify a set of guidelines, which are expected to be useful for researchers and
technology developers aiming to provide participants' support in ubiquitous learning
environments.
Keywords: Learning analytics, monitoring, ubiquitous learning, virtual learning environment,
augmented reality, virtual world.
1. Introduction
With the advance of technology during the last decades, a multiplicity of technology-enabled learning spaces is
emerging. Physical spaces such as classrooms, parks, museums, or the natural environment, are enriched with a
variety of electronic devices: interactive whiteboards, computers, mobile phones, tablets, tabletops, etc. These
devices are actually doors to other virtual learning spaces, like the Web or even 3D virtual worlds (3DVWs), in
which learning is mediated by software tools such as web Virtual Learning Environments (VLEs, e.g.,
Moodle1), 3DVWs platforms (e.g., Second Life2) or Virtual Globes (VGs, e.g., Google Earth3). These multiple
learning spaces have shown different affordances for learning. For example, they can help contextualize
learning, increase student motivation and engagement, enhance spatial skills, and enable the use of several
pedagogical approaches such as active, situated or experiential learning (Dalgarno & Lee, 2010; Dyson,
Litchfield, Lawrence, Raban, & Leijdekkers, 2009; Keller, 2005; Rakshit & Ogneva-Himmelberger, 2008).
There has been substantial research focused on the continuity of the learning experience across spaces (Milrad et
al., 2013) enabling students to benefit from the affordances of different spaces, while learning anytime
anywhere. Technologies like mobile devices, sensors, and Augmented Reality (AR, i.e., the combination of
virtual and physical objects in a physical environment) are helping connect the different spaces, enabling
ubiquitous learning situations (Li, Zheng, Ogata, & Yano, 2004; Wu, Lee, Chang, & Liang, 2013). For instance,
a Google Docs4
* Corresponding author. E-mail:
document generated by a group of students in a classroom can be afterwards used in-context in
a park using AR. In fact, in ubiquitous learning, there is a special emphasis on the physical context where the
learning activity takes place, which is a core factor in the typical educational approaches involving different
spaces (Milrad et al., 2013).
juanmunoz@gsic.uva.es
1 https://moodle.org. Last access April, 2017.
2 http://secondlife.com. Last access April, 2017.
3 https://www.google.com/earth/. Last access April, 2017.
4 https://www.google.com/docs/about/. Last access April, 2017.
2
However, despite the benefits that the ubiquitous learning situations may provide, teachers still face several
difficulties to create and conduct this kind of situations (Delgado Kloos, Hernández-Leo, & Asensio-Pérez,
2012). These difficulties to create and enact learning situations in technologically-complex educational settings
(not only ubiquitous) have been encompassed by the research community under the “orchestration” metaphor
(Prieto, Dlab, Gutiérrez, Abdulwahed, & Balid, 2011; Roschelle, Dimitriadis, & Hoppe, 2013). Ubiquitous
learning situations, where the activities frequently involve a number of separate groups of students interacting
simultaneously from distant locations using different technologies, pose special obstacles to orchestration. One
of these obstacles is that teachers lose awareness of what students do, and need special help to keep track of the
development of the activities and the progress (or lack thereof) of the different groups. One of the key functions
that can help teachers in these orchestration obstacles is monitoring (Dillenbourg, 2013). Monitoring is an
ongoing function that uses the systematic collection of data related to specified indicators to provide the
different participants of a learning activity with indications regarding the progress and results of such activity
(Marriott & Goyder, 2009). Monitoring can aid in different aspects of learning orchestration. For instance, by
informing teachers in order to support decision-making during design, management or assessment of learning
situations (Chatti, Dyckhoff, Schroeder, & Thüs, 2012); or by informing students in order to promote self-
regulated (Tabuenca, Kalz, & Specht, 2014) or collaborative learning (Wecker & Fischer, 2007).
The use of monitoring in blended learning has been intense especially in settings making use of centralized
educational systems such as VLEs (Chatti et al., 2012). Also, the research community has explored monitoring
in web educational systems, in which data coming from distributed sources needs to be triangulated and
integrated (Rodríguez-Triana, Martínez-Monés, & Asensio-Pérez, 2011; Rodríguez-Triana et al., in press; J. L.
Santos et al., 2015). Monitoring complexity increases even more when the learning situations may involve
multiple physical and virtual spaces (Chatti et al., 2012). In ubiquitous learning situations, where, as mentioned,
typically the physical context is relevant, context-aware data usually needs to be collected using a variety of
devices, such as sensors, and be integrated with the already heterogeneous data of traditional distributed
educational systems (e.g., VLEs, Web 2.0 tools, social applications, etc.). However, despite the need for
solutions dealing with monitoring in ubiquitous learning situations in order to help teachers in orchestration
(Long & Siemens, 2011), research in this field is still in its infancy. Most of the orchestration approaches
considering physical spaces beyond the classroom propose solutions for monitoring the activities only in those
physical spaces, typically using mobile devices, without integrating such data with other data coming from other
learning activities carried out in different spaces using other devices (e.g., from a VLE such as Moodle). These
monitoring proposals are usually classified into ubiquitous or pervasive learning analytics (ULA or PLA) and
mobile learning analytics (MLA) (Aljohani & Davis, 2012; Shoukry, Göbel, & Steinmetz, 2014) depending on
whether the monitoring collects context-aware data (Facer et al., 2004; Fulantelli, Taibi, & Arrigo, 2013;
Robles, Gonzales-Barahona, & Fernandez-Gonzales, 2011; P. Santos, Hernández-Leo, & Blat, 2014; Vogel,
Spikol, Kurti, & Milrad, 2010) or not (Seol, Sharp, & Kim, 2011). An alternative approach is weSPOT (Miteva,
Nikolova, & Stefanova, 2015), which integrates the monitoring of activities carried out in different spaces,
although in this case the information provided simply reports about the completion of the activities, skipping
any other significant context-aware information (e.g., the location) about the activities conducted in physical
spaces. Another approach is represented by the systems proposed by Tabuenca et al. (2014) or de la Guía et al.
(2016), which by means of NFC tags placed in physical objects, enable the tracking of actions performed,
respectively, in informal or formal learning activities but, again, without integrating such data with other coming
from other formal or blended learning activities and spaces. Similarly, some researchers have specifically
focused on the orchestration of smart classrooms, proposing monitoring systems for such environments
(Martinez-Maldonado, Kay, Yacef, Edbauer, & Dimitriadis, 2013; Martinez-Maldonado et al., 2016; Medina,
Kawsar, Meseguer, & Ochoa, 2014; Slotta, Tissenbaum, & Lui, 2013). In addition, there are several proposals
for different types of dashboards in tablets and mobile devices, capturing and visualizing traces of classroom or
blended learning activities (Verbert et al., 2014), but without providing support to monitor learning situations
involving multiple physical and virtual spaces.
Therefore, there is a scarcity of proposals enabling the monitoring of activities that take place in different
physical and virtual spaces in ubiquitous learning situations (Rodríguez-Triana et al., in press). In this paper we
describe our research to address this challenge. In order to explore it, we propose a monitoring system that
3
builds on an existing system for the orchestration of ubiquitous learning situations including activities in
physical, web and 3DVW spaces, and making use of a variety of existing technologies (Muñoz-Cristóbal, 2015).
The new monitoring system, which is the focus of this paper, provides features for the monitoring of the
different activities carried out in the different spaces. We have created a prototype of the system, and evaluated
its monitoring support in three studies in authentic settings. Throughout the whole research process, we also
extracted a number of design guidelines, which can be useful for researchers and technology developers
proposing solutions to monitor ubiquitous learning environments.
The structure of the document is the following. In the next section, we describe the research methodology
followed, the developed system and the main evaluation happenings. Section 3 summarizes the results of the
evaluation carried out, and Section 4 presents the main conclusions obtained and the design guidelines that we
extracted from the research process.
2. Method
The general research question we posed was: how can technology support the monitoring of ubiquitous learning
situations with teacher orchestration purposes? Due to the fact that it is a research question centered on
technology, and we aimed to propose technological solutions for the educational domain, we considered
appropriate to follow the Systems Development Research Methodology (Nunamaker, Chen, & Purdin, 1990)
with an underlying interpretive perspective (Orlikowski & Baroudi, 1991) for the overall research process (see
Fig. 1).
To address this research question, we first gathered feedback from a group of researchers/teachers, experts in the
ubiquitous learning and learning analytics fields, in order to gain insights of the monitoring needs generated by
the teacher orchestration of ubiquitous learning situations (Section 2.1). Combining this feedback with a
literature review, and the parallel research works that we were conducting regarding orchestration, ubiquitous
learning and learning analytics (Muñoz-Cristóbal, 2015; Rodríguez-Triana, 2014), we defined the architecture
and developed the prototype of a system (whose final version is described in Section 2.2). In order to explore the
research question defined, such prototype was used and enhanced iteratively in three evaluation studies
involving authentic educational settings (Section 2.3). We followed three cycles of the Systems Development
Research Methodology (see Fig. 1), and the knowledge gained in each study led to improve the system for the
subsequent study. Throughout the whole research process, which spanned a year (2013), we also extracted a
certain number of design guidelines for creating monitoring solutions in ubiquitous learning environments (see
Section 4).
2.1. Experts-feedback
During a workshop on orchestration of ubiquitous learning situations framed within a Spanish research project5
in May, 2013, we used an exploratory questionnaire (including open and closed questions) to collect feedback
from the participants. For creating the questionnaire, we formed a team of six educational technology
researchers (including researchers experts in technology or pedagogy), who defined the questionnaire in an
iterative review process. Then, other six teachers/researchers with technological background and experts in
technology-enhanced learning answered the questionnaire. The main topics covered were: whether monitoring is
difficult in ubiquitous learning and why; how the participants used to monitor the students in across spaces
learning situations; how they would improve these monitoring processes. The profile of the teaches/researchers
as well as the main questions of the questionnaire and a summary of the corresponding answers are included in
the Appendix (Tables A.1 and A.2, and Fig. A.1 and A.2).
Regarding the first topic, the experts agreed that ubiquitous learning situations are more difficult to monitor than
traditional learning situations where a single physical or virtual space is involved. They also acknowledged that
the monitoring difficulties are mainly caused by the number of spaces (either physical or web) as well as by the
5 EEE project, see http://eee.gast.it.uc3m.es. Last access April, 2017.
4
combination of physical and virtual spaces. When asked about the data sources they employed, the most
common ones mentioned were the logs obtained from the technological context, the observations made by the
teachers, and the learning outcomes produced by the students. These results are aligned with the systematic
review done by Rodríguez-Triana et al. (in press). Also the students’ feedback, self-assessment, and geolocation
were reported to be used by some experts. Opposite to the current trend in blended learning, where many
research works focus monitoring on a single data source, five out of six experts stated that they combined at
least three different sources to support the monitoring process across spaces, collecting evidence from each
space. For awareness purposes, the experts reported to rely especially on their own observations, enriched
whenever possible by the review of the artifacts generated by the students, the log analyses and the student
feedback. Then, for reflection, the usage of learning artifacts by the experts increased due to the time available
once the learning activity is over.
It is noteworthy that, in the experiences reported by the experts, apart from activity tracking and geolocation,
the data gathering and analyses of the different sources was not automated, requiring teachers to review the
resources generated by the students in the digital spaces, collecting observations during face-to-face activities,
or directly asking the students about the learning process.
Regarding the data analyses, the experts stated that different indicators and visualizations may be obtained to
monitor the learning process inter and intra-spaces. While the experts highlighted 13 and 15 aspects to be taken
into consideration in single physical or virtual spaces respectively, the list increased up to 26 in ubiquitous
learning contexts. The topics addressed by such aspects focused mainly on the actions done by the students
while learning (e.g., quantity and quality of the interactions between students), who and how are the learners
(e.g., engaged or stuck students), and the context where learning takes place (e.g., artifacts used by space, state
of the design activities). The experts agreed in the importance of monitoring different participants’ types of
interaction (participant-participant, participant-artifact and participant-space), the completion of the proposed
tasks and the difficulties that the students found, the individual contributions (or the lack of them), the
collaboration, and the students’ motivation.
2.2. A system for monitoring ubiquitous learning environments
The complexity of technology-enhaced learning orchestration can be observed by the numerous works in the
literature aiming to support teachers in this endeavor. Moreover, when this orchestration happens in ubiquitous
learning environments is even more challenging due to the conceptual complexity of the designs and the
infrastructure required in order to support the different orchestration tasks across spaces and tools. Over the last
few years, we have been exploring in parallel the orchestration of ubiquitous learning situations (Muñoz-
Cristóbal, 2015), and the design-aware monitoring of blended learning situations (Rodríguez-Triana, 2014). For
the former issue, a relevant system has been proposed, GLUEPS-AR (Muñoz-Cristóbal et al., 2015), to support
teachers in multiple aspects of orchestration of ubiquitous learning situations. GLUEPS-AR allows teachers to
deploy learning designs, which may have been created in different authoring tools, into ubiquitous learning
environments composed by VLEs (web space), AR apps (physical space) and VGs (3DVW space). GLUEPS-
AR also enables teachers to adapt the learning design during the enactment of the learning situations. Previous
evaluations of GLUEPS-AR have shown that the main limitation in its orchestration support was related to
monitoring (Muñoz-Cristóbal et al., 2014). Although GLUEPS-AR provided the teachers with a user interface to
consult the artifacts created by the students, the teachers reported that a dashboard with summarized information
of key indicators would help them to understand what happened during the enactment of the learning situation.
Additionally, in the other research line regarding the monitoring of blended learning, two relevant systems were
proposed, GLUE!-CAS (Rodríguez-Triana et al., 2011) and GLIMPSE (Rodríguez-Triana, Martínez-Monés,
Asensio-Pérez, & Dimitriadis, 2013), aimed at, respectively, data gathering and integration as well as analysis
and monitoring of technology-enhanced learning situations. GLUE!-CAS and GLIMPSE are able to collect data
from heterogeneous sources (web-based blended learning environments and participants’ feedback), and to
provide teachers with monitoring reports structured according to the learning designs initially defined (i.e., the
structure of learning activities, groups of students, and learning tools and resources). However, this approach is
focused on blended learning, without taking into consideration learning situations happening in other non-web
spaces, like the physical space or 3DVWs.
5
Interestingly, the two approaches complement very well, since each one could cover the main monitoring
limitations of the other. Furthermore, both approaches share a same technological architectural philosophy,
since they both are based on the well-known adapter pattern of software engineering (Gamma, Helm, Johnson,
& Vlissides, 1995; Monday, 2003) in order to ease the integration of multiple technologies and facilitate the
adoption of the approaches. Therefore, we can easily abstract both approaches and combine them in a single
system that incorporates all the previous yet complementary monitoring features. For designing the new
monitoring system, we have followed the conceptual model proposed by Martinez-Maldonado et al. (2013) for
orchestration systems on ubiquitous and pervasive learning environments. This model proposes the
implementation of interconnection, regulation, and awareness mechanisms by means of a central orchestration
server. We also took design decisions based, as mentioned before, on a literature review (see Section 1), on the
feedback received by experts teachers/researchers (see Section 2.1), on our previous and parallel experiences in
ubiquitous learning (Muñoz-Cristóbal, 2015) and learning analytics (Rodríguez-Triana, 2014) fields, as well as
on the results of the evaluation studies and corresponding guidelines (see Sections 3 and 4) that helped us
develop the final version described here. The main design decisions taken are the following:
(Dd 1) The system supports the monitoring of activities that involve the use of multiple existing
VLEs (in web spaces), AR clients (in physical spaces) and/or VGs (in 3DVW spaces), by means of
their integration through adapters with a central manager. This design decision defines the scope of the
learning situations supported by the proposed system: those involving existing VLEs, web tools, AR
apps and/or VGs.
(Dd 2) The system provides teachers with design-structured monitoring information. This design
decision is based on the need for providing participants with easily interpretable, actionable and
pedagogically grounded data.
(Dd 3) The system provides monitoring information to teachers by means of a web dashboard, as well
as to teachers and students by means of students’ avatars in AR clients and VGs. This decision
provides the stakeholders with different visualization options, according to the types of the spaces in
which the learning situations take place.
(Dd 4) The system monitors different participants’ interactions: participant-artifact, participant-space
and participant-participant interactions. This decision enables in-depth awareness of the different
actions of the participants.
(Dd 5) The system monitors the completion of tasks, which allows the teachers to be aware of the
state of the learning situation.
(Dd 6) The system monitors the students’ contributions and lack thereof. This decision is intended to
make the teacher aware of the work of the students and able to detect problems.
(Dd 7) The system monitors the location of the participants, which is especially relevant in ubiquitous
learning situations in which the participants can be scattered across spaces.
Fig. 2 shows the proposed architecture of the final system. As the predecessor system, GLUEPS-AR, the system
provides support to multiple aspects of orchestration, enabling teachers to deploy their learning designs, which
can have been created with a variety of authoring tools, into enactment settings that can be composed by
multiple web VLEs, AR apps and VGs (see Fig. 3). Teachers can also manage and adapt the ubiquitous learning
situations at run-time through a user interface. Different virtual artifacts (e.g., Web 2.0 tool instances such as
Google Docs) can be accessed from any of the spaces, due to the capability of the proposed system to integrate
multiple artifact-providers or Tool Mediators (systems able to integrate a set of artifact-providers, e.g., GLUE!
or IMS-LTI6
(Alario-Hoyos & Wilson, 2010)).
The integration of multiple types of tools is achieved by means of the well known adapter pattern (Gamma et al.,
1995; Monday, 2003), including an intermediate software layer (an adapter) that integrates each tool with a
central manager. The adapters standardize the operations of the manager over the different types of tools
integrated, so that the manager always uses the same set of operations defined in a contract. The adapters adapt
these operations to the native operations and APIs of the integrated tools.
6 https://www.imsglobal.org/activity/learning-tools-interoperability. Last access April, 2017.
6
The main change with respect to GLUEPS-AR is the inclusion of architectural elements (marked in red dashed
lines in Fig. 2) for supporting the new monitoring features. Therefore, the upgraded system is able to collect (by
means of the monitoring modules of the adapters) and integrate (by means of the monitoring manager)
interaction data from the multiple technologies used in the different spaces (Dd 1). The monitoring manager
contract defines REST resources and operations to communicate with the adapters of the different enactment
tools (VLEs, AR apps, VGs and virtual artifacts). The information is gathered by the adapters from the tools
depending on the features of the corresponding tool (e.g., following a push or pull approach). This way, the
system collects interaction information about the students’ accesses to the different tools and resources and the
location of the students (Dd 4, Dd 7). Such data is sent to the monitoring manager, where it is integrated and can
be subsequently visualized, at run-time and/or after the enactment, in a dashboard and/or using the enactment
technologies (e.g., representing the location of the students in VLEs, AR apps and/or VGs by means of an
avatar) (Dd 3). In order to visualize the information in the enactment technologies, the adapters can also retrieve
data from the monitoring manager. For example, for visualizing in a VG the run-time location of the students in
a physical space, firstly, the monitoring manager collects (by means of the mobile AR adapter) the run-time
location of the students, and secondly, the VG adapter retrieves such information from the monitoring manager.
In the case of the dashboard, the system structures the information according to the learning design (Dd 2),
indicating activities and groups of students, and provides summarized information of the students’ accesses (and
lack of them) to learning artifacts (Dd 6). Also, the dashboard can provide indirect information regarding
potential problems in the completion of tasks, or in collaboration, due to the absence of the expected interactions
of students with artifacts (Dd 4, Dd 5). Fig. 4 illustrates the different monitoring possibilities offered by the
system.
We have developed a prototype of the proposed system in order to explore its monitoring support. The prototype
supports the monitoring of activities carried out with two VLEs (Moodle and Mediawiki), four augmented
reality apps (Grubert, Langlotz, & Grasset, 2011) (Junaio7, Layar8, Mixare9 and any common QR code reader)
and two VGs (Google Earth and Google Street View10
). Also, the prototype supports the monitoring of the
GLUE! Tool Mediator (Alario-Hoyos et al., 2013), thus collecting information of operations over the multiple
types of web artifacts provided by GLUE!. Finally, the prototype provides run-time visualization of the
students’ location by means of avatars shown in the AR apps and the VGs supported, and post-hoc visualization
by means of a web-based dashboard which generates reports structured according to the learning design. We
used Java for developing manager and adapters, Javascript for the user interface, and MySQL and text files for
the internal repository.
2.3 Evaluation
We have followed the Evaluand-oriented Responsive Evaluation Model (EREM) (Jorrín-Abellán & Stake,
2009) as a framework for the evaluation. The EREM relies on a responsive evaluation approach (Stake, 2004),
and it is conceived as an evaluation model for a wide range of ubiquitous collaborative learning scenarios. This
evaluation method is framed within the interpretive research perspective (Orlikowski & Baroudi, 1991), that
does not pursue statistically significant results or generalizations, rather aiming at a deeper understanding of the
particularity and the richness of the phenomena under study (Guba, 1981), in this case provided by the use of the
proposed system in three authentic cases. During the evaluation, we used a profuse set of data gathering
techniques, which are described in detail in Table 1.
To analyze and interpret the data collected, we conducted a data reduction process (Miles & Huberman, 1994).
We defined an issue as the main conceptual organizer of the evaluation process (see Fig. 5): How does the
proposed system support the participants in the affordable monitoring of ubiquitous learning situations? We
divided such issue into three more concrete topics to help us understand the different dimensions within the
7 https://en.wikipedia.org/wiki/Junaio. Last access April, 2017.
8 https://www.layar.com. Last access April, 2017.
9 http://www.mixare.org. Last access April, 2017.
10 https://www.google.com/maps/streetview. Last access April, 2017.
7
issue: the support provided for monitoring the ubiquitous learning situations (topic 1), the affordability for
monitoring using the proposed system (topic 2) (i.e., if the system can be used for monitoring with a reasonable
effort) and the alignment of the monitoring provided by the proposed system with the teachers practice (topic 3).
Each topic was explored with a number of informative questions.
According to the usual criteria to assure the quality and rigor in interpretive research (Guba, 1981), we have
followed several strategies to increase the credibility, transferability, dependability and confirmability of our
research: prolonged engagement during months of work with the different teachers and persistent observation in
the field; member checking, by receiving feedback from the teachers about results and interpretations;
integration of the thorough collaborative observation reports in a single portfolio, thus enabling a thick
description of the phenomenon under scrutiny, reported in detail to the whole evaluation team; peer review
within the evaluation team to avoid bias; and triangulation of data sources and researchers to crosscheck data
and interpretations.
Fig. 6 illustrates the evaluation process, divided into happenings (evaluation events). The figure also represents
the different data gathering techniques applied in each happening, along with the labels used to refer to them
throughout the text. The evaluation consisted basically of three authentic studies, and feedback gathered from
the teachers after the studies. Three teachers were involved in the evaluation: a pre-service teacher in the first
study (H1 in Fig. 6), and two university teachers who participated in the second and the third studies (H3 and
H5 respectively in Fig. 6). At the end of the first study, feedback was gathered from the pre-service teacher by
means of a questionnaire, an interview, and with an email in which he reflected about the monitoring
capabilities of the system used (H2). In the case of the university teachers, as they both were involved in the
second and third studies, we gathered a preliminary feedback after the second study by recording an evaluation
meeting that occurred just after the enactment (H4), in which the teachers reflected about the learning situation
carried out. In addition, we collected final feedback from them after the third study by means of a questionnaire
and interviews (H6). In the feedback gathered from the teachers, we also included many of the questions used in
the questionnaire of the experts-feedback (see Section 2.1). Thus, we complemented the initial information
obtained from the experts with information gathered in an iterative process involving authentic learning
situations. In the following sections we describe in detail the three evaluation studies.
2.3.1. Study 1: Orientate!
Orientate! was a ubiquitous learning situation carried out by a pre-service teacher in his practicum. It was
conducted with a class with eighteen students of around twelve years old, in a course on Physical Education
belonging to the official curriculum of a primary school. The situation was composed of five sessions taking
place in different physical and web spaces: the classroom (see Fig. 7, left), the school’s playground, a nearby
park, and a wiki-based VLE. Many technologies were used by the participants, such as an interactive
whiteboard, netbooks, tablets, Web 2.0 tools and the Junaio mobile AR app. The objective of the learning
situation was to help develop orienteering skills in the children. During the activities, the pre-service teacher and
the students created different virtual artifacts, which were afterwards accessed from a different space from
where they were created. For instance, they created geolocated quizzes using Google Docs in the VLE at the
classroom, which were accessed later on at specific locations in the park using the Junaio AR app.
In this study, the proposed system supported the pre-service teacher in different orchestration aspects, such as in
deploying the learning situation into the enactment setting, in managing the learning activities, or in adapting
them when facing emerging events. Regarding monitoring, the system collected data from the different
technologies used in the different spaces (Junaio, Web 2.0 tools, wiki-based VLE) by means of the adapters.
These monitoring data were processed by the system’s manager and stored in the internal repository. Both the
proposed system’s user interface and the wiki-based VLE served as a control panel for the pre-service teacher,
since he could view and access, during and after the learning activities, the artifacts generated by the students. In
addition, after the end of the activities, the pre-service teacher reviewed the actions conducted by the students by
means of a report produced by the web dashboard (see Fig. 7, right). The report provided information about how
the learning design unfolded, such as the number of accesses of the different groups of students to the different
learning artifacts in each activity. The report did not provide context-aware information, since the prototype did
not triangulate the information generated by the different sources. Thus, the pre-service teacher needed to access
8
the system’s user interface, or the wiki, and consult the artifacts created by the students if he wanted, for
instance, to be aware of the location where an artifact had been generated. Other context-aware interaction data
was stored in the internal repository, but not provided to the pre-service teacher.
2.3.2. Study 2: Game of Blazons
Game of blazons was a ubiquitous learning situation involving physical and web spaces, which was carried out
by two university teachers (the main teacher of the course and an assistant) and forty seven undergraduate
students of a course on Physical Education in the Natural Environment for pre-service teachers. The learning
situation took place in a medieval village, together with other related learning situations conducted during a
weekend in the village and its surroundings. The situation was aimed at helping students acquire and/or
consolidate different skills and knowledge of the subject (orienteering, hiking, history, culture and environment,
etc.), as well as to be able to prepare and carry out physical education activities with children in a natural
environment. The students were organized into groups and they had to find, using orienteering skills and a paper
map, several stone blazons (i.e., coats of arms) chiseled in houses of the village. As Fig. 8 (left) shows, close to
each blazon, they had to use an AR app (Junaio or a QR code reader) in a mobile device to access Web 2.0 tools
(geolocated or linked to QR codes/AR markers) containing learning resources and instructions of different
activities to be performed (quizzes, challenges, geocaching11
activities, etc.). Before and after the session in the
village, other blended activities were conducted in the classroom and online, with the help of the Moodle VLE.
As in the Study 1, the proposed system supported teachers in different aspects of orchestration, and with regard
to monitoring, data was collected by the different adapters, processed by the manager, and stored in the internal
repository. In addition to participant-artifact interaction data, in this case the system also collected periodically
information about the students, including their location in physical spaces (participant-space interaction).
Moreover, we extended the prototype, and the participant-space interaction information was aggregated and sent
back from the manager to the AR adapters, in order to use the mobile AR apps to visualize continuously the
location of the participants, providing run-time user-awareness using AR. During Game of Blazons, the students
used Junaio to access AR learning resources, while the teachers could see the location of the different groups of
students by means of avatars in Junaio (see Fig. 8, right).
2.3.3. Study 3: 3D Mirrored Campus
This study relied on a ubiquitous learning activity involving web, physical and 3DVW spaces, in the frame of a
session devoted to review the topics addressed in the same course of the Study 2. The same two teachers
participated, together with forty eight students of the class. The students performed the activity taking turns, in
groups of six students (while a group was conducting the activity, the rest of the students were carrying out
other different activities). The activity was complemented with pre- and post-tasks using Moodle. The objective
of the activity was to help students assess and reinforce their orienteering skills and the rest of competencies on
ubiquitous learning that they had acquired during the course, while showing them other technological
possibilities. The six students in each group had to split into two subgroups. One of the sub-groups had to walk
outdoors around the campus following whatever route they wanted, carrying a tablet with the Junaio AR app
active (see Fig. 9, left). The other sub-group, in a classroom, could follow the path of their learning partners,
represented as an avatar, in the 3D view of the Google Earth VG (see Fig. 9, right). The students in the
classroom had to draw on an orienteering paper map the path that the other group was following. When the
group with the tablet returned to the classroom, they also had to draw their followed route on a paper map, and
compare it with the map drawn by their colleagues. Afterwards, they changed roles and repeated the activity.
The main differences of this case with the other two described above are the inclusion of a new kind of space (a
3DVW) thanks to the extension of the prototype to support the integration of VGs, and the fact that the
awareness of the participants’ actions was provided to the students, in order to promote self-regulated and
collaborative learning, as well as to improve the students’ orienteering skills and their motivation. As in the
11 https://www.geocaching.com. Last access April, 2017.
9
previous study, participant-artifact and participant-space interaction data was sent from the different spaces by
the adapters to the manager, processed by the manager and stored in the internal repository. Also, participant-
space information was sent back periodically from the manager to the adapters, in order to represent
continuously the location of the students by means of avatars in the VG. This way, the outdoor location in the
physical space of a group of students using Junaio (Fig. 9, left) was represented indoors at run-time by means of
an avatar in the Google Earth 3D view of the campus (Fig. 9, right).
3. Results
This section describes the main results obtained during the evaluation process, organized according to the data
reduction schema shown in Fig. 5. Table 2 summarizes the main evaluation results as well as illustrates the
triangulation process carried out. Throughout the text, the data sources that support the different assertions are
indicated with labels (see Fig. 6). A selection of excerpts of the data sources is included in the Appendix.
3.1. Topic 1: Ubiquitous monitoring
Topic 1 deals with the support provided by the proposed system to the monitoring of ubiquitous learning
situations. The results of the evaluation suggest that the proposed system enabled the students the run-time
monitoring - by means of a 3DVW - of activities conducted in physical spaces, and enabled the teachers -
by means of web tools, augmented reality and a 3DVW - the run-time and post-hoc monitoring of
learning activities conducted in web, physical and 3DVW spaces. However, the results showed also that
the system could be improved to provide additional information related to the students’ collaboration.
Regarding the support provided by the proposed system to the collection and visualization of the students’
actions, evidence gathered in the different studies shows that the proposed system enabled the data collection
from the different spaces supported: web (by using VLEs and web tools), physical (by using the Junaio AR
browser), and 3DVWs (by using the Google Earth VG) (see e.g., [Db3] in Table A.3 of Appendix). Moreover,
the proposed system enabled the visualization of students’ actions by means of user interfaces in the three
spaces: web in Study 1 (by using the web dashboard), physical in Study 2 (with avatars in the Junaio mobile AR
browser) and 3DVW in Study 3 (with avatars in the 3D view of Google Earth). Fig. 7 (right), Fig. 8 (right) and
Fig. 9 (right) show these monitoring interfaces [Obs1, Obs2, Obs4].
With respect to the time frame supported by the proposed system (i.e., the system’s support to run-time and
post-hoc monitoring), the evaluation showed that even though the proposed system enabled the run-time
monitoring of activities conducted in web spaces by means of the system’s user interface (see [Art1] in Table
A.3), the teachers did not use it at run-time. Only in Study 1, the pre-service teacher acknowledged that the
wiki-based VLE allowed him to be aware of the students’ actions while working in the classroom (see [Int1] in
Table A.3). The system also enabled the run-time monitoring of the students’ location during activities
conducted in outdoor physical spaces. This was a necessity detected in Study 1 and implemented through the
AR and VG user-awareness features of the system for Studies 2 and 3, respectively (see, e.g., Fig. 8 and Fig. 9
[Obs2, Obs4]). The proposed system also enabled teachers the post-hoc monitoring, after the end of the
activities conducted in web and physical spaces, by means of the web user interface, the web-based VLE (both
of them compiled the learning artifacts generated/used by the students) and the web dashboard (with
summarized information) (see Fig. 7 [Obs1] and [Quest1-3] in Table A.3). The system also collected and stored
information regarding the time and the location of the students’ actions, although it was not provided to the
teacher post-hoc.
In the presented studies, the types of interactions to be monitored as well as the need for specific indicators
related to collaboration were conditioned by the pedagogical approach and the space where the learning
activities happened. Regarding the types of interaction monitored by the proposed system (participant-artifact,
participant-participant, and participant-space interactions, see Section 2.2 and Fig. 5), the web user interface of
the proposed system enabled the monitoring of the artifacts generated/used by the students (see [Art1] in Table
A.3), and the web dashboard enabled the monitoring of the total number of accesses of the group of students to
the learning artifacts (participant-artifact interaction) (see Fig. 7 [Obs1]). Also the AR and VG user-awareness
10
features enabled the monitoring of the students’ location by means of avatars (participant-space interaction) (see
Fig. 8 and Fig. 9 [Obs2, Obs4]). Other information regarding the students’ interactions (e.g., time and location
of interactions with artifacts) was also collected and stored in log files, but not provided to the users in any of
the monitoring interfaces. A main limitation detected related to the types of interaction was the lack of
monitoring of participant-participant interactions, which needed to be deduced indirectly (by the interactions
with artifacts, or location changes). Similarly, there was a lack of monitoring information about the students’
collaboration processes. In this case, although the web dashboard is able to provide indirect collaboration
information, by indicating the accesses of the members of a group to a shared artifact (e.g., a text document), in
the cases studied, the students accessed the different tools using a group user name (a teachers’ decision to speed
up the authentication process). Therefore, the dashboard monitoring collaboration feature was not used and the
proposed system enabled teachers to monitor only the outcomes of collaborative learning (e.g., times a group
accessed a learning artifact, learning products generated, etc.) (see, e.g., [Art2] and [Int1] in Table A.3).
Finally, with respect to the actors able to carry out monitoring tasks with the proposed system, the three studies
showed that the system was able to provide monitoring features to both teachers and students, since the teachers
used the system to monitor the students’ actions in the three studies, and the students monitored the location of
their teammates during Study 3. In this case, the monitoring information provided to the students was the
location of other students in physical spaces (see [Obs4] in Table A.3). Table A.3 shows some selected excerpts
of evidence gathered during the evaluation, which can illustrate the mentioned findings.
3.2. Topic 2: Affordability
This topic deals with the affordability for using the proposed system to monitor ubiquitous learning situations.
During the evaluation, the proposed system showed to be easy to use for teachers and students for
monitoring (see [Art2] and [Obs4]A in Table A.4 of Appendix). Only some terms appeared difficult to be
understood at the beginning by the teachers, such as the term “warning” in the web dashboard. Therefore, the
user interfaces of the proposed system could improve their clarity. Also, evidence gathered in the evaluation
suggests that the use of the proposed system for post-hoc monitoring was affordable for the teachers, and
its use for run-time monitoring was affordable for the students (see [Art2], [Obs4]A, [Int1] in Table A.4).
However, its use for run-time monitoring was not affordable for the teachers due to the little available time,
and to the multiple tasks and events a teacher needs to be aware during the enactment, specially in activities
conducted in physical spaces (see [Int2] in Table A.4). The design of the AR/VG run-time user-awareness
feature was focused on simplicity, providing only information about the run-time location of the students. It was
quite clear from what we detected in Study 1 that, even though such awareness feature was necessary, it would
be difficult for the teachers to use it because they are usually overwhelmed with the actual teaching tasks.
Therefore, and despite our focus on simplicity, the teachers did not monitor the position of the students
continuously with the user-awareness feature during Studies 2 and 3. This can be regarded mainly to the time-
demanding learning situations, and to the fact that the AR and VG monitoring features had been developed
shortly before. All in all, the teachers used such features in some few specific moments during the enactment of
Studies 2 and 3. In Study 3 a technological problem arose in the VG user-awareness feature. It was the first
usage of this feature in a real setting, and the prototype did not support more than one user represented
simultaneously in the VG (see [Obs4]B in Table A.4). The teachers had conceived three members of the six-
students group carrying individual tablets outdoor, and the rest of the team monitoring them indoor with the 3D
view of Google Earth. Finally, the activity was conducted with the outdoor subgroup carrying a single tablet,
and the rest of the team monitoring their learning partners indoor using desktop computers. Although this was
not part of the evaluation itself, it is interesting to underline that later on we solved these concurrency problems
and we tested the prototype with eleven simultaneous real users and simulating more than a hundred concurrent
users. Table A.4 shows a selection of excerpts of evidence that illustrate these findings.
3.3. Topic 3: Alignment
Topic 3 deals with the alignment of the monitoring provided by the proposed system with the teachers’ practice
and their beliefs. The evidence gathered during the evaluation suggests that the monitoring features of the
proposed system were useful for the teachers and aligned with their practice. The system helped the
11
teachers achieve their learning goals and evaluate the students (see [Quest1-3] in Table A.5 of Appendix),
and the information provided by the system was assessed as pedagogically meaningful by the teachers (see
[Int3]A in Table A.5). The main teacher of the Studies 2 and 3 identified the AR user-awareness feature as one
of the most interesting findings of the learning situations. He considered that it could be very relevant, for safety
reasons, in many learning activities performed with children. He also described other possible usages in
different learning situations, such as promoting collaboration in physical spaces (e.g., a more experienced
student could help a learning partner) (see e.g., [Obs3] in Table A.5).
The evaluation also helped identify features that the system should include to improve its alignment with
the teachers practice. Some of them are the monitoring of the evolution of learning artifacts (e.g., by capturing
the different versions throughout time of artifacts created and/or modified by students), and the inclusion of the
field-observations of participants (e.g., teachers and researchers). In addition, the pre-service teacher of Study 1
stated that he would have liked to be able to access the dashboard information at run-time during the enactment
so that he could be aware of what students were actually doing. He also highlighted as a limitation the absence
of location information in the dashboard. Moreover, the assistant teacher of Studies 2 and 3 asserted that the
system could improve the aid provided to teachers for evaluating by allowing them to define an initial
categorization of evaluation criteria, and a subsequent association of the students’ actions and artifacts created
with the evaluation criteria. Finally, the main limitation of the proposed system detected during the evaluation
was the lack of a tracking feature in which they could observe in a map, both at run-time and after the
enactment, not just the students’ run-time location, but also the whole paths followed by them, including
information such as the time and location of the different actions performed (e.g., the interaction with artifacts)
(see e.g., [Int2] and [Int3]B in Table A.5). As mentioned in Section 3.1, most of this information is already
collected and stored in the internal registry although it is not currently provided to the users. As an alternative,
the teachers used during Study 2, in parallel with the use of the proposed system, another mobile app
(RunKeeper12
), which the students had activated during the activities, to track the followed path (see e.g.,
[Int3]B in Table A.5). Therefore, the proposed system could be enhanced by allowing the integration of other
types of context-aware applications beyond those currently considered. Table A.5 shows a selection of excerpts
of evidence that illustrates the aforementioned findings.
4. Conclusions
The evidence gathered in the research process suggests that the monitoring system created supported the
participants in the affordable monitoring of ubiquitous learning situations that involved web, AR-enabled
physical, and 3DVW spaces. The three authentic settings where the system was evaluated enabled us to assess
some interesting and innovative characteristics of the proposal. In addition to providing monitoring support in
ubiquitous learning situations involving web, physical and 3DVWs spaces, the proposed system also provided
ubiquitous monitoring support, enabling monitoring in web, physical and 3DVW spaces, by means of,
respectively, a web dashboard, an AR app and a 3D VG. This not only increases the monitoring possibilities of
the proposal, but it can also enrich its educational usage, enabling teachers to adapt the monitoring approach to
their pedagogical ideas, or even to use monitoring as a didactic resource in their learning situations, as in Study
3. The use by the students of the system’s monitoring features was another finding of the evaluation, since it
showcased how the proposal provided monitoring support to both teachers (in Studies 1 and 2) and learners (in
Study 3). It is also relevant to underline its three different monitoring options: the user interface, where a teacher
can access (at run-time and after the enactment) all the artifacts generated by the students; the dashboard, where
relevant information is summarized and organized according to the learning design; and the very same
enactment technologies supporting the learning situation, which provide run-time user-awareness by means of
avatars. The different monitoring options of the proposal provided a flexible adaptation to the needs of very
different learning situations. This design-structured and context-aware monitoring support in ubiquitous learning
environments, providing awareness to teachers and students about the students’ performance in different
physical and virtual spaces, surpasses the capabilities offered by other approaches in the literature. Such
approaches usually focus on a single space, do not integrate the monitoring of the students’ actions in different
12 https://runkeeper.com. Last access April, 2017.
12
spaces, nor relate the information provided to the users with the learning design structure or the context of the
learning activities.
The evaluation results also suggest that the system was easy to use for teachers and students, and its monitoring
features pedagogically meaningful, aligned with the teachers' practice, and helping the teachers to achieve their
learning goals and to evaluate the students. The monitoring features gave the teachers a unified quick view of
the students work and some potential problems in the dashboard, as well as a global picture of the students’
location by means of avatars in AR apps and VGs. This information helped the teachers to have a glimpse of
potential problems regarding the location of the students or the lack of access to relevant learning resources.
Also, visualizing the location of their learning partners helped the students to develop orientation skills.
Besides the positive findings, the reported studies have been useful to identify challenges that need further
exploration. These challenges address both run-time and post-hoc support. Regarding synchronous support, an
important line of research is related to the design of tools that provide teachers with monitoring capabilities they
are able to handle at run-time, since in the three cases studied the teachers were overwhelmed during the
enactment due to the limited time available, and the monitoring tools could distract instead of helping them.
Regarding this matter, it is interesting that the teachers demanded run-time monitoring features, but at the same
time they acknowledged as complex the use of any monitoring facilities during the enactment, especially in
physical spaces. This poses a relevant challenge to explore. Another demand identified in the cases was the need
for a tracking facility able to integrate the location information with other meaningful elements of the learning
situation, such as events’ timestamp, participant-artifact interactions, or artifacts generated/accessed, that could
help teachers and students review and reflect on it. Also, additional information to the one provided by the
proposed monitoring system was identified as relevant, such as data related to participant-participant
interactions and the motivation and problems of the students during the activities.
It is noteworthy that the usage of well-known software engineering design principles and standards (e.g., adapter
pattern, REST) allowed us to implement adapters able to collect data from different data sources and spaces. We
could also follow the same approach for the analysis tools that need to consume the collected data. While in our
studies we implemented ad-hoc analyses and visualizations, using activity tracking standards such as
CALIPER13 or xAPI14
we could get advantage of existing solutions that rely on them.
All these benefits and challenges, together with the rest of results and knowledge gained throughout the research
process give some insights to the general research question that we posed, how can technology support the
monitoring of ubiquitous learning situations with teacher orchestration purposes? Thus, we integrated
monitoring features into a system focused on the orchestration of ubiquitous learning environments, improving
the monitoring support provided by alternative approaches to ubiquitous learning situations. Also, during the
whole research, we have extracted a number of design guidelines, which can be useful for researchers and
technology developers focused on the development of monitoring solutions for ubiquitous learning. Table 3
compiles the design guidelines, which we have organized according to the dimensions identified by Chatti et al.
(2012) in their reference model for learning analytics, so that a set of guidelines can help designers in each
dimension of the reference model.
The first dimension (data and environments) points out the need for taking into account multimodal and
contextualized data, from all the spaces incorporated to the learning situation. Monitoring should respond to the
fact that emerging users or artifacts can appear throughout the learning situation, and also to the fact that not all
the necessary data will be available. In the latter case, alternative data sources must be looked for, or the
monitoring system should make the user aware of the potential lack of reliability of the results. With regard to
the recipients of the analysis (stakeholders), the guidelines stress the fact that different participants may be
involved in the monitoring process, and that this process should be adapted to the needs and capabilities of each
one of them. These capabilities may be dependent not only on the user, but on the particular situation, which
13 http://www.imsglobal.org/activity/caliperram. Last access April, 2017.
14 https://www.adlnet.gov/adl-research/performance-tracking-analysis/experience-api. Last access April, 2017.
13
may pose a high workload. This workload must be taken into account to ensure the affordability of the
monitoring support. The study shows that there may be different purposes (objectives) for the monitoring, which
leads to the need for adapting the support provided to these different goals. Also, several guidelines emerged
regarding how monitoring support should be provided to the users (methods): it is necessary to integrate the data
coming from different sources or spaces, in automatic ways to provide unified visions of the learning
environment; there is also a need for supporting run-time functions in the cases where monitoring is to be
provided while the learning situation is running; it is recommendable to use standards to ensure better
interoperability; it is important to consider scalability, as the number of users and artifacts grow faster when
considering several spaces; finally, being aware of the pedagogical intentions of the teachers may have positive
impact on the effectiveness and efficiency of the monitoring process.
The studies had some limitations that define our immediate future work. We plan to further explore the
combination of the different monitoring features of the system, since each monitoring option was used in a
different learning situation. In addition, some of the interaction-data gathered from the enactment technologies
is not currently included in the visualizations. We need additional research in order to improve the monitoring
features with this information, such as creating the tracking maps demanded by the involved teachers. In fact,
the current version of the monitoring system leaves to the teacher all the responsibility for the interpretation of
the data. We plan to enrich the existing system with more advanced analytical features and test whether more
intelligent ways of support are effective to help teachers orchestrate ubiquitous learning situations. Moreover,
we plan to investigate how to include other sources of information to be able to monitor additional indicators
besides those currently implemented in the monitoring system explored, such as participant-participant
interactions, and students’ motivation. Also, we aim to complement, in future iterations of the research, the
results presented in this paper, with other types of data gathering techniques, such as think-aloud protocols,
brainstorming or workshops, for further exploring the necessities of teachers for monitoring ubiquitous learning
environments. Finally, further research would be necessary to explore the scalability of the approach so that it
could eventually be used in massive educational environments, as well as the possible integration in the
monitoring system of other context-aware external applications and devices, or other types of monitoring
interfaces to visualize the information, such as ambient displays (Hernández-Leo et al., 2012; Medina et al.,
2014).
References
Alario-Hoyos, C., Bote-Lorenzo, M. L., Gómez-Sánchez, E., Asensio-Pérez, J. I., Vega-Gorgojo, G., & Ruiz-
Calleja, A. (2013). GLUE!: An architecture for the integration of external tools in Virtual Learning
Environments. Computers & Education, 60(1), 122-137.
Alario-Hoyos, C., & Wilson, S. (2010, November). Comparison of the main alternatives to the integration of
external tools in different platforms. Paper presented at the International Conference of Education,
Research and Innovation (ICERI 2010), Madrid, Spain.
Aljohani, N. R., & Davis, H. C. (2012, September). Significance of Learning Analytics in Enhancing the Mobile
and Pervasive Learning Environments. Paper presented at the 6th International Conference on Next
Generation Mobile Applications, Services and Technologies (NGMAST 2012), Paris, France.
Chatti, M. A., Dyckhoff, A. L., Schroeder, U., & Thüs, H. (2012). A Reference Model for Learning Analytics.
International Journal of Technology Enhanced Learning, 4(5/6), 318-331.
Dalgarno, B., & Lee, M. J. W. (2010). What are the learning affordances of 3-D virtual environments? British
Journal of Educational Technology, 41(1), 10-32.
de la Guía, E., Camacho, V., Orozco-Barbosa, L., Lujan, V. B., Penichet, V. R. M., & Lozano, M. (2016).
Introducing IoT and Wearable Technologies into Task-Based Language Learning for Young Children.
IEEE Transactions on Learning Technologies, 9(4), 366-378.
Delgado Kloos, C., Hernández-Leo, D., & Asensio-Pérez, J. I. (2012). Technology for Learning across Physical
and Virtual Spaces. Journal of Universal Computer Science, 18(15), 2093-2096.
Dillenbourg, P. (2013). Design for classroom orchestration. Computers & Education, 69, 485-492.
Dyson, L. E., Litchfield, A., Lawrence, E., Raban, R., & Leijdekkers, P. (2009). Advancing the m-learning
research agenda for active, experiential learning: Four case studies. Australasian Journal of
Educational Technology, 25(2), 250-267.
14
Facer, K., Joiner, R., Stanton, D., Reid, J., Hull, R., & Kirks, D. (2004). Savannah: mobile gaming and learning?
Journal of Computer Assisted Learning, 20(6), 399-409.
Fulantelli, G., Taibi, D., & Arrigo, M. (2013, November). A Semantic Approach to Mobile Learning Analytics.
Paper presented at the 1st International Conference on Technological Ecosystem for Enhancing
Multiculturality (TEEM 2013), Salamanca, Spain.
Gamma, E., Helm, R., Johnson, R., & Vlissides, J. (1995). Design patterns: Elements of reusable object-
oriented software. Reading, MA, USA: Addison-Wesley.
Grubert, J., Langlotz, T., & Grasset, R. (2011). Augmented Reality Browser Survey. Technical Report
Retrieved April 2017, from http://www.academia.edu/download/10843267/arbrowser_icg_final.pdf
Guba, E. G. (1981). Criteria for assessing the trustworthiness of naturalistic inquiries. Educational
Communication and Technology Journal, 29(2), 75-91.
Hernández-Leo, D., Nieves, R., Arroyo, E., Rosales, A., Melero, J., & Blat, J. (2012). SOS: Orchestrating
Collaborative Activities across Digital and Physical Spaces Using Wearable Signaling Devices.
Journal of Universal Computer Science, 18(15), 2165-2186.
Jorrín-Abellán, I. M., & Stake, R. E. (2009). Does Ubiquitous Learning Call for Ubiquitous Forms of Formal
Evaluation? An Evaluand Oriented Responsive Evaluation Model. Ubiquitous Learning: An
International Journal, 1(3), 71-82.
Keller, C. (2005). Virtual learning environments: Three implementation perspectives. Learning, Media and
Technology, 30(3), 299-311.
Li, L., Zheng, Y., Ogata, H., & Yano, Y. (2004, September). A framework of ubiquitous learning environment.
Paper presented at the 4th International Conference on Computer and Information Technology (CIT
2004), Wuhan, China.
Long, P. D., & Siemens, G. (2011). Penetrating the Fog: Analytics in Learning and Education. EDUCAUSE
Review, 46(5), 30-32, 34, 36, 38, 40.
Marriott, N., & Goyder, H. (2009). Manual for monitoring and evaluating education partnerships. Paris,
France: International Institute for Educational Planning.
Martinez-Maldonado, R., Dimitriadis, Y., Clayphan, A., Muñoz-Cristóbal, J. A., Judy, K., Prieto, L. P., &
Rodríguez-Triana, M. J. (2013, November). Integrating orchestration of ubiquitous and pervasive
learning environments. Paper presented at the 25th Australian Computer-Human Interaction
Conference: Augmentation, Application, Innovation, Collaboration (OzCHI 2013), Adelaide,
Australia.
Martinez-Maldonado, R., Kay, J., Yacef, K., Edbauer, M.-T., & Dimitriadis, Y. (2013, June). MTClassroom and
MTDashboard: Supporting Analysis of Teacher Attention in an Orchestrated Multi-tabletop
Classroom. Paper presented at the 10th International Conference on Computer Supported Collaborative
Learning (CSCL 2013), Madison, Wisconsin, USA.
Martinez-Maldonado, R., Schneider, B., Charleer, S., Buckingham Shum, S., Klerkx, J., & Duval, E. (2016,
April). Interactive Surfaces and Learning Analytics: Data, Orchestration Aspects, Pedagogical Uses
and Challenges. Paper presented at the 6th International Learning Analytics and Knowledge
Conference (LAK 2016) Edinburgh, United Kingdom.
Medina, E., Kawsar, F., Meseguer, R., & Ochoa, S. F. (2014, June). Situated Micro-displays for Activity-Aware
Systems. Paper presented at the 2nd International Conference on Distributed, Ambient and Pervasive
Interactions (DAPI 2014), Heraklion, Crete, Greece.
Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis. An expanded sourcebook. Thousand Oaks,
California, USA: SAGE Publications Inc.
Milrad, M., Wong, L.-H., Sharples, M., Hwang, G.-J., Looi, C.-K., & Ogata, H. (2013). Seamless learning: an
international perspective on next-generation technology-enhanced learning. In Z. L. Berge & L. Y.
Muilenburg (Eds.), Handbook of Mobile Learning (pp. 95-108). Abingdon: Routledge.
Miteva, D., Nikolova, N., & Stefanova, E. (2015, November). Data analytics support for inquiry based
learning. Paper presented at the International Conference on Interactive Mobile Communication
Technologies and Learning (IMCL 2015), Thessaloniki, Greece.
Monday, P. B. (2003). Exploring the Architecture Adapter Pattern Web Service Patterns: Java Edition. New
York: Apress.
Muñoz-Cristóbal, J. A. (2015). Supporting teacher orchestration of across-spaces learning situations. PhD
Thesis, Universidad de Valladolid, Spain.
15
Muñoz-Cristóbal, J. A., Martínez-Monés, A., Asensio-Pérez, J. I., Villagrá-Sobrino, S. L., Hoyos-Torío, J. E., &
Dimitriadis, Y. (2014). City Ads: Embedding Virtual Worlds and Augmented Reality in Everyday
Educational Practice. Journal of Universal Computer Science, 20(12), 1670-1689.
Muñoz-Cristóbal, J. A., Prieto, L. P., Asensio-Pérez, J. I., Martínez-Monés, A., Jorrín-Abellán, I. M., &
Dimitriadis, Y. (2015). Coming down to Earth: Helping teachers use 3D virtual worlds in across-spaces
learning situations. Educational Technology & Society, 18(1), 13-26.
Nunamaker, J., Jay F., Chen, M., & Purdin, T. D. M. (1990). Systems development in information systems
research. Journal of Management Information Systems, 7(3), 89-106.
Orlikowski, W. J., & Baroudi, J. J. (1991). Studying information technology in organizations: Research
approaches and assumptions. Information Systems Research, 2(1), 1-28.
Prieto, L. P., Dlab, M. H., Gutiérrez, I., Abdulwahed, M., & Balid, W. (2011). Orchestrating technology
enhanced learning: a literature review and a conceptual framework. International Journal of
Technology Enhanced Learning, 3(6), 583-598.
Rakshit, R., & Ogneva-Himmelberger, Y. (2008). Application of Virtual Globes in education. Geography
Compass, 2(6), 1995-2010.
Robles, G., Gonzales-Barahona, J. M., & Fernandez-Gonzales, J. (2011, April). Implementing Gymkhanas with
Android smartphones: A multimedia m-learning game. Paper presented at the Global Engineering
Education Conference (EDUCON 2011), Amman, Jordan.
Rodríguez-Triana, M. J. (2014). Linking scripting & monitoring support in blended CSCL scenarios. PhD
Thesis, Universidad de Valladolid, Spain.
Rodríguez-Triana, M. J., Martínez-Monés, A., & Asensio-Pérez, J. I. (2011). Monitoring Collaboration in
Flexible and Personal Learning Environments. Interaction Design and Architecture (s), 11-12, 51-63.
Rodríguez-Triana, M. J., Martínez-Monés, A., Asensio-Pérez, J. I., & Dimitriadis, Y. (2013). Towards a Script-
aware Monitoring Process of Computer-supported Collaborative Learning Scenarios. International
Journal of Technology Enhanced Learning, 5(2), 151-167.
Rodríguez-Triana, M. J., Prieto, L. P., Vozniuk, A., Boroujeni, M. S., Schwendimann, B. A., Holzer, A., &
Gillet, D. (in press). Monitoring, Awareness and Reflection in Blended Technology Enhanced
Learning: a Systematic Review. International Journal of Technology Enhanced Learning.
Roschelle, J., Dimitriadis, Y., & Hoppe, U. (2013). Classroom orchestration: Synthesis. Computers &
Education, 69, 523-526.
Santos, J. L., Verbert, K., Klerkx, J., Duval, E., Charleer, S., & Ternier, S. (2015). Tracking Data in Open
Learning Environments. Journal of Universal Computer Science, 21(7), 976-996.
Santos, P., Hernández-Leo, D., & Blat, J. (2014). To be or not to be in situ outdoors, and other implications for
design and implementation, in geolocated mobile learning. Pervasive and Mobile Computing, 14, 17-
30.
Seol, S., Sharp, A., & Kim, P. (2011, July). Stanford Mobile Inquiry-based Learning Environment (SMILE):
Using Mobile Phones to Promote Student Inquires in the Elementary Classroom. Paper presented at the
The 2011 World Congress in Computer Science, Computer Engineering, and Applied Computing
(WORLDCOMP 2011), Las Vegas, Nevada, USA.
Shoukry, L., Göbel, S., & Steinmetz, R. (2014, November). Learning Analytics and Serious Games: Trends and
Considerations. Paper presented at the 2014 ACM International Workshop on Serious Games, Orlando,
FL, USA.
Slotta, J. D., Tissenbaum, M., & Lui, M. (2013, April). Orchestrating of Complex Inquiry: Three Roles for
Learning Analytics in a Smart Classroom Infrastructure. Paper presented at the 3rd International
Conference on Learning Analytics and Knowledge (LAK 2013), Leuven, Belgium.
Stake, R. E. (2004). Standards-based and responsive evaluation. Thousand Oaks, CA, USA: SAGE Publications
Inc.
Tabuenca, B., Kalz, M., & Specht, M. (2014, November). Binding Daily Physical Environments to Learning
Activities with Mobile and Sensor Technology. Paper presented at the European Immersive Education
Summit (EiED 2014), Vienna, Austria.
Verbert, K., Govaerts, S., Duval, E., Santos, J. L., Van Assche, F., Parra, G., & Klerkx, J. (2014). Learning
dashboards: an overview and future research opportunities. Personal and Ubiquitous Computing,
18(6), 1499-1514.
16
Vogel, B., Spikol, D., Kurti, A., & Milrad, M. (2010, April). Integrating Mobile, Web and Sensory Technologies
to Support Inquiry-Based Science Learning. Paper presented at the 6th IEEE International Conference
on Wireless, Mobile and Ubiquitous Technologies in Education (WMUTE 2010).
Wecker, C., & Fischer, F. (2007, July). Fading Scripts in Computer-supported Collaborative Learning: The
Role of Distributed Monitoring. Paper presented at the 7th International Conference on Computer
Supported Collaborative Learning (CSCL 2007), Rutgers University, New Brunswick, NJ, USA.
Wu, H.-K., Lee, S. W.-Y., Chang, H.-Y., & Liang, J.-C. (2013). Current status, opportunities and challenges of
augmented reality in education. Computers & Education, 62, 41-49.
Appendix
This appendix describes the profile of the experts researchers/lecturers who answered the exploratory
questionnaire used together with a literature review to obtain initial insights of the technological needs to
monitor ubiquitous learning situations (see Section 2.1). The appendix also includes the main questions of the
questionnaire and a summary of the answers. In addition, this appendix includes different excerpts of evidence
obtained during the evaluation.
Table A.1 Profile of the experts lecturers/researchers who answered the initial exploratory questionnaire
1
PhD ICT. ICT expert. 3 years lecturing in university, using VLEs as a teacher. Mainly lecturing in
classroom and blended learning. 5 years researching Technology-Enhanced Learning (TEL) in VLEs
and blended learning.
2
MSc ICT. ICT expert. 3 years lecturing in university, using VLEs as a teacher. Mainly lecturing in
classroom and blended learning. 3 years researching in learning analytics.
3
PhD ICT. ICT expert. 14 years teaching expertise in university, mainly in classroom and blended
learning. 12 years researching TEL, 2 years researching ubiquitous learning.
4
PhD ICT. ICT expert. 7 years lecturing in university, using VLEs as a teacher. Mainly lecturing in
classroom and blended learning. 5
years researching TEL. Occasional experience teaching and
researching learning activities involving multiple physical and virtual spaces in Secondary Education
and university.
5
MSc ICT. ICT expert. 3 years lecturing in university, using VLEs as a teacher. Mainly lecturing in
classroom and blended learning. 5 years researching in TEL 3 years researching in learning analytics.
6
PhD ICT. ICT expert. 1 year teaching in Secondary Education and 11 years teaching in the university.
10 years teaching in learning situations involving physical and virtual spaces. 8 years researching in
TEL. Occasional work as a researcher in learning situations involving multiple physical and virtual
spaces.
Table A.2 Main open-ended items of the exploratory questionnaire used to gather insights of the monitoring
needs in ubiquitous learning situations
Itemsa
Answers summaryb
1. In the learning situations I conduct that are enacted in
a single physical space (such as a classroom or the school
playground), I am able to know everything that happens
17
2. In the learning situations I conduct that are enacted in
a single virtual space (such as a wiki, Moodle or a web
tool), I am able to know everything that happens
3. In the learning situations I conduct that are enacted in
multiple physical spaces (such as a classroom or the
school playground), I am able to know everything that
happens
4. In the learning situations I conduct that are enacted in
multiple virtual spaces (such as a wiki, Moodle or a web
tool), I am able to know everything that happens
5. In the learning situations I conduct that are enacted in
multiple physical and virtual spaces (such as a classroom
or the school playground, a wiki, Moodle or a web tool),
I am able to know everything that happens
6. In the learning situations I conduct that involve
multiple physical and virtual spaces (such as a classroom
or the school playground, a wiki, Moodle or a web tool),
is more difficult to know what is happening than in
traditional learning situations in a single space (e.g.,
conducted in a classroom or in Moodle)
In case of agreement with the last assertion:
7. Such difficulty is due mainly to the physical spaces
8. Such difficulty is due mainly to the virtual spaces
18
9. Such difficulty is due mainly to the combination of
physical and virtual spaces
10. Such difficulty is due mainly to the number of spaces
11. In activities conducted out of the classroom (e.g., in
the natural environment or in a museum), it is important
to know the physical locations and times where the
students were (e.g., the path they followed)
12. In activities conducted out of the classroom (e.g., in
the natural environment or in a museum) in which
augmented reality is used, it is important to know to what
virtual artifact the students accessed, and when they did
13. In activities conducted out of the classroom (e.g., in
the natural environment or in a museum) in which
augmented reality is used, it is important to know when
and where the students accessed to the augmented reality
app
aItems 1-13: closed items (6-point scale [1 = strongly disagree, 2 = disagree, 3 = somewhat disagree, 4 =
somewhat agree, 5 = agree, 6 = strongly agree] + NA = Don't know/No answer).
bWe summarize the answers as “diverse” when there are multiple different agreeing and disagreeing answers
from the experts for a same item, without a common trend.
19
Fig. A.1 Summary of the main open-ended items of the exploratory questionnaire used to gather insights of the
data sources used by the experts when monitoring ubiquitous learning situations. We summarize the experts’
answers by means of tags between brackets
20
Fig. A.2 Summary of the main open-ended items of the exploratory questionnaire used to gather insights of the
aspects identified by the experts to be monitored in ubiquitous learning situations. The first column contains the
21
list of aspects to be monitored identified by the experts. Then, each aspect is classified per topic according to the
indicator categories suggested in Rodríguez-Triana et al (in press), namely: context, learner, action, social,
content and result related. To provide an overview of the interest of each aspect per space, the table summarizes
the number of experts that identified the aspects as relevant in the different spaces. Finally, the last column
provides the total number times that the experts mentioned each aspect regardless of the specific purpose
Table A.3 Selected excerpts of evidence related to the Topic 1 (ubiquitous monitoring)
Data source
Excerpts
[Db3]
Log generated by the Google Earth adapter (periodic trace log):
#date; app type; event type; courseId; Course name; username; deviceId (anonymized hash);
Ip address (anonymized hash); location
Fri Dec 20 12:21:10 UTC 2013;earth;trace;602066390-425238058;Sesión repaso en campus
(20/12/2013);usuario16;1185211859;156635050;41.662797449778466,-4.706636060615092
[Art1]
The screen-recording videos after each session show how the prototype’s user interface
enables the access to artifacts created by the different groups of students, and to the wiki
course.
[Int1]
“Regarding when there were ongoing activities in web sites, yes [the system allowed me to be
aware of the students’ actions], because I directly, from the teacher view, can manage all the
groups. (...) They start to work in the questions 1, 2, 3, 4 and 5, and if they are making the
question 2, I can access from the teacher view to all the groups, and see the question 1, realize
that it is wrong, and then, go back to the group [in the classroom] and correct them.”
[Quest 1-3]
To the assertion “I consider that the used systems allowed me, after the end of the activities, to
know what the students had done in virtual spaces (such as web tools, wikis Moodle, etc.)”,
two of the three teachers answered 5, “Agree”, and the other one answered 6, Strongly agree”.
To the assertion “I consider that the used systems allowed me, after the end of the activities, to
know what the students had done in physical spaces (such as classroom, school playground,
natural environment, etc.)”, the three teachers answered 5, “Agree”.
[Art2]
[Email from teacher 1 talking about the dashboard:] “I can see that all the groups accessed the
resources we had created for them.”
[Obs4]
“I arrive to the classroom and there is a group [of students working]. I see that a map in
Google Earth is being projected, in which the avatar of one of their teammates, who is located
in front of the Faculty of Education, is moving.”
Table A.4 Selected excerpts of evidence related to the Topic 2 (affordability)
Data source
Excerpts
[Art2]
[Email from teacher 1 talking about the dashboard:] “I think that, with the legend included at
the top, and the distribution of the tables in sessions, groups and activities, the tables are very
clear.”
[Obs4]
A
“Also group 4 is doing it well with the 3D view. (...) They are managing Google Earth without
a hitch. They only ask how to switch from 3D view to bird’s eye view.”
“Group 5 starts doing the activity. A student goes out of the building with the tablet, and the
rest stay in the classroom with Google Earth. They use the 3D view, and they manage it very
easily.”
22
[Int1]
“The main problem was to control them while they were doing the activity. (...) I mean, while
they were doing the activity in the web space and after they have done it in physical spaces is
easier to control them.”
[Int2]
“During the activities, the problem of this kind of content, or subject, is that during the
activities a te
acher has a lot of things in which to think about, to do and to dedicate. (...) I think
that it is complicated due to the subject itself, which forces you to be paying attention to many
other things.”
[Obs4]
B
“The technology has failed in the activity. The [Google Earth’s] map didn’t update and Junaio
crashed in the three ipads.”
Table A.5 Selected excerpts of evidence related to the Topic 3 (alignment)
Data source
Excerpts
[Quest1-3]
To the assertion “I think that the used systems helped me evaluate the students in these
activities”, two teachers answered 5, “Agree”, and the other one answered 4, “Somewhat
agree”, and he specified “I think that I took little advantage of the evaluation [possibilities]
that the system provides.”
[Int3]
A
“(...) During the activities, it is the same that we did with Google Earth. Because if you see
where they are moving, you can analyze with a group what they are doing. With a student
looking at Google Earth. And see their mistakes, and make them to argue such mistakes.”
[Obs3]
[Teacher 2 talking about the AR user-awareness feature] “Impressive, I think it’s impressive.
And impressive for other very interesting usages that I have in mind (...) In addition, the
orienteering is an activity that has … when you design an activity for children, some basic
safety issues have to be considered. (...)”
[Int2]
“And, for instance, in the orienteering subject, to be able to associate with a track, to see the
track itself, to comment it live.”
[Int3]
B
“That the system would make a report with the times (...) between a place and another (...). I
mean: did you get there? Yes. You have scanned that at 15:32. And you get there at 15:40. So,
that the system provides you with the time between one place and another. (...) And it would
be nice if it would make even the map. I mean, to integrate the runkeeper inside.”
23
Figures
Fig. 1 Research process followed, encompassing three cycles of the Systems Development Research
Methodology proposed by Nunamaker et al. (1990)
Fig. 2 Architecture of the proposed system
24
Fig. 3 Ubiquitous learning life cycle supported by the proposed system, illustrated with screenshots of the
different tools involved
25
Fig. 4 Monitoring options provided by the proposed system, illustrated with screenshots
Fig. 5 Schema used for the data reduction process
26
Fig. 6 Evaluation happenings and data gathering techniques used during the evaluation
Fig. 7 Session in the classroom during Study1 [Obs1] (left), and partial view of the web dashboard used in
Study 1 [Obs1] (right). The name of the teams appears in green, meaning that the groups have submitted their
work in the terms previously defined by the teachers and therefore, no warnings are shown by the system
27
Fig. 8 Students at a blazon in a village during Study 2 [Obs2] (left), and students’ avatars showed in the Junaio’s
map view during Study 2 [Obs2] (right)
Fig. 9 A student outdoors in the university campus during Study 3 [Obs4] (left), and student’s avatar in the 3D
view of the Google Earth VG during Study 3 [Obs4] (right). The avatar shows the current location of the student
in the campus
Tables
Table 1 Data gathering techniques and labels used to quote them along the text
Data gathering
technique
Description
Label
Collection of
participant-generated
artifacts
Collection of a diverse set of electronic artifacts generated by teachers and
students. Aimed at registering the use of the proposed system by the
participants, gathering the opinions of the teachers, and complementing the
observations with information of the learning artifacts generated. Types of
data collected include emails with teachers, and learning designs and
products.
[Art]
Logs and database
Logs and database entries with the monitoring events collected from the
[Db]
28
entries
different technologies, to register participants’ actions, and to verify that
data was gathered properly.
Observations
Naturalistic, semi-structured observations to register the actions,
impressions and other emergent issues of teachers and students during the
enactment. Conducted by different experienced observers (five in Study 1,
two in Study 2, and four in Study 3). The data collected were audio/video
recordings, pictures, observation notes and screenshots of mobile devices
and computers.
[Obs]
Questionnaires
Web-based exploratory questionnaire composed of open-ended and closed
items (6-point scale [1 = strongly disagree, 2 = disagree, 3 = somewhat
disagree, 4 = somewhat agree, 5 = agree, 6 = strongly agree] + Don't
know/No answer). Aimed at getting the initial opinions of the teachers
over a wide range of matters before conducting the interview.
[Quest]
Interviews
Semi-structured, face-to-face, one-to-one conversation with the teachers
(recorded and transcribed), to capture their opinions in depth, after an
initial analysis of other data sources (e.g., observation data, questionnaire
answers, etc.).
[Int]
Table 2 Main results of the evaluation process
Topic
Results
Supporting data
Ubiquitous
monitoring
The proposed system enabled the students the run-time monitoring of
activities conducted in physical spaces, and enabled the teachers the run-
time and post-hoc monitoring of learning activities conducted in web,
physical and 3DVW spaces. Such monitoring was provided by means of
web tools, augmented reality and a 3DVW. The main limitation of the
system referred to the monitoring of the students’ collaboration.
Art1-3, Db1-3,
Obs1-2, Obs4,
Quest1-3, Int1,
Int3
Affordability
The proposed system was easy to use for teachers and students, and
affordable for teachers for post-hoc monitoring. However, their use for
run-time monitoring was not affordable for the teachers, due mainly to the
lack of available time, especially during activities in physical spaces.
Art2, Obs3-4,
Quest1-3, Int1-
3
Alignment
The monitoring features of the proposed system were pedagogically
meaningful, aligned with the teachers' practice, and helped them to achieve
their learning goals and to evaluate the students. However, the system
should be improved to include: a run-time dashboard with summarized
information of the students’ actions; post-
hoc information about the
evolution of learning artifacts (e.g., different versions of a document
through time); observations of the teacher; and a tracking feature
representing the path followed by the students and information about their
actions (e.g., times and location of interaction with artifacts).
Art2, Obs1-4,
Quest1-3, Int1-
3
Table 3 Design guidelines for monitoring solutions in ubiquitous learning environments, indicating the sources
from which they emanated in the research: experts’ feedback [EF] or evaluation topics [Topic1] [Topic2]
[Topic3]
Dimensions
Design guidelines
29
Data and
environments: What
kind of data does the
system gather, manage,
and use for the
analysis?
● Monitoring benefits from multimodal data, involving not only logs and
learning artifacts, but also information provided directly by the
participants (e.g., teacher observations or student feedback). [EF] [Topic3]
● When different spaces are involved, consider gathering data from each
space. [EF]
● The learning context is not static. Elements such as the artifacts or users to
be monitored may emerge during the scenario. Automatic detection of
emergent elements is necessary to enable their monitoring. [Topic1]
● In case of absence or shortage of available/accessible data, find alternative
data sources or acknowledge the lack of evidence to avoid unreliable
analyses. [Topic1]
● Data has to be associated with contextual information, to enable the users
access this context through inspectable interfaces (e.g., timestamped
interactions, artifacts, users, etc.) [EF] [Topic 1]
● When the situation includes outdoor activities, consider tracking user and
resource location [EF] [Topic 1]
Stakeholders: Who is
targeted by the
analysis?
● The monitoring support must take into account users' workload and
availability in order to provide actionable data at the moments the user can
handle them. [EF] [Topic2]
● Indicators and visualizations must be compliant with each stakeholder's
tasks and data literacy levels. [Topic1] [Topic2]
Objectives: Why does
the system analyze the
collected data?
● Take into account users' purpose when designing data analyses and
visualization. Monitoring may support different teacher and student
purposes: awareness, assessment, self-regulation, group management, etc.
The information needs of the users change with the purpose, and it is very
important to provide exactly what each actor needs, avoiding overloading
them with unnecessary information. [Topic1] [Topic3]
Methods: How does
the system perform the
analysis of the
collected data
● To get an overall picture of the learning scenario, it is necessary to
integrate data obtained from different sources and spaces. [EF]
●
To increase the efficiency and accuracy of the system and reduce the
monitoring workload of the users, automate the data gathering and
integration as much as possible. [EF]
● Consider that aspects/indicators to be monitored can be intra- or inter-
spaces (specific of a space, or common across-spaces). [EF]
● Focus on meaningfulness and contextualization: take into account the
teachers' pedagogical intentions or assessment criteria to identify relevant
indicators and to provide meaningful information to the participants.
[Topic1] [Topic3]
● The use of well-
known design principles and standards (such as design
patterns) enables interoperability and integration. This is specially
important in ubiquitous learning scenarios, where the use of multiple and
diverse technologies is the rule and not the exception. [Topic1]
●
Focus on affordability: identify which aspects can be checked
synchronously, during the learning activity. Provide run-time analyses and
visualization for these aspects, and leave other aspects for post-hoc,
asynchronous inspection. [Topic1]
● Depending on the scenarios to be monitored, it is necessary to deal with
the scalability in terms of number of students and artifacts regarding both
monitoring infrastructure and visualizations. [Topic2]
30
Captions
Fig. 1 Research process followed, encompassing three cycles of the Systems Development Research
Methodology proposed by Nunamaker et al. (Nunamaker et al., 1990)
Fig. 2 Architecture of the proposed system
Fig. 3 Ubiquitous learning life cycle supported by the proposed system, illustrated with screenshots of the
different tools involved
Fig. 4 Monitoring options provided by the proposed system, illustrated with screenshots
Fig. 5 Schema used for the data reduction process
Fig. 6 Evaluation happenings and data gathering techniques used during the evaluation
Fig. 7 Session in the classroom during Study1 [Obs1] (left), and partial view of the web dashboard used in
Study 1 [Obs1] (right). The name of the teams appears in green, meaning that the groups have submitted their
work in the terms previously defined by the teachers and therefore, no warnings are shown by the system
Fig. 8 Students at a blazon in a village during Study 2 [Obs2] (left), and students’ avatars showed in the Junaio’s
map view during Study 2 [Obs2] (right)
Fig. 9 A student outdoors in the university campus during Study 3 [Obs4] (left), and student’s avatar in the 3D
view of the Google Earth VG during Study 3 [Obs4] (right). The avatar shows the current location of the student
in the campus
Table 1 Data gathering techniques and labels used to quote them along the text
Table 2 Main results of the evaluation process
Table 3 Design guidelines for monitoring solutions in ubiquitous learning environments, indicating the sources
from which they emanated in the research: experts’ feedback [EF] or evaluation topics [Topic1] [Topic2]
[Topic3]