Virtual Museums as Learning Agents

Abstract

Virtual solutions for exhibiting museum collections are no longer a novelty, as such
experiences already exist in the world, but the remote use of museum collections for learning purposes
has so far not been widely used in the educational environment. This article analyzes virtual museum
applications by evaluating them from a learning perspective, including 25 criteria in the evaluation
rubric divided into three groups: (i) Technical performance; (ii) information architecture; and (iii)
educational value. This will enable educators to select the most appropriate material for their
specific learning purpose and to plan the most appropriate learning strategies by organizing training
sessions to acquire knowledge that can be enhanced by museum information and teaching students
digital skills in evaluating information available in the digital environment, analyzing its pros and
cons to teach them how to develop new innovative solutions. The research is carried out from a
phenomenological perspective; to be more precise, virtual museums are analyzed using the principles
of transcendental design and a hermeneutic design is used to interpret the resulting data. A total of
36 applications of virtual museums were analyzed, whereupon the results were compiled using static
data analysis software, while 13 applications were used for the hermeneutic data analysis. The results
suggest that the strength of virtual museums is in information architecture, but less attention is paid
to the educational value of the material, which points to the need to change the principles of virtual
museum design and emphasizes the role of teachers in using virtual museums as learning agents.

Full text

sustainability
Article
Virtual Museums as Learning Agents
Linda Daniela
Faculty of Education, Psychology and Art, Scientific Institute of Pedagogy, University of Latvia, LV1086 Riga,
Latvia; linda.daniela@lu.lv
Received: 19 February 2020; Accepted: 27 March 2020; Published: 30 March 2020


Abstract:
Virtual solutions for exhibiting museum collections are no longer a novelty, as such
experiences already exist in the world, but the remote use of museum collections for learning purposes
has so far not been widely used in the educational environment. This article analyzes virtual museum
applications by evaluating them from a learning perspective, including 25 criteria in the evaluation
rubric divided into three groups: (i) Technical performance; (ii) information architecture; and (iii)
educational value. This will enable educators to select the most appropriate material for their
specific learning purpose and to plan the most appropriate learning strategies by organizing training
sessions to acquire knowledge that can be enhanced by museum information and teaching students
digital skills in evaluating information available in the digital environment, analyzing its pros and
cons to teach them how to develop new innovative solutions. The research is carried out from a
phenomenological perspective; to be more precise, virtual museums are analyzed using the principles
of transcendental design and a hermeneutic design is used to interpret the resulting data. A total of
36 applications of virtual museums were analyzed, whereupon the results were compiled using static
data analysis software, while 13 applications were used for the hermeneutic data analysis. The results
suggest that the strength of virtual museums is in information architecture, but less attention is paid
to the educational value of the material, which points to the need to change the principles of virtual
museum design and emphasizes the role of teachers in using virtual museums as learning agents.
Keywords:
virtual museums; technical performance; information architecture; educational value;
learning agent; learning perspective; evaluation rubric
1. Introduction
Different virtual solutions allow diversification in access to information, learning, and getting
to places you would not otherwise be able to go [
1
]. Virtual solutions are becoming more accessible,
attractive, and immersive and incorporate an element of the avatar that is directly related to parts of
the body and provides a better understanding of the reality seen [
2
], and the individual has the ability
to interact with virtual environment artefacts, build their own understanding, develop their cognitive
processes, and foster their creativity and the development of new innovation. The developed technical
solutions’ graphical design, artefact aesthetics, information architecture, interoperability and also the
possibility of their usage in an educational setting are analyzed.
The research is based on the idea that virtual solutions and smart devices that complement each
other can serve as learning agents, but in order to be fully utilized in the educational environment they
need to be evaluated, and the aim of this study is to evaluate virtual museums from a phenomenological
perspective to understand their potential for educational purposes. In short, the study seeks to answer
the following research question: What is the potential of virtual museums from a learning perspective,
and can they be used as learning agents? In order to achieve the aim and to find the answer to the
research question, the objectives of the research are:
Sustainability 2020,12, 2698; doi:10.3390/su12072698 www.mdpi.com/journal/sustainability

Sustainability 2020,12, 2698 2 of 24
1.
Analyses of mobile virtual museum applications with previously developed and approbated
evaluation rubric [
3
], which is updated for the needs of this research from three perspectives that
synergistically complement each other: (i) Technical performance; (ii) information architecture;
and (iii) educational value, using the principles of transcendental design
2.
Quantitative analyses of the data collected to find out the best apps from an educational perspective
and qualitative data analyses by using hermeneutic design to interpret the resulting data.
-
The author used the developed and validated evaluation rubric [
3
], which was updated to add
more structure and a few more criteria (see Appendix A).
-
Applications were searched for on the App Store using the keywords “virtual museums”, and 56
virtual museum applications were found to be available at the time of the research. After their
initial evaluation, 36 applications were evaluated in accordance with the developed evaluation
rubric and 13 apps were chosen for in-depth analyses.
-
The obtained results were analyzed from a learning perspective, using quantitative and qualitative
data analysis methods.
This paper is structured as follows: Section 2analyzes recent literature on museums as a learning
agent from a learning perspective and the role of information architecture in the development of digital
materials. In Section 3, an explanation of the research design and methods used is given. In Section 4,
the research results and a discussion of these results are presented, Section 5is devoted to the discussion
and in Section 6conclusions based on the data collected during the research are drawn.
2. Virtual Museums as Learning Agents
2.1. Learning in Museums
Along with the development of virtual solutions in different directions, such as:
i)
Where virtual reality (VR) experiences can be enjoyed in specially created VR theatres using the
necessary hardware, software headsets, multi-projected environments, and physical environments
to produce real-life images, sounds, and other sensations [
4
] that are transferred to the VR user
through virtual simulation and the transfer of this experience is mediated through a mobile
device, tablet, console or computer [5];
ii)
Where visual experiences can be complemented by a variety of immersive experiences
(haptic, smell, temperature, etc.);
iii) Where the VR environment is mixed with the real environment—it is also possible to extend their
range of use.
As a result, the use of virtual solutions is expanding, but it is also necessary to broaden the scope
of research to include the use of these different opportunities in education. For example, Garz
ó
n
and Acevedo (2019), in their literature review on the use of augmented reality (AR) in education,
concluded that there is only one article evaluating AR from the perspective of educational sciences [
6
].
Virtual solution capabilities also need to be evaluated before they can be used as a tool for scaffold
learning and the development of metacognition [
7
,
8
], rather than only to entertain and fascinate people.
For museums, virtual solutions are tools to help provide information about the museum that
visitors may need, but this is not currently the focus of research. In the following, virtual museums
will be analyzed from a perspective where they can serve as a tool to overcome the various constraints
faced by museums themselves, such as the need for rooms to display all the artefacts available to them,
the need to attract visitors, and the desire of visitors to interact with museum artefacts as much as
possible. Virtual museums can serve as a tool for learning by entertainment, where learning takes
place through active exploration, collaborating with virtual agents [
9
], or as an interaction with a
virtual narrator that is humanlike and capable of communicating, while using the museum. The use of
different narrators is recommended to give the user of a virtual solution a sense of presence [
10
] where

Sustainability 2020,12, 2698 3 of 24
it is possible to interact with artefacts (smart objects), which are technically simple embedded systems
equipped with sensors, actuators, and networking capabilities [
11
]. The Ars Electronica Center for
Electronic Art in Linz, Austria, which houses a CAVE-projection VR in which users stand between
six walls of a room-sized cube, is considered to be the first museum to use virtual solutions in its
exhibitions [12].
Given that VR experiences with immersion effects are expensive to produce and require human
presence to experience this immersion, museums are increasingly using a variety of other virtual
solutions that connect to museums using personal smart devices and combine virtual-created artefacts
and real environment scenes:
a.
To interest society in museum collections because virtual solutions allow people to explore
other dimensions of information and one of the challenges facing museums is fighting for
visitors’ interest;
b.
To display artefacts and situations that would otherwise not have been displayed, such as
artefacts that no longer exist or historical situations that are no longer experienced;
c.
To preserve and allow access to a variety of exhibitions that an individual can visit virtually
without time and national boundaries, as exhibitions are variable, but preserving them virtually
can preserve contemporary history, thus, contributing to the sustainability of cultural values;
d.
To contribute to the development of an inclusive society, as access to museum experiences in this
way is more accessible, both in terms of physical access (such as people with reduced mobility) and
in terms of cost (for example, getting to a museum in Paris can be quite an expensive adventure).
Nowadays, museums have the task of not only storing and preserving historical values, but also
of using them for learning purposes [
13
]. From the perspective of educational sciences, learning in
museums is one form of situated learning [
14
,
15
] where access to a variety of knowledge is possible,
providing learning opportunities that enhance individuals’ interest in knowledge accumulated in
museums, interacting with virtual artefacts, and enabling them to interact with substance and
concreteness with the museum’s intangible artefacts [
16
]. Such learning can also support culturally
responsive teaching [
17
], and there are initiatives taken to blend learning in the museum and use of
technologies. For example, MyArtSpace project [
18
]. However, despite the advantages of this type
of learning, there are some limitations, since situated learning also involves active interacting with
artefacts in a given environment, but these artefacts must be preserved for the future, which creates a
contradiction in their practical use [
3
]. Thus, there is a need to look for solutions to engage students,
to promote active learning, and to provide it with the available technological affordances. Virtual
solutions and their use for educational purposes have great potential as they can be used as learning
agents [
19
], and agents for preserving cultural values. Their use can also address the conflict between
the principles of active learning used in a situated learning environment and the need to preserve
historical values. State-of-the-art virtual solutions make it possible to create virtual museums and
VR/AR experiences, as well as to provide people with a variety of information to enhance the museum’s
message and empower knowledge and awareness [
20
]. The important place in learning takes different
kind of mobile learning, which is platform-independent, utilizes web-based platforms and incorporate
in learning mobile applications to support learning anywhere and anytime, to move beyond static
content delivery [
21
]. Educational researchers use different terms to name the learning through mobile
devices, but the most appropriate term for the learning in virtual museums which tends to blur the
borders between real space and virtual space seems ‘authentic mobile learning’ introduced by Burden
and Kearney [
22
]. Thus, virtual museums can be considered as learning agents for upward learning
curves, as such solutions can help broaden the spectrum of collaboration with digital materiality and
help visualize and spatialize abstract concepts [
23
]. There are studies that have found that using VR
solutions stimulates learning motivation because it makes learning more interactive [
24
], and makes it
possible to get feedback in the learning process itself if the information is properly constructed [
25
,
26
],
which highlights the importance of information architecture in a digital learning environment. Their

Sustainability 2020,12, 2698 4 of 24
potential is also based on the idea that technical systems can also be agents from a psychological,
sociological and learning perspective [
27
,
28
]. Virtual museums have specific characteristics that
support the idea that they can serve as learning agents because:
i) It is possible to augment analogue reality with digital information;
ii)
Integration takes place in real time and in a coordinated way;
iii)
They facilitate combining different resources: Text, websites, video, audio and 3D;
iv)
They are interactive; and
v)
The involvement of the individual is necessary for the creation of the content [29].
The use of technological solutions as agents in the learning process helps students to analyze,
assimilate, contextualize and synthesize their knowledge, thereby providing new levels of thinking
and also, positively influencing students’ learning motivation by developing a deeper understanding
of specific knowledge [
30
]. In this way, virtual museums are perceived as learning agents that offer
access to information concentrated in museums and restricted artefacts within them and facilitate
situational learning [
15
] or authentic mobile learning [
22
]. In such way, individuals can use their
smart devices to connect to this knowledge, where learning, to a certain extent, can be organized in a
specific environment and under certain circumstances that would not be possible without the use of
virtual solutions.
2.2. Smart Learning Environments
There are articles in the research literature that justify students learning differently nowadays [
31
];
however, the use of virtual museums as learning agents requires the teachers’ seamless presence to
provide students with the appropriate learning experience, thereby becoming a knowledge transmitter,
and thus, forming a multi-agent network as anything can be considered as a learning agent. The role
of the teacher in this network of agents is still very important [
32
], because the virtual solutions
available today cannot completely replace the human pedagogical element; as Garz
ó
n and Acevedo’s
study concluded [
6
], such solutions have medium learning gains and are difficult to use because
teachers do not have the appropriate pedagogical, content and technological competences [
33
]. There
are also studies that conclude that students are not yet ready to learn using such solutions [
29
,
34
].
This demonstrates the need to enhance the baggage of smart pedagogic solutions so that various
technological affordances, including virtual museums, are used as learning agents, as these solutions
not only solve access problems, but are considered to be used for scaffolding learning in authentic
mobile learning space. They are involved in the process of information perception, which facilitates
the memorization of information, and thus, the construction of certain knowledge [
35
], and strengthen
learning and bring joy to learning itself through experiencing physical phenomena in a mixed reality
(MR) environment [36].
Learning using a variety of technological solutions was defined by Spector as smart learning [
37
],
though this type of learning is sometimes defined under the umbrella term “digital learning” [
38
,
39
],
while other authors choose the term “digital agency” to describe the whole spectrum of the digital
environment. The pedagogical principles to be considered, when organizing a technology-enhanced
learning process are called smart pedagogical principles [
40
], and when planning and organizing
technology-enhanced learning these should be considered to have not only fascination effects, but also
learning outcomes.
Another factor that is important when thinking about using virtual museums for learning is
information architecture, which can support the learning flow; conversely, the information flow can
be so complicated that it is inconsistent with the perceptions of the user that its only outcome is a
fascination with the technological affordances, not its use as a learning agent. At present, the Internet
and the information available there is not perceived as a “different and separated world” or domain,
but rather as part of a larger mechanism in which the consumption and retrieval of information take
place in different contexts using multiple physical and physical-digital overlapping and interacting

Sustainability 2020,12, 2698 5 of 24
devices [
41
]. The information architecture has moved away from a single artefact to an interconnected
network, some parts of which may not even be online or even digital at all, and one solution is the
Internet of Things (IoT), which is developing broader information and semantic environment [
42
].
Virtual museums that can be used with personal smart devices can be considered as one example of
IoT that can serve as learning agents.
Lacerda, Lima-Marques and Resmini [
43
], with reference to Resmini and Lacerda [
44
], and Benyon
and Resmini [
45
], define information architecture units as “actor-driven, information-based, semantic
constructs connecting individual touchpoints into (a) transient architecture”, assuming that actors
(virtual museum users) use agents (virtual museums) to cross the boundaries of the information
space that connect the contact points to achieve the desired position in the future, which from the
educational perspective means scaffold learning outcomes and metacognition. This information space
brings together previously unrelated information spaces in various ecosystems that participate in the
wider infosphere [
46
], combining individuals, artefacts, places and events, and act as learning agents.
Lacerda, Lima-Marques and Resmini define 16 information architecture principles that can be divided
into three categories, which are: (i) Architectural principles that take into account the perspective of
information space and artefacts; (ii) human principles that take into account the subjective perspective
of subject/object interaction; and (iii) systemic principles that take into account the perspective between
artefacts, actors and systems in information ecosystems [43].
Fowler emphasizes three stages of learning through VR that can also be taken into account for
virtual museums:
i)
Conceptualization (where the learner learns and interprets different facts, concepts and theories
and receives information);
ii) Construction (where the learner evaluates facts and concepts, applies knowledge in an interactive
way, solves or analyzes problems, tests the use of concepts in new situations, and observes real-life
experiences by building on his/her own knowledge of the experience);
iii)
Discussion (the learner discusses what he/she has learned and his/her own learning) [47].
As can be seen, getting information is indicated right from the first stages of learning so that
the individual can then construct his or her knowledge and apply it in new contexts. If the flow of
information has been inadequate, it also jeopardizes the learner’s advancement to the next stages.
It proves that virtual museums can be used as learning agents if the information provided by the
agent is consistent, constructive and aesthetically and pedagogically correct, which requires that
the information architecture is in synergy with smart pedagogies, where visual, audial and textual
information (and, in some specific cases, immersive information) should be taken into account.
When an image (otherwise perceived as an item of information) is analyzed, cognitive processes are
activated, and this facilitates metacognitive development [
7
]. The accurate mapping of data with
the visualizations used is critical, as disregarding information architecture principles can result in
the loss of relevant information or in exceeding the allowable amount of irrelevant information [
48
],
and this has an impact on the learning process as it can lead to cognitive overload. The virtual museum
application should be simple, usable and aesthetically pleasing as the app needs to work efficiently
and not break down during use [44].
This article focuses on the evaluation of virtual museums from an educational perspective, because
despite the widespread use of virtual solutions, there is still a lack of research focusing on the utility,
challenges, efficiency and parameters of their use [
48
]. There are studies showing that MR (which is
often used as an umbrella term for various VR solutions, including AR solutions) has a positive effect
on student motivation and willingness to engage in learning activities [
6
,
49
,
50
]. This educational
evaluation of various virtual museums is important for a number of reasons:
i)
To explore the potential of using these experiences from a subject-oriented perspective and from
the perspective of motivation and [
50
] cognition sensory development, emotional development
and/or the development of specific skills [
51
,
52
]. In general, this could be defined as the learning

Sustainability 2020,12, 2698 6 of 24
outcome perspective. For example, learning about the events of a historical period, getting into
that environment, or accessing places that would not otherwise be available.
ii)
To identify gaps in the supply of these experiences and to fill them with other pedagogical
activities. For example, evaluating whether a particular virtual museum includes an aspect of
assessing learning outcomes or whether it is appropriate for students with different learning needs.
iii)
To help students navigate through a range of experiences and use these virtual museums as
learning agents. For example, if students have a desire to learn about the architecture of a
historical period or if students need to learn how to deal with hazardous substances, etc. then
virtual solutions can provide them with such opportunities.
iv)
To help teachers scaffold learning to enable the students to become skilled users of technology
and creators of new technological solutions. For example, if a teacher is working with students to
evaluate a museum’s technology solution and look for ways to improve it, such as improving the
layout of artefacts, improving visual graphics, changing the information flow, and so on.
3. Research Design Methodology
The research is carried out from a phenomenological perspective; to be more precise, virtual
museums are analyzed using the principles of transcendental design and a hermeneutic design is
used to interpret the resulting data, which deviates from the descriptive nature with which the
phenomenological approach is most often associated [
53
,
54
]. No individual persons are involved in
the research, nor are the personal data of individuals used.
An evaluation of 36 virtual museums was conducted to conceptualize the existing and potential
capabilities of virtual museums as learning agents (see the full list of evaluated experiences in
Appendix B), using the researcher’s personal smart devices (an iPhone and iPad) to connect to them.
Virtual museums were selected by entering the keywords “virtual museum” in the App Store, and their
free apps were chosen based on the idea that virtual museum visits are accessible to anyone with a
smart device and an internet connection and that no other resources are required. No other selection
criteria were applied according to the country of application, museum theme, or other formal criteria.
A total of 56 applications were selected, but 14 of them were intended solely to inform museum visitors
about the museum’s opening hours, ticket prices, access, exhibitions, and other practical information.
Another four experiences were excluded from the evaluation because their suggested language was
unknown to the author (French, Polish, and Chinese). Two more apps were excluded because, despite
being listed as free, there was a charge for attending each exhibition and space. After an in-depth
evaluation of all criteria, 36 virtual museum applications were selected for analysis.
An evaluation rubric was used to ensure the objectivity of the data analyzed, but there was
still some subjectivity as the researcher’s perspective was used to evaluate particular criteria [
55
].
During the evaluation of the experiences, the educational perspective was the most dominant one as
the researcher represents the field of education, and during the evaluation, the students’ perspective
was kept in mind by using the researcher’s personal experience of working with students of different
age groups. As the basic structure of the evaluation rubric had already been developed in the previous
research phase [
3
], with the result that only the order of the criteria needed to be changed and five new
criteria added, the initial phase of the study took a short period of time. The selected applications were
then evaluated, and the obtained results were quantitatively and qualitatively evaluated, according to
the comments made during the evaluation of each virtual museum application. The selected virtual
museums were analyzed between September 2019 and January 2020. The evaluation rubric was used
to evaluate the virtual museums and contained 25 criteria divided into three groups: (i) Technical
performance (11 criteria); (ii) information architecture (5 criteria); and (iii) educational value (8 criteria),
as well as one criterion for specifying the age groups for which the assessed virtual museum may be
eligible. The boundaries of these groups are not strictly separated (see Figure 1), because all factors
interact and influence information perception processes, thus, affecting the ability to store information
in long-term memory, analyze it and synthesize new knowledge concepts, which is the primary goal

Sustainability 2020,12, 2698 7 of 24
of the educational process. Assuming the virtual museums are used as learning agents, they were
rated in accordance with the developed evaluation rubric, which was developed inductively based
on extensive literature analysis, the personal experience of the author and a pre-approved evaluation
tool [
3
] based on the principles of an analytical rubric [
55
]. Each of the criteria in the rubric is evaluated
on three levels, each of which has its own description (see Appendix A). After evaluating each of the
criteria, the possibility to add comments describing the specific criterion and the chosen rating was
provided. During the quantitative analyses of the results, the levels were indicated by numbers where
1 indicated the lowest level, 2 indicated the medium level, and 3 indicated the highest level of the
criterion. The structure of the evaluation tool was adapted from the work of Stevens and Levi [56].
Sustainability 2020, 12, x FOR PEER REVIEW 7 of 28
value (8 criteria), as well as one criterion for specifying the age groups for which the assessed virtual
museum may be eligible. The boundaries of these groups are not strictly separated (see Figure 1),
because all factors interact and influence information perception processes, thus, affecting the ability
to store information in long-term memory, analyze it and synthesize new knowledge concepts, which
is the primary goal of the educational process. Assuming the virtual museums are used as learning
agents, they were rated in accordance with the developed evaluation rubric, which was developed
inductively based on extensive literature analysis, the personal experience of the author and a pre-
approved evaluation tool [3] based on the principles of an analytical rubric [55]. Each of the criteria
in the rubric is evaluated on three levels, each of which has its own description (see Appendix A).
After evaluating each of the criteria, the possibility to add comments describing the specific criterion
and the chosen rating was provided. During the quantitative analyses of the results, the levels were
indicated by numbers where 1 indicated the lowest level, 2 indicated the medium level, and 3
indicated the highest level of the criterion. The structure of the evaluation tool was adapted from the
work of Stevens and Levi [56].
Figure 1. Structure for the evaluation of virtual museums.
3. Research Results and Analyses
The obtained quantitative results were analyzed using Excel, where the mean values for each
group of criteria and for each virtual museum application were calculated separately. Most virtual
museums are dedicated to the arts (paintings, sculptures, architecture); the next largest group is
history museums (state history, history of war, history of a factory, inventions of Tesla). There were
separate museums for some objects (Nefertiti, Tank Tour), one museum for Dineladi cave and one
museum on the Solar System.
3.1. Quantitative Results of Analyses
The quantitative data analysis, summarizing the averages of all 36 virtual museums analyzed
(see Figure 2), shows that the highest scores are in the criteria that characterize information
architecture, and there are five museums (Civilizations, UMA/Universal Museum of Art, Houghton
Revisited, Nikola Tesla Experience, Tank Tour) with the highest rating (3) in this category. For technical
performance criteria, the highest mean was 2.73, which was for Civilizations, followed with a mean of
2.55 by Pompeii and Houghton Revisited. In contrast, for educational value criteria, the highest mean
was 2.57 for Civilizations, which was followed by Daily Art with a mean of 2.29 and Tank Tour with
Information
architecture (5)
Educational
value (8)
Technical
performance
(11)
Virtual
museums
Figure 1. Structure for the evaluation of virtual museums.
4. Research Results and Analyses
The obtained quantitative results were analyzed using Excel, where the mean values for each
group of criteria and for each virtual museum application were calculated separately. Most virtual
museums are dedicated to the arts (paintings, sculptures, architecture); the next largest group is history
museums (state history, history of war, history of a factory, inventions of Tesla). There were separate
museums for some objects (Nefertiti, Tank Tour), one museum for Dineladi cave and one museum on
the Solar System.
4.1. Quantitative Results of Analyses
The quantitative data analysis, summarizing the averages of all 36 virtual museums analyzed
(see Figure 2), shows that the highest scores are in the criteria that characterize information architecture,
and there are five museums (Civilizations, UMA/Universal Museum of Art, Houghton Revisited, Nikola Tesla
Experience, Tank Tour) with the highest rating (3) in this category. For technical performance criteria,
the highest mean was 2.73, which was for Civilizations, followed with a mean of 2.55 by Pompeii and
Houghton Revisited. In contrast, for educational value criteria, the highest mean was 2.57 for Civilizations,
which was followed by Daily Art with a mean of 2.29 and Tank Tour with 2.14. These indicators
are summarized for all evaluation criteria, where some of the technical criteria received very low
ratings—e.g., Possibility to interact with the narrator (mean 1.06) and Possibilities for people with special

Sustainability 2020,12, 2698 8 of 24
needs (mean 1.14)—thus, reducing the rating of the technical criteria group as a whole, but it is clear that
the lower endpoints are the educational value criteria for using these applications as learning agents.
Sustainability 2020, 12, x FOR PEER REVIEW 8 of 28
2.14. These indicators are summarized for all evaluation criteria, where some of the technical criteria
received very low ratings—e.g., Possibility to interact with the narrator (mean 1.06) and Possibilities for
people with special needs (mean 1.14)—thus, reducing the rating of the technical criteria group as a
whole, but it is clear that the lower endpoints are the educational value criteria for using these
applications as learning agents.
It can also be seen that there are five museums that received the lowest possible point (1)
for
Figure 2. Evaluation results_all virtual museums (mean). (Abreviation EON is company title EON
Reality Inc. – and UMA – Universal Museum of Arts).
The mean scores of all the evaluation criteria were further analyzed to determine which the
highest and lowest values were across all the analyzed virtual museums (see Figure 3). Within the
group of technical performance criteria, the highest results are for Use of the material (mean 2.58) and
The graphics of the material (mean 2.56), while the lowest rates are for Possibility to interact with the
narrator (mean 1.06) and Possibilities for people with special needs (mean 1.14). In the information
architecture group, the highest scores are for The sequence of information flow (mean 2.31) and Structure
of the information provided (mean 2.25), and the lowest scores are for Information about the historical
period of the artefact (mean 2.03), and Information provided during the use of the material (mean 2.11). For
the group of criteria that describe the educational value, the highest scores are for Additional
information is given in audial form (mean 1.72) and Connectivity with other information (mean 1.72), and
the lowest for Elements of gamification (mean 1.14) and Knowledge test (mean 1.17).
0
1
2
3
Chateu de Versailes
Pompietouch
Nefertiti VR by EON
The National Museum of Iraq
Macao Museum
Bosch VR
MagiChapel
Civilisations
Memory Museum
Batle of Batina Memorial museum
UMA
Toumanian museum
Museum of Applied Arts Budapest
Pompeii
Houghton Revisited
Colm Cille
Never built New York
Nicola Tesla experience
Asolo
Haramain exhibition
Bilbo
Museo del San Michele
Viewmuseum
Tank tour
Hermes
Hermitage
Daily Art
Muzeum Archeologiczne
BMW museum
Rijks Museum
Edu Must
Toile de Joy
Museum of Portimao
Dinaledi
Solar System
Museum of Architecture
Technological performance Information architecture Educational value
Figure 2.
Evaluation results_all virtual museums (mean). (Abreviation EON is company title EON
Reality Inc. – and UMA – Universal Museum of Arts).
It can also be seen that there are five museums that received the lowest possible point (1) for
each of the criteria, and as such these museums cannot be analyzed because of their low technical
performance and low educational value.
The mean scores of all the evaluation criteria were further analyzed to determine which the
highest and lowest values were across all the analyzed virtual museums (see Figure 3). Within the
group of technical performance criteria, the highest results are for Use of the material (mean 2.58)
and The graphics of the material (mean 2.56), while the lowest rates are for Possibility to interact with
the narrator (mean 1.06) and Possibilities for people with special needs (mean 1.14). In the information
architecture group, the highest scores are for The sequence of information flow (mean 2.31) and Structure
of the information provided (mean 2.25), and the lowest scores are for Information about the historical period
of the artefact (mean 2.03), and Information provided during the use of the material (mean 2.11). For the
group of criteria that describe the educational value, the highest scores are for Additional information is
given in audial form (mean 1.72) and Connectivity with other information (mean 1.72), and the lowest for
Elements of gamification (mean 1.14) and Knowledge test (mean 1.17).
4.1.1. Technical Performance and Information Architecture
In the following, applications of virtual museums whose mean were higher than 2 points for
technical performance and information architecture criteria were selected, totaling 24 applications
(see Figure 4). It can be seen that the mean values for most applications are higher in information

Sustainability 2020,12, 2698 9 of 24
architecture criteria, and these technological features can be used as a supporting tool to ensure learning
via virtual museums.
Sustainability 2020, 12, x FOR PEER REVIEW 9 of 28
Figure 3. Mean value for all criteria (green: Technical performance⎯average mean 1.96; blue:
Information architecture⎯average mean 2.17; red: Educational value⎯average mean 1.46).
3.1.1. Technical Performance and Information Architecture
In the following, applications of virtual museums whose mean were higher than 2 points for
technical performance and information architecture criteria were selected, totaling 24 applications
(see Figure 4). It can be seen that the mean values for most applications are higher in information
architecture criteria, and these technological features can be used as a supporting tool to ensure
learning via virtual museums.
1
1.2
1.4
1.6
1.8
2
2.2
2.4
2.6
Structure of the material
Use of the material
The graphics of the material
Artifacts are aesthetically manifested
Dimensions of the artifact
Transferability of the artifact
Interactivity of the material
Possibility to interact with the narrator
Possibilities for people with special needs
The risk of cybersickness
Perception of the material
Structure of the information provided
Information provided during the use of the
material
The sequence of information flow
Information about the historical period of
the artifact
Information about the place of origin of the
artifact
Additional information is given in audial
form
Additional information is given in written
form
Connectivity with other information
Knowledge test
Elements of gamification
The material challenges the user to find out
more information
Focus of attention
Mean value for all criteria
Figure 3.
Mean value for all criteria (green: Technical performance—average mean 1.96; blue:
Information architecture—average mean 2.17; red: Educational value—average mean 1.46).
Sustainability 2020, 12, x FOR PEER REVIEW 10 of 28
Figure 4. Technical performance and information architecture (green: Technical performance; blue:
Information architecture).
3.1.2. Educational Value
Analyzing the educational value criteria, it can be concluded that the criterion It is possible to
organize group activities was not technically available in any of the virtual museums, and therefore this
criterion is excluded from further analysis. An in-depth analysis of virtual museum applications was
performed for those apps where the mean score of the criteria exceeds 2 points, totaling 13
applications (see Figure 5).
Two museums with a mean score of more than 2 are not included in the analysis below, however:
The Macao Museum, because all the information that could be read is only available in Chinese, and
the Chateau de Versailles, excluded because a lot of the information is only in French.
Taken together, the results indicate that the scores for all criteria groups are different, and the
mean educational value criteria score is higher than the technical performance score for only one of
the virtual museum applications (Daily Art). This confirms the idea that virtual museum applications
can only be used for educational purposes if a person is motivated to seek additional information or
already has information that may help them to understand the information provided in the
application. Otherwise, teachers’ seamless participation is needed.
1
1.5
2
2.5
3
Chateu de Versailes
Pompietouch
Nefertiti VR by EON
Macao Museum
Bosch VR
MagiChapel
Civilisations
UMA
Museum of Applied Arts Budapest
Pompeii
Houghton Revisited
Colm Cille
Never built New York
Nicola Tesla experience
Bilbo
Tank tour
Daily Art
Muzeum Archeologiczne
BMW museum
Rijks Museum
Edu Must
Toile de Joy
Museum of Portimao
Dinaledi
Technical performance and information architecture
Figure 4.
Technical performance and information architecture (green: Technical performance; blue:
Information architecture).

Sustainability 2020,12, 2698 10 of 24
4.1.2. Educational Value
Analyzing the educational value criteria, it can be concluded that the criterion It is possible to
organize group activities was not technically available in any of the virtual museums, and therefore this
criterion is excluded from further analysis. An in-depth analysis of virtual museum applications was
performed for those apps where the mean score of the criteria exceeds 2 points, totaling 13 applications
(see Figure 5).
Sustainability 2020, 12, x FOR PEER REVIEW 11 of 28
Figure 5. Evaluation results of the best virtual museums (mean).
3.2. Hermeneutic Data Analysis
The following is a hermeneutic analysis of comments made during the evaluation of virtual
museum applications. This section analyzes only the 13 applications which were selected for deeper
analyses in the previous stage because of the mean scores (see Figure 5) that were included in the in-
depth analysis in the tables below: Table 1 summarizes the analysis of technical criteria, Table 2
summarizes the information architecture criteria analysis, and Table 3 includes information on the
analysis of educational value criteria. The tables are divided into three columns: The first lists the
criteria, the second shows the average mean value for each criterion, and the third shows the results
of the hermeneutic analysis.
Table 1. Evaluation of technical solutions.
Technical solutions
Criteria Avg.
mean Comments
Structure of the
material 2.39 Most of the virtual museums’ material has a clear structure. For
most of the museums, two to eight languages are available
Use of the
material 2.58
It is easy to understand how to use the material; in some cases, there
is an introduction to the material and an explanation of how to use
it. Some of the materials lack instructions but the user can
understand the principles because the material is developed with
intuitive logic, but there are also cases where the logic of the
material is unclear (Colm Cille).
There are also cases where there is no option to go back to the
previous stage and the only option is to close the app (Colm Cille).
In the case of the UMA (Universal Museum of Art), there are several
exhibitions (9) and several tours (3). There are also 6 upcoming
exhibitions.
The graphics of
the material 2.56
For most of the museums, the graphics were well developed, and in
cases where photos were used, they were high quality.
There were two apps where the graphics can be changed. In one
case (Pompeii) there were three levels—one where the user can see
how the building looks now, another where the user can see how
0
0.5
1
1.5
2
2.5
3
Bosch VR
Civilisations
UMA
Houghton
Revisited
Colm Cille
Nicola Tesla
experience
Tank tour
Daily Art
Toile de Joy
Dinaledi
Evaluation results of the best virtual museums
Technological performance Information architecture Educational value
Figure 5. Evaluation results of the best virtual museums (mean).
Two museums with a mean score of more than 2 are not included in the analysis below, however:
The Macao Museum, because all the information that could be read is only available in Chinese, and the
Chateau de Versailles, excluded because a lot of the information is only in French.
Taken together, the results indicate that the scores for all criteria groups are different, and the
mean educational value criteria score is higher than the technical performance score for only one of
the virtual museum applications (Daily Art). This confirms the idea that virtual museum applications
can only be used for educational purposes if a person is motivated to seek additional information or
already has information that may help them to understand the information provided in the application.
Otherwise, teachers’ seamless participation is needed.
4.2. Hermeneutic Data Analysis
The following is a hermeneutic analysis of comments made during the evaluation of virtual
museum applications. This section analyzes only the 13 applications which were selected for deeper
analyses in the previous stage because of the mean scores (see Figure 5) that were included in the
in-depth analysis in the tables below: Table 1summarizes the analysis of technical criteria, Table 2
summarizes the information architecture criteria analysis, and Table 3includes information on the
analysis of educational value criteria. The tables are divided into three columns: The first lists the
criteria, the second shows the average mean value for each criterion, and the third shows the results of
the hermeneutic analysis.
Although some of the applications were excluded from the in-depth analysis, a positive example
would certainly be Petite Galerie, in which it is possible to obtain information for people with visual or
hearing impairments when the user first opens the app, the graphics offer the option of an alternative
method of information provision for people with visual/hearing impairments. Unfortunately, we could
not investigate this further because the app was only available in French.

Sustainability 2020,12, 2698 11 of 24
Table 1. Evaluation of technical solutions.
Technical Solutions
Criteria Avg. Mean Comments
Structure of the material 2.39 Most of the virtual museums’ material has a clear structure. For most of the museums, two to eight languages are available
Use of the material 2.58
It is easy to understand how to use the material; in some cases, there is an introduction to the material and an explanation of
how to use it. Some of the materials lack instructions but the user can understand the principles because the material is
developed with intuitive logic, but there are also cases where the logic of the material is unclear (Colm Cille).
There are also cases where there is no option to go back to the previous stage and the only option is to close the app (Colm Cille).
In the case of the UMA (Universal Museum of Art), there are several exhibitions (9) and several tours (3). There are also 6
upcoming exhibitions.
The graphics of the
material 2.56
For most of the museums, the graphics were well developed, and in cases where photos were used, they were high quality.
There were two apps where the graphics can be changed. In one case (Pompeii) there were three levels—one where the user can
see how the building looks now, another where the user can see how the building looked in ancient times, and a third where
the user can add people into the landscape. Another app was Bilbo (not included in the deeper analyses), where the user could
change the time of day and see how the buildings look in daylight and how they look during the night.
For Tank Tour, all the visual elements are in high quality except the historical videos, which are in a lower quality as they were
taken in a particular time.
Unfortunately there were quite a few museums calling themselves a virtual museum where a 360
◦
camera was used as the only
technical solution (for example Macao Museum, Battle of Batina)
Artifacts are aesthetically
manifested 2.44
In the case of the Magi Chapel, all the information given is about one fresco. The artefacts (art pieces) are exhibited as a whole
and can be zoomed in on to see the details at the Museum of Applied Arts Budapest. In Pompeii, the artefacts are located to give an
impression of how it looked in ancient times compared to how it looks now. There is also the possibility to switch on a
function where you can see the people in these buildings. This helps to understand what kind of purposes the building was
used for and also gives an impression of how people were dressed. In Colm Cille, the developers have kept the artefacts as they
are now, but for some artefacts, about which it is said that they are lost, high quality solutions are used to show how they may
have looked. In Dinaledi, the VR experience in the cave is very realistic and bones are located in the places where they were
found by scientists.
In UMA, all the exhibitions and materials are well represented and it seems like a real exhibition.
Daily Art shows one masterpiece per day with information about it and links to other information.
Dimensions of the artifact 1.56
When experiencing the material of the Magi Chapel, it is easy to move around in 360◦and see all the details from different
dimensions. The size of this Magi Chapel was based on the original architecture and dimensions but it is not possible to move
things around.
Possibilities to see artefacts from different dimensions were only available for a few museums (Civilisations, Colm Cille, Magi
Chapel). It was possible to see art pieces’ smaller details by zooming in on the details.
In Colm Cille it is possible to read about the majority of the artefacts, but there are some that can only be seen in the hall and are
not reachable.

Sustainability 2020,12, 2698 12 of 24
Table 1. Cont.
Technical Solutions
Criteria Avg. Mean Comments
Transferability of the
artifact 1.36
Just a few virtual museums provided the opportunity to transverse the artefacts in detail. For some of the museums it is not
necessary to transform the art pieces if the details can be zoomed in on, but historical museums or technical museums would
benefit from such an option.
Artefacts are easy to transverse in the Civilisations app and some of the artefacts are transversable in Colm Cille. In the Pompeii
app it was possible to transverse part of the building to see how it looked in ancient times and part of it could be left in the way
it looks now to see the contrast.
Interactivity of the
material 1.56
Only a few materials have been developed to ensure the users’ interaction with the museum. In most cases, the only
interactivity is to choose the sequence of the information and the possibility to zoom in on the picture. In Pompeii Touch, the
material can be transferred and the visitor can look at the artefact as it looks now, the scene can be changed to see how it
looked in times when people lived there, and people can be added to the scene. Some artefacts are movable.
For Magi Chapel, the user can choose a point on the fresco they want to listen and learn about, and the background will change
to the one they pick. This is the only form of interactivity between this product and the experiencer.
In Pompeii it is possible to interact with the material by adding people and changing the landscape from that of ancient times to
nowadays and vice versa by using the erase function.
In Colm Cille it is possible to transverse artefacts and interact with the scene.
Possibility to interact with
the narrator 1.06
Of all 36 museums, only two had an option to listen to the narrator. In the Nikola Tesla Experience it was Tesla himself who told
the stories, and in Dinaledi although the narrator was not visible, information was given in audial form. There were no apps
where the user could interact with the narrator.
Possibilities for people
with special needs 1.14
There were only a few museums where some special features were provided for people with special needs.
In Civilisations and Tank Tour it was possible to change the size of the text. For some apps, there was a possibility to get the
same information in written and audial form.
No other specific features were developed.
The risk of cybersickness 2.53
Cybersickness, which can be an issue in virtual reality, was not a problem in most of the virtual museums because there was an
option to see the material without using VR glasses. Only one app (Dinaledi) can possibly cause problems with cybersickness
because it takes quite a long time to go through the red gates, which can be challenging for those who do not like flashing
images. It is easier to go through these gates on a screen but it is disturbing with VR glasses and there is no reason to make the
journey so long.
Perception of the material 2.44
Most of the museums are well developed and their information is easy to perceive. It is only in Colm Cille, due to the terms
used in the information provided about the artefacts, that it is sometimes hard to grasp what is meant. These terms will largely
only be understandable for those who are interested in a particular subject or particular historical period, unless additional
information is given before the use of the app.

Sustainability 2020,12, 2698 13 of 24
Table 2. Evaluation of information architecture.
Information Architecture
Criteria Avg. Mean Comments
Structure of the
information provided 2.25
Most museums have well-structured information, but some apps’ academic vocabulary or terminology might be hard to
understand for some of their audience: For example, in the MagiChapel app, Renaissance, Palazzo Medici Riccardi, and
frescoes. Some of the terminology contains Italian terms. Hence, this educational material requires historial knowledge on the
part of its audience. In Colm Cille most of the information is structured very well but some objects that are included in the app
can be seen but are not reachable and information about them is not given.
Information provided
during the use of the
material
2.11
Information about artefacts or art pieces is provided for most of the museums, but in some cases it is very short (Pompeii Touch).
In Magi Chapel, some parts are hard to understand without prior knowledge of Italian history, such as the Medici family during
the Renaissance, painting materials, light sources and art historical terms like portraits, perspective and symbolism. This
material requires knowledge of Italian history, art history, the Italian language, and religion. However, the audience will have
different levels of understanding. The experiencers who are well-educated in history will get a deep and full understanding of
this material. On the contrary, experiencers who lack historical knowledge will miss some of the information provided by the
background audio. The perfect audience for this material is someone who wants to learn about Italian history and the artistic
history of the Magi Chapel.
Colm Cille has the same problem, where prior knowledge is needed to understand the text and the context.
The sequence of
information flow 2.31
In most museums the user can change the sequence of the information. In some cases, when smaller details are zoomed in on,
it is possible to understand their symbolical meaning (Museum of Applied Arts Budapest). In Dinaledi there is the possibility of
seeing the bones in a different sequence, but it is not possible to change the sequence of information provided by the narrator.
Information about the
historical period of the
artifact
2.03
For some museums a small amount of information is given, but some background knowledge is also needed to interpret the
given information (Pompeii Touch, Museum of Applied Arts Budapest, Pompeii, Houghton Revisited, Nikola Tesla Experience). In Magi
Chapel, the user can take a closer look at the frescoes and listen to the journey of the Magi. The date, name and art historical
information are all given by the background audio. The experiencer has the opportunity to dig into Italian history and the
history of Renaissance families, but some background knowledge is needed. Otherwise it is hard to understand the context.
Information about the
place of origin of the
artifact
2.14
Some of the museums are missing information on the place of origin of the artefact (Pompeii Touch, Pompeii). In Civilisations, all
the information about the origin of the artefact is given clearly as written text. For example, it states that the mummy was
discovered in Egypt.
The best experience is provided in Daily Art where some brief information on the art pieces, the artist, the place of its
exhibition, and its context is given, and links to additional information are also provided.

Sustainability 2020,12, 2698 14 of 24
Table 3. Evaluation of educational value.
Educational Value
Criteria Avg. Mean Comments
Additional information is
given in audial form 1.72
Only 8 museums provided audial information but it was well prepared. If there was an option to choose among different languages,
the researcher tried those languages that were known to them, and in these cases the information provided was very similar.
Unfortunately it was possible only in a few cases to go deeper and find more information, and it was not possible to find out more
information in audial form at any museum.
Additional information is
given in written form 1.58
In most cases, written information was provided. In some cases, it was very short and not sufficient to understand the context (Pompeii
Touch, Museum of Applied Arts Budapest, Pompeii, Houghton Revisited, Toile de Joy). There were only a few museums where, besides the
information that was provided in the app, links to other information were added (Tank Tour, Daily Art, Dinaledi) or other sources could
be opened in the same app (Magi Chapel). In most cases it was written information.
Connectivity with other
information 1.72
Only a few museums (Pompeii Touch, Civilisations, Colm Cille, Tank Tour) provided some connectivity with other information provided
in the museum. For example, Magi Chapel gives information not only about the Magi Chapel but also the history of Florence, the
religious meaning of each symbol, the atmosphere of the Renaissance and each member of the Medici family. Each part of the
information is smoothly connected with the others. This material contains many historical facts.
In Dinaledi an option is provided whereby the user can open a link to the museum itself and find much more information.
Knowledge test 1.17
Virtual museums usually do not provide the option of a knowledge test.
This test is only provided in Colm Cille and if your answers are not correct, you can try again. Different questions are asked each time
to ensure that knowledge is being tested.
Elements of gamification 1.14
Although principles of gamification are often used for educational purposes in virtual museums, these elements were very seldom
used; one element is included in Magi Chapel—to explore the historical jewel of the Magi Chapel. In Civilisations, golden points can be
collected for each artifact. In Pompeii, the option to ‘erase’ a building to see how it looked in ancient times gives some game-like feeling.
The material challenges
the user to find out more
information
1.31
Most of the virtual museums do not challenge the user to find out more information unless they are personally interested in a topic.
Some elements of a challenge can be found in Civilisations as the user must pick artefacts by moving the globe. In the Nikola Tesla
Experience, the way that the narrator (Tesla) tells the user the information can be assumed to be a challenging element. In Tank Tour, the
possibility to climb in the tank and the historical videos can be seen as motivators to find out more information. Daily Art is great
because the sentences in the text are formulated in a way to invite the user to search for more information. UMA material is so well
prepared that it challenges the user to find out more about particular art pieces and their artists.
Focus of attention 1.61
There are very rare cases where specific features are developed to support the focus of attention to introduce some information. Some
elements can be found when a figure is zoomed in on or an artefact can be transverse, but from the educational perspective, the focus
of attention is supported in following virtual museums—Pompeii Touch (to find particular buildings), Magi Chapel (to see all the details
and follow the information), Civilisations (to move the globe and find artefacts and information about them), Pompeii (because there are
three levels to how the buildings can be seen), Colm Cille (because it is possible to move the artefacts), the Nikola Tesla Experience
(because the narrator (Tesla) speaks in an encouraging manner), Tank Tour (because of the different forms of information provided),
and UMA (because the exhibitions are created in an interesting way and the information provided is written in a way that encourages
the user to read it all).

Sustainability 2020,12, 2698 15 of 24
5. Discussion
In his work, Fowler [
47
] emphasizes three stages of learning through VR that can also be taken
into account for virtual museums: (i) Conceptualization; (ii) construction, and (iii) discussion. It is
in the light of these stages of learning and from the perspective of the analyzed virtual museums,
it can be concluded that in the conceptualization phase, where a student should learn and interpret
facts, concepts and theories, it is possible to learn facts through a virtual museum’s information and to
receive additional information about it. But the teacher must check to ensure that the learning path of
the students has been successfully selected and the development of cognitive processes is underway if
they are to acquire the skills to interpret facts and concepts. Only in one of the analyzed applications
(Colm Cille) was there a built-in knowledge assessment test, but it was more focused on testing memory
processes for factual knowledge; the opportunity to interpret the facts presented in the applications
was not included in any of them.
The second stage is where the learner has to construct new knowledge or competences using
learned facts and concepts in new contexts and compare different experiences. In short, none of the
virtual museum applications analyzed provided such an opportunity, so the support of a teacher
is needed here too to provide this knowledge, which can be done through discussions and group
work, including on information gained from virtual museums and information provided by teachers.
This means that the third stage can then be reached through the pedagogical work of a teacher.
Considering learning in museums as a form of active learning, the potential for virtual museums
exists. This was seen in virtual museum applications that showed specific locations (Dinaledi, Pompeii,
Toile de Joy). Even better if the application provided the possibility to interact with artefacts (Civilisations),
interact with the narrator (Nikola Tesla Experience) or explore art pieces in detail (MagicChapel, UMA).
This can both ensure that individual reflection is triggered, while interacting with the nature and
concreteness of the museum’s intangible artefacts [
16
], and support culturally responsive teaching [
17
],
thus, scaffolding the learning of new knowledge. Thus, virtual museums can be considered as learning
agents [
28
] for upward learning curves, as such solutions can help broaden the spectrum of collaboration
with digital materiality and help visualize and spatialize abstract concepts [
23
], and involve learners
as co-designers and co-researchers [
57
]. Museums can be used as situated learning spaces [
14
,
15
],
but during the evaluation process it was found out that only a few aspects of situated learning were
introduced in apps of virtual museums; therefore, this requires the presence of a teacher seamlessly
manipulating various tools [
14
], sometimes a virtual museum to foster learners’ interest, but sometimes
additional information provided by the teacher to ensure that the information provided in the virtual
museum has the proper context, because in order to appreciate a work of art it is also necessary to
know about forms of artistic expression and the symbolic meaning of the artwork. As it suggested
by Aguayo, Eames, and Cochrane, learning through mobile applications should be authentic to the
context, integrated within and across learning areas and scaffolded for a clear learning pathway [
20
],
and there should be interconnection between the virtual world and physical action [
58
] to ensure
authentic mobile learning [
22
]. Such synergy between different pieces of information will help to
develop advanced knowledge and contribute to the development of metacognitive processes.
The teacher needs to ascertain the level of knowledge of the learner, so a knowledge assessment is
needed to ensure that the learner has understood the information received and is able to interpret and
apply it in other contexts. If this is not technologically possible through a virtual museum, then the
teacher should provide it.
The reception of feedback supports the learning process, but the downside is that the provision of
feedback was not possible in any of the virtual museums, so it is necessary to get this feedback from
the teacher instead, which indicates that beyond the information architecture criteria being evaluated
here, it is also necessary for virtual museums to include some features that allow feedback to be given.
There are only a few applications where it is possible to change the way information is received,
and it can be concluded that this type of learning contributes to some extent to inclusive education
from the perspective of accessing information in museums through an application, such as providing

Sustainability 2020,12, 2698 16 of 24
people with mobility issues or disabilities (reduced mobility, socio-economic situation) with access to
this knowledge. However, in most of the evaluated virtual museum applications, it is not possible to
change the way information is received because it is not possible to change the language, to switch
between written and audial information (or vice versa), or to turn on sign language. This confirms that
not all principles of inclusive education have been taken into account, and these must be considered
both from an information architecture point of view and from a technological perspective. If a teacher
wants to use one of the applications to make the learning process more interesting, he or she must
evaluate the specific needs of the learners to ensure that all learners have equal access to information.
Although there are articles that have analyzed the role of virtual narrators [
7
] or the user of the
virtual solution having a sense of presence [
10
], unfortunately only two of the applications analyzed
used a narrator and only one was human-like, which shows that this potential is not yet being fulfilled.
The results of this analysis of virtual museums echo those of Quinn [
32
], in that the role of the
teacher in this network of agents is still very important because the virtual solutions available today
cannot completely replace human pedagogical work, as also concluded by Garzón and Acevedo [6].
Of the 36 applications analyzed, 24 had a mean of over 2 (out of 3) in their technical performance,
and information architecture criteria and were structured sequentially and logically to ensure that the
information included was easy to read. The artefacts in these 24 applications were also aesthetically
pleasing to ensure that people will most likely use these apps. Unfortunately, there were 12 apps which
were of low quality, and this means that a third of them are not really usable.
Meanwhile, in terms of the educational value criteria, only six of the evaluated virtual museum
applications’ mean is 2 or higher, which means that while most freely available applications can
be considered as learning agents that stimulate interest, supplement existing knowledge with new
knowledge and provide a change in forms of learning, they cannot be considered as full-fledged
learning agents that completely replace the teacher’s work.
It should be noted that this study has limitations, the first of which involves selecting only those
virtual museum applications that are available to the user free of charge. The results may be different
for applications for which there is a charge. The second limitation is that despite the fact that the
evaluation of virtual museums was carried out using a structured evaluation, it still involves an aspect
of subjectivity, since the evaluation was carried out by a person with a certain amount of technological
experience, pedagogical principles and information architecture conditions, as the valuations are largely
due to the evaluator’s current knowledge. This limitation can be mitigated, albeit not completely
eliminated, by the continuation of research activities, the expansion of the research base and the
development of deeper details of the evaluation section.
Despite these limitations, the study can be considered useful for:
i)
Educators who want to expand their learning environment by offering virtual museum solutions;
ii)
Learning designers, who will want to evaluate their developed virtual museums from the three
perspectives on offer;
iii)
museum staffwho want the materials they offer to be used not only for entertainment, but also
for educational purposes.
6. Conclusions
Only one application (Colm Cille) had built-in knowledge assessment test focused on testing
memory processes for factual knowledge; the opportunity to interpret the facts presented in the
applications was not included in any of them.
None of the virtual museum applications analyzed provided an opportunity to construct new
knowledge or competences using learned facts and concepts in new contexts.
Virtual museums can be considered as learning agents for upward learning curves, as such
solutions can help broaden the spectrum of collaboration with digital materiality and help visualize
and spatialize abstract concepts, but the material should be prepared to take in mind learning logic.

Sustainability 2020,12, 2698 17 of 24
In summary, all the information analyzed indicates that, if virtual museums are intended to
promote knowledge and cognitive development, the material alone can only be used without the
teacher’s seamless presence if the individual is very interested in a particular subject and has prior
knowledge in order to understand the information, or if he or she is highly motivated to seek additional
information on his or her own. Otherwise, the active involvement of a teacher is needed to use such
technological solutions to interest students in a subject. Unfortunately, the range of applications is
not yet considered sufficient to provide learning in museums, where they also have a significant
educational value, but they can be used as learning agents if the teacher evaluates the application
in advance to understand what pedagogical activities are needed both before and after using the
virtual museum.
Funding: This research received no external funding.
Acknowledgments:
These findings provide a basis for further research into virtual museums, such as their use in
educational settings to assess their impact on learning outcomes, pupils’ metacognitive load, learning preferences,
learning outcomes, and so on.
Conflicts of Interest: The author declares no conflict of interest.
Appendix A
Criterions for the evaluation of VR/AR experiences in museums.
All the criteria should be evaluated by ticking the correct level according to the evaluator’s opinion.
Only in the last row, where the evaluator’s opinion on the age group should be given, can more than
one answer be chosen.
The list of criteria includes three evaluative aspects:
1. Technological solutions
2. Information architecture
3. Educational value
Structure of the
material
The content is well
structured
The content is structured
but the structure is not
logical
The content is
fragmented and not
structured according to
some kind of logic
  
Comments can be added here:
Use of the material It is easy to understand
how to use the material
It is not very
understandable how to
use the material
It is hard to understand
how the material should
be used
  
Comments can be added here:
The graphics of the
material
The graphical elements
are well structured and
visualizations are in high
quality
The graphical elements
are randomly structured
and visualizations could
be of a better quality
The graphical elements
are poorly structured
and visualizations are in
low quality
  

Sustainability 2020,12, 2698 18 of 24
Comments can be added here:
Artifacts are
aesthetically
manifested
All artifacts are
represented by the
harmonious coming
together of the parts; the
juxtaposition of the
structural elements in
space represents the
identity of the design
through its composition
or configuration
The main artifacts are
aesthetically manifested,
but there are some
problems with the
juxtaposition of the
structural elements or
with their composition or
configuration
There are quite a lot of
problems with the
juxtaposition of the
structural elements in
space and their
composition or
configuration
  
Comments can be added here:
Dimensions of the
artifact
It is easy to move the
artifact and see it from
different dimensions
(outside and inside)
It is possible to move the
artifact and see it from
different outside
dimensions
It is possible to see the
artifact only from a few
outside dimensions
  
Comments can be added here:
Transferability of the
artifact
It is easy to transverse
the artifact in smaller
details
It is possible to
transverse the artifact in
a few details
It is not possible to
transverse the artifact in
detail
  
Comments can be added here:
Interactivity of the
material
There are different forms
of interactivity
There is some
interactivity
People cannot interact
with the material
  
Comments can be added here:
Possibility to interact
with the narrator
There is a possibility to
talk with the narrator
(visible person)
There is a narrator
(visible person) who
provides information but
there is no possibility to
interact with him/her
There is no narrator
present as a visible
person

Comments can be added here:
Possibilities for people
with special needs
The material is prepared
in a way that people with
diverse special needs can
use it, and it is clearly
indicated how to use it
The material is prepared
in a way that people with
some specific special
needs can use it, but it is
not available for all
groups of special needs,
and it is indicated which
groups can use it
The material is prepared
for the general public,
and there is no
possibility to switch the
way in which the
information is provided
  
Comments can be added here:
The risk of
cybersickness
The risk of cybersickness
is completely reduced
The risk of cybersickness
can be a problem for
some groups of people
There is a high risk of
cybersickness
  

Sustainability 2020,12, 2698 19 of 24
Comments can be added here:
Perception of the
material
It is easy to perceive the
material
The material is well
prepared but sometimes
it is hard to perceive due
to the complexity of the
information
It is hard to perceive the
material
  
Comments can be added here:
Structure of the
information provided
The information
provided is well
structured and easy to
understand
Some parts of the
information are well
structured but some
information lacks
structure and it is not
easy to understand
There is no structured
information provided
  
Comments can be added here:
Information provided
during the use of the
material
All the information is
given in an easy to
understand way even
without previous
knowledge on the topic
Some parts of the
information are given in
an easy to understand
way, but some parts are
hard to understand
without previous
knowledge on the topic
The information is hard
to understand (due to
complexity,
fragmentation or other
problems)
  
Comments can be added here:
The sequence of
information flow
The sequence of the
information flow can be
changed according to the
decision of the person
who explores the
material
Some parts of the
information can be
skipped or changed
The sequence of the
information flow cannot
be changed according to
the decision of the
person who explores the
material
  
Comments can be added here:
Information about the
historical period of the
artifact
Information about the
historical period is given
and it is clear to
understand
Information about the
historical period is given
but it is hard to
understand
Information about the
historical period is not
given
  
Comments can be added here:
Information about the
place of origin of the
artifact
Information about the
place of origin of the
artifact is given and it is
clear to understand
Information about the
place of origin of the
artifact is given but it is
hard to understand
Information about the
place of origin of the
artifact is not given
  
Comments can be added here:
Additional information
is given in audial form
A lot of additional
information in audial
form is given
Some additional
information in audial
form is given
No additional
information in audial
form is given
  

Sustainability 2020,12, 2698 20 of 24
Comments can be added here:
Additional information
is given in written form
A lot of additional
information in written
form is given
Some additional
information in written
form is given
No additional
information in written
form is given
  
Comments can be added here:
Connectivity with other
information
There is smooth
connectivity with other
parts of the information,
other artifacts, other
historical facts, etc.
There is fragmented
connectivity with other
parts of the information,
other artifacts, other
historical facts, etc.
There is no connectivity
with other information
  
Comments can be added here:
Knowledge test The possibility to test
knowledge is included in
different parts of the
material and on different
aspects of the
information provided
There is a possibility to
test knowledge, but it is
only on a few aspects of
the material
There is no possibility to
test the knowledge
included
  
Comments can be added here:
Possibility to organize
group activities
It is possible to organize
group activities for an
unlimited number of
participants
It is possible to organize
group activities for small
groups (up to 3 people)
It is not possible to
organize any group
activities while
interacting with the VR
experience
  
Comments can be added here:
Elements of
gamification
Elements of gamification
are used to attract people
and to keep them
focused
Some elements of
gamification are used but
on a fragmented basis
Elements of gamification
are not used
  
Comments can be added here:
The material challenges
the user to find out
more information
The material is
developed in a way to
challenge the user to find
out more information
There are some elements
that challenge the user
but these are not used on
a regular basis
The material does not
challenge the user to find
out more information
  
Comments can be added here:
Focus of attention The material is
developed in a way that
keeps the user’s
attention focused on the
experience all the time
There are some parts
where the focus of the
user’s attention is
stimulated but not
throughout the
experience
The material is
interesting but there are
no specific features to
capture the attention of
the user
  
Comments can be added here:
The age group for
which the material can
be used
Adults (18+) School-age children
(7-18)
Minors (up to 7)
  
Comments can be added here:

Sustainability 2020,12, 2698 21 of 24
Appendix B
The list of analyzed VR museums
1. Chateu de Versailes
2. PompieTouch
3. NEFERTITI VR by EON
4. National Museum of Iraq
5. Macao Museum
6. Bosch VR
7. MagiChapel
8. Civilisations
9. Memory Museum
10.
Batle of Batina Memorial museum
11.
Universal Museum of Art
12.
Toumanian museum
13.
Museum of Applied Arts Budapest
14.
Pompeii
15.
Houghton Revisited
16.
Colm Cille
17.
Never built New York
18.
Nicola Tesla experience
19.
Asolo
20.
Haramain exhibition
21.
Bilbo
22.
Museo del San Michele
23.
Viewmuseum
24.
Tank tour
25.
Hermes
26.
Hermitage
27.
Daily Art
28.
Muzeum Archeologiczne
29.
BMW museum
30.
Rijks Museum
31.
Edu Must
32.
Toile de Joy
33.
Museum of Portimao
34.
Dinaledi
35.
Solar System
36.
Museum of Architecture
References
1.
Kapp, K.M. 3 Instructional Design Strategies for Virtual Reality Learning. 2017. Available online: https:
//elearningindustry.com/instructional-design-strategies-virtual-reality-learning (accessed on 23 January 2020).
2.
Lombardo, J.M.; L
ó
pez, M.A.; Garcia, V.M.; L
ó
pez, M.; Cañadas, R.; Velasco, S.; Le
ó
n, M. Practica. a virtual
reality platform for specialized training oriented to improve the productivity. Int. J. Interact. Multimed. Artif.
Intell. 2018. [CrossRef]

Sustainability 2020,12, 2698 22 of 24
3.
Daniela, L.; Aierken, Y. The educational perspective on Virtual Reality experiences of cultural heritage.
Changing museums. In New Perspectives on Virtual and Augmented Reality: Finding New Ways to Teach in a
Transformed Learning Environment; Daniela, L., Ed.; Routledge: London, UK, 2020.
4. Zyda, M. From visual simulation to virtual reality to games. Computer 2005,38, 25–32. [CrossRef]
5.
Kim, G. Designing Virtual Reality Systems: The Structured Approach; Springer: Berlin/Heidelberg, Germany,
2005; ISBN 978-1-85233-958-6.
6. Á
lvarez, J.G.; Acevedo, J. Meta-analysis of the impact of Augmented Reality on students’ learning gains.
Educ. Res. Rev. 2019,27, 244–260.
7. Sweller, J. The worked example effect and human cognition. Learn. Instr. 2006,16, 165–169. [CrossRef]
8.
Bannan, B.; Cook, J.; Pachler, N. Reconceptualizing design research in the age of mobile learning. Interact.
Learn. Environ. 2015,24, 938–953. [CrossRef]
9.
Lepouras, G.; Katifori, A.; Vassilakis, C.; Charitos, D. Real exhibitions in a virtual museum. Virtual Real.
2004,7, 120–128. [CrossRef]
10.
Papaefthymiou, M.; Kateros, S.; Georgiou, S.; Lydatakis, N.; Zikas, P.; Bachlitzanakis, V.; Papagiannakis, G.
Gamified AR/VR Character Rendering and Animation-Enabling Technologies. In Mixed Reality and
Gamification for Cultural Heritage; Ioannides, M., Magnenat-Thalmann, N., Papagiannakis, G., Eds.; Springer:
Cham, Switzerland, 2017; pp. 333–357. ISBN 978-3-319-49606-1.
11.
Kopetz, H. Real-Time Systems: Design Principles for Distributed Embedded Applications (Real-Time Systems Series);
Springer: Boston, MA, USA, 2011; ISBN 978-1-4419-8236-0.
12.
Craig, A.B.; Sherman, W.R.; Will, J.D. Developing Virtual Reality Applications: Foundations of Effective Design;
Morgan Kaufmann: Burlington, MA, USA, 2009; ISBN 978-0-12-374943-7.
13.
Andre, L.; Durksen, T.; Volman, M.L. Museums as avenues of learning for children: a decade of research.
Learn. Environ. Res. 2016,20, 47–76. [CrossRef]
14.
Herrington, J.; Oliver, R. An instructional design framework for authentic learning environments.
Educ. Technol. Res. Dev. 2000,48, 23–48. [CrossRef]
15.
Lave, J.; Wenger, E. Situated Learning. Legitimate Peripheral Participation; University of Cambridge Press:
Cambridge, UK, 1991; ISBN 978-0-5214-2374-8.
16.
Schneider, K. How to promote entrepreneurial identity through edutainment? J. Entrep. Educ.
2019
,22, 1–12.
17.
Re’Vell, M.D. Moving Toward Culturally Restorative Teaching Exchanges: Using Restorative Practices to
Develop Literacy across Subject Area-Content. Int. J. Smart Educ. Urban Soc. 2019,10, 53–69. [CrossRef]
18.
Sharples, M.; Lonsdale, P.; Meek, J.; Rudman, P.D.; Vavoula, G.N. An Evaluation of MyArtSpace: A Mobile
Learning Service for School Museum Trips; Norman, A., Pearce, J., Eds.; University of Melbourne: Melbourne,
Australia, 2017; pp. 238–244.
19.
Kizilkaya, L.; Vince, D.; Holmes, W. Design Prompts for Virtual Reality in Education. In Artificial Intelligence
in Education; Springer: Cham, Switzerland, 2019; pp. 133–137. ISBN 978-3-030-23206-1.
20.
Styliani, S.; Fotis, L.; Kostas, K.; Petros, P. Virtual museums, a survey and some issues for consideration.
J. Cult. Heritage 2009,10, 520–528. [CrossRef]
21.
Aguayo, C.; Eames, C.; Cochrane, T. A Framework for Mixed Reality Free-Choice, Self-Determined Learning.
Res. Learn. Technol. 2020,28. [CrossRef]
22.
Burden, K.; Kearney, M. Conceptualising authentic mobile learning. In Mobile Learning Design: Theories and
Application; Churchill, D., Lu, J., Chiu, K.F.T., Fox, B., Eds.; Springer: Singapore, 2016; pp. 27–42.
23.
Sırakaya, M.; Cakmak, E.K. The Effect of Augmented Reality Use on Achievement, Misconception and
Course Engagement. Contemp. Educ. Technol. 2018,9, 297–314. [CrossRef]
24.
Huang, H.-M.; Liaw, S.-S.; Lai, C.-M. Exploring learner acceptance of the use of virtual reality in medical
education: a case study of desktop and projection-based display systems. Interact. Learn. Environ.
2013
,24, 3–19.
[CrossRef]
25.
Atilola, O.; Tomko, M.; Linsey, J. The effects of representation on idea generation and design fixation: A
study comparing sketches and function trees. Des. Stud. 2016,42, 110–136. [CrossRef]
26.
Jou, M.; Wang, J. Investigation of effects of virtual reality environments on learning performance of technical
skills. Comput. Hum. Behav. 2013,29, 433–438. [CrossRef]
27.
Fink, R.D.; Weyer, J. Interaction of Human Actors and Non-Human Agents. Sci. Technol. Innov. Stud.
2014,10, 47–64.

Sustainability 2020,12, 2698 23 of 24
28.
Latour, B. Reassembling the Social: An Introduction to Actor-Network-Theory; Oxford University Press: Oxford,
UK, 2005.
29.
Barroso Osuna, J.; Guti
é
rrez-Castillo, J.J.; Llorente-Cejudo, M.D.; Valencia Ortiz, R. Difficulties in the
Incorporation of Augmented Reality in University Education: Visions from the Experts. J. New Approaches
Educ. Res. 2019,8. [CrossRef]
30.
Daniela, L.; Strods, R. Robot as Agent in Reducing Risks of Early School Leaving. In Innovations, Technologies
and Research in Education; Daniela, L., Ed.; Cambridge Scholars Publishing: Newcastle upon Tyne, UK, 2018;
pp. 140–158. ISBN 978-1-5275-0622-0.
31.
Rubene, Z. The Portret of a Contemporary Child and Youngster in the Global Education Space. Int. J. Smart
Educ. Urban Soc. 2018,9, 17–26. [CrossRef]
32.
Quinn, M. Digital Schools? Teachers Still Matter. J. Chart. Coll. Teach.
2019
, Specials Issue, Education
Technology. 9–11. Available online: https://impact.chartered.college/article/digital-schools-teachers-still-
matter/(accessed on 18 August 2019).
33.
Mishra, P.; Koehler, M. Technological pedagogical content knowledge: A framework for teacher knowledge.
Teach. Coll. Record 2006,108, 1017–1054. [CrossRef]
34.
ˇ
Cernochov
á
, M.; Vonkova, H.; Št
í
pek, J.; ˇ
Cern
á
, P. How Do Learners Perceive and Evaluate Their Digital
Skills? Int. J. Smart Educ. Urban Soc. 2018,9, 37–47. [CrossRef]
35.
Moro, C.; Stromberga, Z.; Stirling, A. Virtualisation devices for student learning: Comparison between
desktop-based (Oculus Rift) and mobile-based (Gear VR) virtual reality in medical and health science
education. Australas. J. Educ. Technol. 2017,33, 1–10. [CrossRef]
36.
Yannier, N.; Hudson, S.E.; Wiese, E.S.; Koedinger, K.R. Adding Physical Objects to an Interactive Game
Improves Learning and Enjoyment. ACM Trans. Comput. Interact. 2016,23, 1–31. [CrossRef]
37.
Spector, J. Conceptualizing the emerging field of smart learning environments. Smart Learn. Environ.
2014,1, 1–10. [CrossRef]
38.
Boughalem, M.A.; Khaldi, M. Social Constructivism and Digital Learning. Int. J. Smart Educ. Urban Soc.
2019,10, 13–22. [CrossRef]
39.
Luckin, R.; Bligh, B.; Manches, A.; Ainsworth, S.; Crook, C.; Noss, R. Decoding Learning: The Proof, Promise
and Potential of Digital Education; NESTA: London, UK, 2012; Available online: https://media.nesta.org.uk/
documents/decoding_learning_report.pdf (accessed on 23 January 2020).
40.
Daniela, L. Smart Pedagogy for Technology Enhanced Learning. In Didactics of Smart Pedagogy: Smart
Pedagogy for Technology Enhanced Learning; Daniela, L., Ed.; Springer: Cham, Switzerland, 2019; pp. 3–22.
ISBN 978-3-030-01550-3.
41.
Resmini, A. Preface. In Reframing Information Architecture. Human-Computer Interactions Series (Human-Computer
Interaction Series); Resmini, A., Ed.; Springer: Cham, Switzerland, 2014; pp. v–vi. ISBN 978-3-319-06491-8.
42.
Hinton, A. Understanding Context: Environment, Language, and Information Architecture; O’Reilly Media:
Sebastopol, CA, USA, 2014; ISBN 0787721884064.
43.
Lacerda, F.; Lima-Marques, M.; Resmini, A. An Information Architecture Framework for the Internet of
Things. Philos. Technol. 2018, 1–18. [CrossRef]
44.
Resmini, A.; Lacerda, F. The architecture of cross-channel ecosystems: from convergence to experience.
In Proceedings of the 8th International Conference on Management of Digital EcoSystems, Biarritz, France,
1–4 November 2016; pp. 17–21, ISBN 978-1-4503-4267-4.
45.
Benyon, D.; Resmini, A. User experience in cross-channel ecosystems. In Proceedings of the 31st International
BCS Human Computer Interaction Conference (HCI 2017), 3–6 July 2017; London, UK, BCS Learning and
Development: Swindon, UK, 2017; pp. 272–280.
46.
Floridi, L. The Fourth Revolution: How the Infosphere is Reshaping Human Reality; Oxford University Press:
Oxford, UK, 2014; ISBN 9780199606726.
47.
Fowler, C. Virtual reality and learning: Where is the pedagogy? Br. J. Educ. Technol.
2015
,46, 412–422.
[CrossRef]
48.
Ware, C. Visual Thinking: For Design; Morgan Kaufmann: Amsterdam, The Netherlands, 2008; ISBN
9780123708960.
49.
Bacca, J.; Baldiris, S.; Fabregat, R.; Graf, S.; Kinshuk, D. Augmented Reality Trends in Education: A Systematic
Review of Research and Applications. Educ. Technol. Soc. 2014,17, 133–149.

Sustainability 2020,12, 2698 24 of 24
50.
Cheng, K.-H. Reading an augmented reality book: An exploration of learners’ cognitive load, motivation,
and attitudes. Australa. J. Educ. Technol. 2017, 53–69. [CrossRef]
51.
Delello, J.A. Insights from pre-service teachers using science-based augmented reality. J. Comput. Educ.
2014,1, 295–311. [CrossRef]
52.
Mannion, J. Growth Headset? Exploring the use of Virtual Reality and Augmented Reality in Schools. J. Chart.
Coll. Teach.
2019
, Specials Issue, Education Technology. 5–8. Available online: https://impact.chartered.college/
article/growth-headset-exploring-virtual-reality-augmented-reality-schools/(accessed on 23 January 2020).
53.
Edmonds, W.A. Phenomenological perspective. In An Applied Guide to Research Designs; Edmonds, W.A., Ed.;
Sage: Thousand Oaks, CA, USA, 2017; pp. 168–176. ISBN 9781483317274.
54. Mathison, S. Encyclopedia of Evaluation; Sage: Thousand Oaks, CA, USA, 2005; ISBN 9780761926092.
55.
Quinlan, A.M. A Complete Guide to Rubrics: Assessment Made Easy for Teachers of K-College, 2nd ed.; Rowman &
Littlefield Education: Lanham, MD, USA, 2012; ISBN 978-1607096733.
56.
Stevens, D.D.; Levi, A.J. Introduction to Rubrics: An Assessment Tool to Save Grading Time, Convey Effective Feedback,
and Promote Student Learning, 2nd ed.; Stylus Publishing: Sterling, VA, USA, 2013; ISBN 978-1579225889.
57. Sharples, M. Mobile learning: Research, practice and challenges. Distance Educ. Chin. 2013,3, 5–11.
58.
Edmonds, R.; Smith, S. Location-based mobile learning games: Motivation for and engagement with the
learning process. In Proceedings of the Mobile Learning Futures—Sustaining Quality Research and Practice
in Mobile Learning, 15th World Conference on Mobile and Contextual Learning, mLearn 2016, Sydney,
Australia, 24–26 October 2016.
©
2020 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).