Conference PaperPDF Available

TOWARDS AUGMENTED REALITY EDUCATIONAL AUTHORING

Authors:

Abstract and Figures

Nowadays, there is a growing interest in Augmented reality (AR) as a field of research, as well as a domain for developing a broad variety of applications. Since the coining of the phrase "Augmented reality" in 1990, the technology has come a long way from research laboratories and big international companies to suit the pockets of millions of users all over the world. AR's popularity among younger generations has inspired an effort to utilize AR as a tool for education. For teachers, starting with AR educational authoring, we selected some important milestones of the history of the field with the focus on the specific domain of educational applications. We comment on Videoplace and Construct3D projects in more detail. Finally, we draw a few implications from the available literature for educational authoring.
Content may be subject to copyright.
TOWARDS AUGMENTED REALITY EDUCATIONAL
AUTHORING
Zuzana Berger-Haladová & Andrej Ferko
Comenius University
Mlynska dolina, 842 48 Bratislava, Slovakia
zhaladova@gmail.com, ferkoaf@gmail.com
Abstract: Nowadays, there is a growing interest in Augmented reality (AR) as a
field of research, as well as a domain for developing a broad variety of
applications. Since the coining of the phrase "Augmented reality" in 1990, the
technology has come a long way from research laboratories and big international
companies to suit the pockets of millions of users all over the world. AR’s
popularity among younger generations has inspired an effort to utilize AR as a
tool for education. For teachers, starting with AR educational authoring, we
selected some important milestones of the history of the field with the focus on
the specific domain of educational applications. We comment on Videoplace and
Construct3D projects in more detail. Finally, we draw a few implications from the
available literature for educational authoring.
Keywords: Augmented reality, taxonomy, authoring, digital cultural heritage.
INTRODUCTION
Augmented reality as an extension to GeoGebra (Brzezinski, 2018) presented an
important milestone. We survey AR ideas in the context of the related field of
virtual reality (VR) systems, which serve to enhance imagination, interaction and
immersion. Our focus is educational authoring devices that exploit ideas from
digital cultural heritage.
Myron Krueger named the new technology Artificial reality in the mid 70s, but
Jaron Lanier’s name Virtual reality won. In the year 2016, Dieter Schmalstieg and
Tobias Holerer predicted that immersion would would not only be an important
quality measure in VR systems, but in AR systems as well. Krueger’s Videoplace
(1975) artistic goal was to establish a novel art of interaction. It projected
silhouettes of users on the wall in real-time, where they interacted and despite the
2D nature of the simple virtual world they had a strong experience of “being
E-Learning and STEM Education
Scientific Editor Eugenia Smyrnova-Trybulska
“E-Learning”, 11, Katowice-Cieszyn 2019, pp. 587-608
Zuzana Berger-Haladová, Andrej Ferko
588
there”. The breathtaking educational goal of Videoplace was never reached. Myron
Krueger presented fantastic 2D creatures to groups of children. They were expected
to observe the artificial reality, name the unnamed objects, self-organize seminars
to plan their further research, subdivide the workload and, eventually, discover for
adults the new methods of research. The author assumed, that there are research
methods which were not noticed by “adult science” and he relied on creativity of
children… Videoplace visual artists had to create visible objects and their
behaviours to be different from anything known. The Artificial reality mixed real
and virtual to challenge imaginative, interactive and immersive discovery.
VR and AR are very close research fields and in spite of the clear delineation of
the terms, it is sometimes hard for the public to distinguish between them. In the
reality-virtuality continuum, defined by Milgram et al. in (Milgram et al., 1995)
we can see that AR is a form of a broader mixed reality, which lies between entirely
real and completely virtual environments. Azuma in his seminal paper (Azuma, 1997)
defines AR as systems that have the following three characteristics:
1.
they combine real and virtual,
2.
they are interactive in real time,
3.
they are registered in 3D.
In other words, this new medium uses real world surfaces for immediate
projections, which are put into the same coordinate system. While Videoplace
registered real user silhouette in the 2D virtual world of fantasy, AR computer
vision subsystem registers the augmenting images into the 3D real world, and the
user into 3D virtual one, which provides immediate imagination and immersion.
We explain the difference in a more structured way in two AR classifications
below.
When defining virtual reality, we have to enclose the 3rd and the 2nd point
from the AR definition. Another important aspect of virtual reality is immersion in
the virtual environment, but when defining the AR, we should use the term
ultimate immersion because there is nothing more immersive than the reality
itself.
Despite the differences, these two fields have a connected history. For
example, Sutherland’s Ultimate display (Sutherland, 1965) or head-mounted
display (Sutherland, 1968) are important milestones in both AR and VR
history. The term virtual reality has been used to describe different things, for
example a theatre, but in the ’80s, the term virtual reality was coined and
popularized by Jaron Lanier (as referring to immersive environments created by
applications with visual and 3D effects, Lanier, 2010) and the boom started at the
beginning of the ’90s. Not long after, the phrase augmented reality was coined by
Tom Caudell in 1990 (Caudell and Mizell, 1992) and the boom started with the
beginning of the new century. We describe a chronologic selection of 20 VR/AR
Towards Augmented Reality Educational Authoring
589
milestones (ideas, papers, projects, books) in Part 1 and AR systems in general in Part
2. Part 3 compares two classifications to cover the AR field more generally. Finally,
we identify the potential in AR authoring methodology. We discuss the implications
and apply our approach to the Construct3D project in Part 4.
1 SELECTED MILESTONES (1966-2016)
In view of the possibility that any chronological selection remains arguable, we
identified the following “minimal set” of 21 cited findings in the first 50 years of
VR/AR. This evolution resulted in a relatively general AR system, which we
present in Part 2. Analogously with computer graphics reference model, it can
serve to specify the particular architecture and application functionality. The
variety of options can be found in two prominent textbooks (Bimber and Raskar,
2005), (Schmalstieg and H öl l er e r , 2016). To reduce the reductionism risk, we
describe the classifications of technical alternatives in Part 3. Besides important
landmarks, we included several famous items, which influenced world VR/AR
popularity in given time. In Part 5, we will explain data streams in one session with
the “classic” educational application, Construct 3D. Adding a new chapter to the
three already submitted (Prodromou 2019), we explore AR ideas in the context of
the related field of VR systems. They discuss AR mathematics teaching and its
qualitative evaluation (Babinska, Dillingerova, and Korenova, 2019), rich
hardware options (Bohdal, 2019), and adequate didactic evaluation (Kostrub and
Ostradicky, 2019). All these aspects should be taken into account by an author of a
novel AR educational project.
1966 Ivan Sutherland presented his concept of the ultimate display. His idea,
however, goes beyond the limits of VR and AR that we know today. In
his paper (Sutherla nd, 1965), he remarked that: "The ultimate display
would, of course, be a room within which the computer can control the
existence of matter. A chair displayed in such a room would be good
enough to sit in. Handcuffs displayed in such a room would be confining,
and a bullet displayed in such a room would be fatal". This device is
considered the first AR interface. In 1968, Sutherland presented his
popular head-mounted display (Sutherland, 1968).
1975 Myron Krueger experimented with computer generated art and
interaction. In the Videoplace project, a computer responded to gestures
and interpreted them into actions. The audience could interact with their
own silhouettes generated from the video camera (Krueger et al.,
1985).
1978 Professor Steve Mann is wearing the HMD (or HUD) since 1978. In 2001
Peter Lynch shot about him the film called Cyberman. Much of the film
was created by Mann himself with his EyeTap (Mann, 2004). EyeTap is the
HUD (heads-up display mounted in glasses), which records the reality
Zuzana Berger-Haladová, Andrej Ferko
590
with the camera, creates an virtual information and merges the reality
seen by the user with a virtual information using a beam splitter. (Again,
there are multiple meanings of HMD or HUD.)
1990 Tom Caudell, the researcher who developed the AR system supporting
the aircraft manufacturing in the Boeing factory (Caudell and Mizell,
1992), coined the phrase augmented reality.
1991 The concept of "ubiquitous computing" was presented by Weiser
(Weiser, 1991) in the beginning of the ’90s. The goal of "ubiquitous
computing" is to provide computer interfaces that are natural for the
users, to develop the computers which are not visible but
"omnipresent" in everyday life. This concept is closely connected to
the possibilities and techniques of the AR and the fusion of the fields is
known as the ubiquitous AR. (In the year 2016, Schmalstieg and Höllerer
proposed Weiser-Milgram spectrum of AR options).
1993 The CAVE: Audio Visual Experience Automatic Virtual Environment was
presented to the public. CAVE contributed to public awareness of VR.
1993 Steven Feiner, Blair MacIntyre, et al. published two major AR papers. The
first paper (Feiner et al., 1993b) presents the KARMA (knowledge based
AR for maintenance assistance) system which uses the optical see through
head-mounted display that "explains simple end-user laser printer
maintenance". The second paper (Feiner et al., 1993a), presents 2D
information windows in the AR, a technique, which is nowadays broadly
used in smartphone (pseudo) AR systems.
1997 Ronald T. Azuma published the first survey (Azuma, 1997) on AR. He
gave the definition of augmented reality, which is considered the most
relevant. Also, he listed the biggest problems of AR as the registration
and the sensing errors. The paper presents a broad survey of different
applications of AR in medical, manufacturing, visualization, path planning,
entertainment and military fields.
1999 ARToolkit was developed by H. Kato in the Nara Institute of Science
and Technology. In 1999 Kato and Billinghurst published their paper
(Kato and Billinghurst, 1999) about using HMD and markers for the
conferencing system, based on the method proposed by Rekimoto
(Rekimoto, 1996). ARToolkit is a computer library for the tracking of
the visual markers and their registration in the camera space
(http://www.hitl.washington.edu/artoolkit/). With the ARToolkit one can
easily develop AR applications with virtual models assigned to different
markers. For an example see Figure 4.
2000 Hannes Kaufmann, Dieter Schmalstieg and Michael Wagner introduced
Construct3D, a three-dimensional geometric construction tool based on the
collaborative AR system ‘Studierstube’. The setup uses a stereoscopic
Towards Augmented Reality Educational Authoring
591
head mounted display (HMD) and the Personal Interaction Panel (PIP)
- a two-handed 3D interaction tool that simplifies 3D model interaction.
Applications in mathematics and geometry education at t he high
school and university levels were discussed.
2002 Bruce Tomas developed the first AR outdoor game called ARQuake
(Thomas et al., 2000). It was an AR version of the computer game
Quake. Different versions of the system (2000 2002) used the optical
see through head-mounted display, mobile computer stored in the
backpack, haptic gun or handheld device with button, head tracker,
digital compass, GPS system and/or markers. It allowed the user to
walk around in the real world and shoot virtual enemies from the Quake
game.
2005 Oliver Bimber and Ramesh Raskar published the first book on Spatial
Augmented Reality (Bimber and Raskar, 2005). They describe and
categorize AR systems into 3 categories: head-mounted, handheld, and
spatial, and then focus on the spatial systems. The main difference between
spatial AR and other categories is that in SAR the display is separated
from the users of the system and so is suitable for bigger groups of users.
SAR systems usually consist of digital projectors, which display graphical
information directly onto physical objects. Bimber and Raskar describe the
technique of calibration of several projectors, which compensate the
inequality and the colour of the surface.
2007 Klein and Murray in their paper (Klein and Murray, 2007) proposed a
method for a markerless tracking for small-workspace AR applications.
They track a calibrated handheld camera in a previously unknown
scene without any known objects or initialization target, while
building a map of this environment.
2009 Although the spatial AR (and the projection mapping techniques) was
introduced several years before, the biggest boom in the urban
projection mapping was in 2009-2010. As the most famous examples
we have to mention the projection mapping during the 600th anniversary
of Orloj the astronomical tower clock situated at Old Town Square in
Prague in 2010 (the macula, 2010), or 20092011 NuFormer
Projections in the Netherlands (NuFormer, 2011).
2010 When Microsoft released Kinect, the motion sensing input device for the Xbox
360 console, it was expected to be "the birth of the next generation of
home entertainment" (Takahashi, 2009) but not a milestone in the AR
history. The Kinect sensor developed by PrimeSense company became a
really cheap ($150) source for the depth information for AR applications,
i.e. how far is the real object in the scene. The sensor itself consists of the
rgb camera, the infrared projector which projects a pattern of dots and
the detector which establishes the parallax shift of the dot pattern for each
Zuzana Berger-Haladová, Andrej Ferko
592
pixel. Instead of (x,y) we have (x,y,z) measurement of scene geometry.
Kinect holds the Guinness World Record of being the "fastest selling
consumer electronics device" (8 million units in its first 60 days). When
the first hackers broke into the device and found the way how to control
the sensors it took only 2 months and hundreds of AR application using
Kinect sensor appeared on the Internet. For the top examples see 12 best
Kinect hacks (Vsauce, 2010).
2011 Qualcomm presented Vuforia the software development platform for AR.
Vuforia enables the usage of real-world image markers and development of
native applications with support for iOS, Android, and Unity 3D
(Qualcomm, 2013).
2016 Pokemon GO, an AR game developed by Niantic for iOS and Android
devices, was released in summer 2016. The game became massively
popular and had been downloaded more than 500 million times
worldwide by the end of the year 2016.
2016 Microsoft Hololens headset launched for developers. In was the first
AR head mounted display to hit the market in 2016.
2016 Dieter Schmalstieg and Tobias Höllerer published the important textbook
Augmented reality: Principles and practice (Schmalstieg and
H öl l er e r , 2016). The detailed presentations are available for free
download and correct academic use from
http://www.augmentedrealitybook.org.
2. AR SYSTEMS
There is a discussion in the AR community whether the definition created in
the 90s still suffices the requirements of the users. Especially in the
commercial sphere there exist many applications which are categorized as AR
applications, but do not fulfil the second or third of Azuma’s rules. These
applications usually lie within the reality-virtuality continuum, but cannot be
considered AR. This lack of true commercial AR leads to misclassification
also in some scientific publications.
For example, in the extensive survey (Olsson and Salo, 2011) published in
the proceedings of ISMAR authors decided to include 2 kinds of applications:
AR browsers which they defined as: "...usually includes the delivery of points
of interest (POI), user-created annotations, or graphics based on the GPS
location of the device and orientation of the built-in magnetometer" and image-
recognition-based AR which was defined as: "based on connecting surrounding
objects, products, and other physical targets with digital information with the
help of visual recognition. By identifying quick response (QR) codes, bar-
codes, other graphic markers, or the objects themselves..." Here, we strictly
Towards Augmented Reality Educational Authoring
593
follow Azuma’s definition and we decided to call the systems not fulfilling these
rules the pseudo-AR ones.
Figure 1. A scheme of AR system
Source: Own work
AR systems can consist of many different elements, depending on the type of the
application. We can divide these elements into four categories: inputs
(sensors), outputs (projectors, displays), computers and accessories. It is
necessary for every AR system to have at least one sensor for the estimation
of the user’s position (camera, GPS receiver), one device to display the AR
or to add virtual objects into user’s view frustum (display, projector) and
some device capable of processing data (computer). All the different
components, elements, subsystems, necessary for the AR system can be
incorporated in one device, for example a smartphone, tablet or notebook with a
built-in webcamera. Many types are explained in (Bimber and Raskar, 2005),
(Schmalstieg and H öl l er e r , 2016).
In Figure 1, we can see the scheme of the common AR system equipped with
a camera, a computer and a display. As the first step, the position of the real
camera in space has to be estimated and the alignment (registration) of the real
camera to the graphics camera has to be done. Visual (or other types of)
markers, pattern matching or local features matching are usually used for the
estimation of the rotation and translation of the camera to the object to be
augmented (we will focus on the registration of the virtual and real camera in the
next section). The virtual objects are then merged with the real scene and the
augmented video is created and displayed.
Zuzana Berger-Haladová, Andrej Ferko
594
3. TWO CLASSIFICATIONS OF AR APPROACHES
The AR systems can be categorized by different factors, including the application
area, the possibility of more people collaborating or the size of the full system.
In the following section we present two different classification schemes of the
AR applications. The first one was developed by Bimber and Raskar in
(B im b er and Ra s ka r , 2005) and it presents a device-based categorization
(3.1). The second scheme is our own classification based on the way of
augmentation of virtual and real world. The user’s immersion is the key aspect
of the AR systems. Our classification (3.2) is inspired by the survey from
Azuma (A z u ma , 1997) and it can be helpful in taking project decisions.
3.1 Device based classification
The categories proposed in (B i mb er and R as k ar , 2005) are based on how the
output device is connected with the user. If the user wears the device on his head,
we talk about head-mounted devices. The systems designed to be carried in hand
belong to the handheld category and stationary systems not carried by the user are
the spatial category.
3.1.1
Head-mounted devices
The head-mounted category consists of five main types of devices: Optical see-
through HMD, Video see-through HMD, HMProjectors, HMProjective display
and retinal displays. For more information about HMDs see (Cakmakci and
Rolland, 2006).
Optical see-through head-mounted display Azuma (1997) states that: "Optical
see-through HMDs work by placing optical combiners in front of the user’s
eyes. These combiners are partially transmissive, so that the user can look
directly through them to see the real world. The combiners are also partially
reflective, so that the user sees virtual images bounced off the combiners from
head-mounted monitors. The optical combiners usually reduce the amount of light
that the user sees from the real world. Since the combiners act like half-silvered
mirrors, they only let in some of the light from the real world, so that they can
reflect some of the light from the monitors into the user’s eyes."
Video see-through head-mounted display. This type of HMD was defined
(A zu ma , 1997) as: "Video see-through HMDs work by combining a closed-
view HMD with one or two head-mounted video cameras. The video cameras
provide the user’s view of the real world. Video from these cameras is
combined with the graphic images created by the scene generator, blending
the real and virtual. The result is sent to the monitors in front of the user’s
eyes in the closed-view HMD."
Head-mounted projectors beam the generated images onto the ceiling and use two
half-silvered mirrors to integrate the projected stereo image in front of the
user.
Towards Augmented Reality Educational Authoring
595
Head-mounted projective displays redirect the image created by miniature projectors
with mirror beam combiners so the images are beamed onto retro-reflective
surfaces in front of the users eyes.
Retinal displays use low-power semiconductor lasers to project modulated light
directly onto the retina of human eye. The main disadvantage of this technique
is that it provides only a non-stereoscopic monochromatic image (Bimber and
Raskar, 2005).
3.1.2 Handheld devices
Handheld devices are nowadays the most popular platforms for the AR applications.
These devices usually incorporate all the necessary sensors, computer and display
(or projector) in one portable gadget. Common handheld devices are
smartphones, tablets, palmtops or notebooks. Although most of the published
papers in the area of mobile AR focus on these particular devices, there were also
some efforts to build special handheld devices, for example iLamps (Raskar et
al., 2005). In iLamps Raskar et al. presented object augmentation with a handheld
projector utilizing a new technique for adaptive projection on non-planar surfaces
using conformal texture mapping.
3.1.3 Spatial devices
The spatial category encloses different solutions designed to be fixed within the
environment (not to be worn in the hand or on the head). An example of spatial
solutions are: PC stations with a webcamera, the CAVE (cave automatic virtual
environment) (Cruz-Neira et al., 1993), Projection mappings (the macula,
2010; NuFormer, 2011), Virtual showcase (Bimber et al., 2001). The Fish tank
is the title of a system consisting of a computer station equipped with a
webcamera and a monitor which are used for AR at home. The CAVE is an
immersive virtual reality/scientific visualization system, which lies between VR and
AR. The CAVE is a room-sized cube where three to six of the walls are used as
projection screens.
The Virtual Showcase developed by Bimber et al. (Bimber et al., 2001) presents
a projection-based multiviewer AR display device which consists of half silvered
mirrors and the graphical display. In this device the user can see real objects
inside the showcase (through the half-silvered mirrors) merged with virtual
objects or layers displayed on the projection screen under the showcase. This
technique makes use of the concept of Pepper’s ghost developed in 1862 (Burns,
2010).
3.2 Perception-of-reality-based classification
In our classification we start from Azuma’s work (Azuma, 1997) and we divide
AR systems based on the way they create the augmented experience. The first
category includes applications which create AR by adding the virtual
information (3D models, images, text) to the record of reality. The second
Zuzana Berger-Haladová, Andrej Ferko
596
category includes systems which create AR by displaying/projecting virtual
information directly in front of our vision of reality. Table 1 relates the device-
based classification and perception-of-reality-based classification.
3.2.1 The record of reality mixed with virtual information (added to
record)
All of the video see-through approaches belong to this category. The video see-
through device basically consists of the camera which records the reality and the
display (or a projector with a projection screen) which provides the user with
the reality mixed with the virtual information (the augmented experience). This
category includes video see-through head-mounted displays, most existing
handheld devices (smartphones, tablets, palmtops, netbooks) and the Fish tank
solutions.
3.2.2 Reality mixed with virtual information (added to reality)
This category includes all the applications in which the virtual information is
projected directly on the real world objects, or onto the optical see-through device.
The typical representatives of these approaches are the projection mapping
applications, for example, the projection on the astronomical tower clock Orloj
situated in the centre of Prague (the macula, 2010). Other systems which
belong to this category are optical see-through head-mounted displays, retinal
displays, head-mounted projectors, head-mounted projective displays, CAVE
(Cruz-Neira et al., 1993), Virtual showcase (Bimber et al., 2001), and also
some handheld solutions (for example, iLamps (Raskar et al., 2005), as
described in the section 4.1.2).
The morphological Table 1 offers for AR systems a two-dimensional
orientation, which can be helpful both for study and authoring. In fact, any AR
system was built following such a project decision. The classification is open. We
can add a next row for retinal display, for example.
Figure 2 illustrates some general AR building blocks (Bimber and Raskar,
2005) in 3 layers. We added selected user responses above. Tracking and
registration, display technology and rendering represent fundamental components
(subsystems). "On top of this base level, three more advanced modules can be
found: interaction devices and techniques, presentation, and authoring... Ideas and
early implementations of presentation techniques, authoring tools, and interaction
devices/techniques for AR applications are just emerging", wrote the authors in
2005. "Some of them are derived from the existing counterparts in related areas
such as VR, multimedia, or digital storytelling. Others are new and adapted more to
the problem domain of AR. However, it is yet too early to spot matured concepts
and philosophies at this level". "The third layer, the application, is finally the
interface to the user. Using AR, our overall goal is to implement applications that
are tools, which allow us to solve problems more effectively. Consequently, AR is
no more than a human-computer interface which has the potential to be more
Towards Augmented Reality Educational Authoring
597
efficient for some applications than others." In other words, there are 2 phases of
communication: authoring and presentation.
Table 1
Relating the device based classification and perception of the reality based
classification.
added to record
added to reality
head-
mounted
video see-through HMD
(Museum wearable (Spar acino, 2002))
optical see-through HMD
(S ut h e r l an d ’ s HMD
(Sutherland, 1968))
handheld
mobile/tablet AR
(e.g. museum guides (Bruns et al., 2007;
Miyash ita et al., 2008), (Kusunoki et
al., 2002; Ba y et al., 2006; F öc k l er et
al., 2005))
optical see-through handheld
displays handheld projections
(iLamps (Raskar et al.,
2005))
spatial
fish tank
mirror projections (Kalman, 1960).
Pepper’s ghost (Adr ien and
Clair e, 2013) projection
mappings (the macul a, 2010)
holographic displays (Bimber
et al., 2006)
Source: Own work
Figure 2. The AR building blocks and an example of user responses,
recognizing objects, generating associations, fixing meaning, and interacting
eventually.
Source: Own work based on Bimber and Raskar, 2005
Zuzana Berger-Haladová, Andrej Ferko
598
About a decade later, there is a brief Chapter 10 Authoring in (Schmalstieg and
H öl l er e r , 2016, pp. 329-344): "Based on a definition of setup for input and output,
the authoring defines a story (application logic), driven by interaction and
influencing actors arranged on stages". Multimedia objects are named actors
here, the story can be described as a state machine, and game is not taken into
account. The shift from single word authoring in the first AR book to a separate
chapter in the second prominent AR textbook can be extrapolated to a vision of
an universal AR authoring tool and workflow, like PowerPoint for a wide public,
Movio for a virtual heritage community... Moreover, "AR has the potential to
become the leading user interface metaphor for situated computing... The
world becomes the user interface... (Schmalstieg and H öl l er e r , 2016)" The
authoring, teaching, and learning can be everywhere, any time. How to solve the
authoring problem more effectively? The AR literature does not offer either an
universal authoring tool, or the methodology. Therefore we propose to apply the
novel theoretical framework introduced in Theorizing digital cultural heritage
(Cameron and Kenderdine, 2010). In other words, VR evaluation is more
matured than AR one. We are going to explain this with the Construct3D
example.
4. AUTHORING IMPLICATIONS
Combining parts of real and virtual worlds, human communication organizes the
content using stories and games (monologue, dialogue). A "Virtual Museum" can
be defined as a multimedia semiotic system, which offers a set of
microstories or game moves to communicate the given message, main story,
part of metastory. Virtual museums present multimedia collections for
visualization, activization, and even hermeneutics (presenting invisible). A
recent prognosis is given in (Papagiannakis, 2018) "Storytelling, presence, and
gamification are three basic fields that need to be taken into account when
developing novel mixed reality applications for cultural heritage...".
How should one measure virtual museum quality? The key human experience
with stories and games is a depth of immersion, which has five levels: curiosity,
sympathy, identification, empathy, transportation (Glassner, 2009). The strongest
form of gameplay immersion is flow experience (Csikszentmihalyi), “the
sense that the outer world has fallen away (Glassner, 2009). The question is
how to measure a quality of story/game immersion. The general answer lies
in the level of interestingness. In particular, the quality measurement can be
obtained using quantitative, qualitative, and virtual museum engagement factor
(Sherwood) measurements (Visits/visitors*duration) after (Camer on and
Kenderdine, 2010). The time of engagement is proportional to the level of
interestingness. E.g. the winning MOOC seems to be the world-famous Coursera
hit Learning How To Learn, with over 3 million subjects, who were engaged for
12 recommended hours (https://www.coursera.org/learn/learning-how-to-learn).
Towards Augmented Reality Educational Authoring
599
The specific quality measure for education with AR is proposed by (Kostrub
and Ostradicky, 2018). According to SAMR (Puentadura, 2018), there are
four levels of technology contribution in the classroom: Substitution,
Augmentation, Modification, and Redefinition. These SAMR model levels
proposed by Puentadura can be compared against classical Bloom’s taxonomy
of educational goals. For example, "redefinition" (Computer technology allows
new tasks that were previously inconceivable) can be seen as Evaluating
and/or Creating level as defined by Bloom.
The user can recognize visual percepts with growing complexity: single pixel,
output primitive, graphical object, semiotic representation, pattern, metaphor and
even an enthymeme. Enthymeme experience changes the user into the co-author,
the student into an cooperating (self-)teacher. This sharing of untold, "the body of
proof", "the strongest of rhetorical proofs...” can be exemplified with classical
syllogism "Socrates is mortal because he’s human", where one of premises is not
stated (All humans are mortal. Socrates is human. Therefore, Socrates is mortal.)
In virtual museology (Cameron and Kender dine, 2010) the enthymeme means
a top achievement, presenting unpresented, visualising invisible. These
museologic enthymemes are not reduced to rhetoric only, they consist of
multimedia objects (actors in AR system).
We learn in three ways only: 1. by pain, via amygdala, no repetitions, 2. by
repeating, via thalamus, 3. by discovery, enjoying endorphins, expressing AH,
AHA or HAHA (Koestler, 1964). The third way activates multiple brain parts in a
symphony of reconnections (Aamodt, 2009), A. Koestler calls this bisociation.
These two observations led us to define local interestingness.
Painful learning tradition was stopped by Comenius. Repetitive learning tradition
prevails today, motivated not by internal interestingness, but by external needs. The
third one, learning by discovery, is interesting itself, pleasant and funny. Using this
opinion, we can comment on AR educational authoring, as well.
Arthur Koestler discovered this a posteriori definition of interestingness, which we
use as a technique of making any text or image sequence locally interesting. What
was interesting, causes the AH, AHA or HAHA reaction in three areas of human
creativity Art, Science, and Humour (ASH). We live to escape from the banal
associative mental life to maximize our cultural capital with bisociations, the acts
of creation at the side of an author and, hopefully, at the side of the reader or
virtual museum visitor. If we are not sure with AHA, we are halfway, expressing
audible HM... How to reach HM? Ask a question, use rhetoric.
The global interestingness of any story/game is given by its theme (Rizvic,
2013). The local interestingness can be authored or evaluated by the bisociations,
causing AHA reactions. The engagement can be improved by a set of rhetorical
devices (e.g., pause, question, metaphor, intonation, repetition, and even an
entymeme), gamification, funology (from usability to enjoyment) (Blythe,
Zuzana Berger-Haladová, Andrej Ferko
600
2004). Good presentation ideas improve local interestingness of given
communication. While rhetoric organizes oral presentation, AR application
can profit from these well-working attention-getting tricks using sets of
mutimedia objects, as well. For example, very inspiring metaphors for
explaining algorithms can be found in (Forisek and Steinova, 2013). The
HAHA reactions were measured and even classified by Huron (Huron,
2004).
The virtual visitors enrich and train their multiple intelligences (after Gardner)
and they are expected to achieve various educational goals (within Bloom’s
taxonomy) with given level of attention. We understand educational AR authoring
as a specific subsystem of virtual museum in a wide sense, e.g. educational
content with GeoGebra YouTube Channel is a specific educational virtual
museum or exhibition. AR may serve as an ubiquitous or standalone subsystem
within any educational unit, story, or serious game. Using this approach, one
can author or even evaluate the AR system effects. Let us apply the proposed
approach to comment on Construct3D demo.
4.1 Classical project Construct3D on Youtube in 90 seconds overview
The far-reaching educational goal of Videoplace was a sort of visionary dream.
The successful practical project appeared decades later, it was Construct3D
(Kaufmann et al., 2000), which has been the most cited project in the field of
mathematics education using VR. What is the theme of Construct3D?
Figure 3. Spatial and temporal modelling of Construct3D session
Source: Own work based on Schmalstieg and Höl l erer, 2016
Towards Augmented Reality Educational Authoring
601
“Spatial abilities present an important component of human intelligence. The
term spatial abilities includes five components, spatial perception, spatial
visualization, mental rotations, spatial relations and spatial orientation... Generally,
the main goal of geometry education is to enhance special abilities by training
spatial skills. As shown in various studies... spatial abilities can be improved by
virtual reality (VR) technology.” For any given task, it is possible to make the
geometric solution visible for the teacher, but not for students. The theme means
the global interestingness.
Construct3D offers both visualization and activization of students. This can be
demonstrated by a 90-second YouTube video named Construct3D Overview
(Kaufmann, 2009). The real and virtual spaces are merged, augmented with an
interactive PIP (Personal Interaction Panel) and 3D geometric objects (cone,
cylinder, globe, and annotated coordinate axes). The user experience serves
immersing collaborating students to improve their imagination, interaction, and
spatial skills. There are local interestingness devices like real-time feedback or
indicating the top of cone by a red marker, however, the closing part of the video
offers a new level of local interestingness two views from two viewpoints in a
single screen. This trick destroys the illusion of single view and moves us to
another virtual space, where we can compare. The comparison leaves the field of
associations to bridge over two contexts, to bisociate, and, if one is fortunate
enough, this results in an AHA, AH or HAHA moment (Koestler). This possible
effect may bring a bit of the personal discovery for an activated student, and even
result in seeing the invisible, a percept not directly presented by AR system.
Construct3D is implemented using StudierStube. The data flows are indicated in
Figure 3, which we adapted after (Schmalstieg and H ö ll e r er , 2016).
What are the properties of geometric objects in this setup? They are inspired by
real world, physical universe, but they have to be modelled in mathematics. This
means that they consist from infinite number of points. This number must be
reduced for computer representation, typically using textured triangles. In final
implementation are these AR actors stored with given precision. The four
universes of computational mathematics (world, model, representation,
implementation) offer a basis for visualization (pixels, triangles, objects, iconic and
symbolic representations, metaphors...). However, some parts of virtual and real
scene should serve for augmenting and interaction. Such real-time actions require
careful optimizations [Lack19], especially for AR mobile applications, where the
computational power is limited.
4.2 AR workshop for kids
The first scientific exhibition for the general public in Slovakia, named Virtual
World 2012, took place in a large shopping centre, Avion. We included the AR
workshop for school kids there. The aim was to teach creation of an own AR
by pupils from elementary and secondary schools, or people with some
programming skills. First, we demonstrated tasks manageable for a given age
Zuzana Berger-Haladová, Andrej Ferko
602
category, like adding virtual objects (virtual information) to the real scene
(reality). Further, we explained basic requirements such as registering a 3D object
and real time interaction in medicine, advertising, sport news, design, cultural
heritage, and entertainment, of course.
Afterwards, the workshop continued with the practical part to familiarize the
participants with the selected tool (ZooBurst.com) to create their AR message.
The youngest authors prepared their own fairy tales, which appeared in the reality
of the image webcam on the computer, using printed black and white images
(marks, marker). The secondary school students worked in Flash, using Flartoolkit,
which allowed them to combine their own 3D model with their own AR
application.
The "adult informatics" authors were challenged with ARtoolkit, multiple
markers and 3D objects. They were provided with explanations of algorithms
used to detect markers and display 3D models. Our main concern was that each
participant, after graduating the workshop, understood the concept of AR and had
sufficient mastery to create his/her own AR application. We can conclude that
this sort of “novelty” interestingness is positively influenced by their own
creativity.
Figure 4. Photo from the workshop for high school students, the AR was
created using Artoolkit.
Source: Own work
CONCLUSION
We selected an overview of AR ideas in the context of the related field of VR
systems, which serve to support imagination, interaction and immersion. The
importance of immersion for AR systems evaluation is expected to grow
(Schmalstieg and H ö ll e r er , 2016). Naturally, the quality can be measured
either by standard didactic quantitative or qualitative methods, but the research in
Towards Augmented Reality Educational Authoring
603
virtual museology offers more matured authoring and success metrics. The
authors (Camer on and Kenderdine, 2010) combine theory of appraisal and
ideas from rhetoric into a promising double theoretical framework. However, we
propose to think practically, in terms of global and local interestingness as devices
to improve engagement, user experience, immersion. Virtual museum authoring
and research offer valuable alternatives of educational inspiration. For specific
maths-oriented educational purposes, we recommend studying and enriching the
Construct3D research line.
Acknowledgements
This research was supported by the grant KEGA 012UK-4/2018 “The Concept of
Constructionism and Augmented Reality in the Field of the Natural and Technical
Sciences of the Primary Education (CEPENSAR)”.
REFERENCES
Aamodt, S. & Wang, S. (2009). Welcome to Your Brain. Bloomsbury
USA. ISBN 9781596912830.
Adrien, M. & Claire, B. (2013). Emotion with the leap motion - pepper’s ghost
technique. Retrieved from http://vimeo.com/71216887 (accessed 1 June 2019).
Altshuller, G. (1999). The Innovation Algorithm: TRIZ, systematic innovation
and technical creativity. Technical Innovation Center Worcester.
Azuma, R. (1997). A survey of Augmented reality. Presence: Teleoperators and
Virtual Environments, 6(4): 355385.
Babinska, M., Dillingerova, M., Korenova , L. (2019). AR application and
future mathematics teachers. A chapter submitted to (Prodomou, 2019).
Bay, H., Fasel, B., & Van Gool, L. (2006). Gool. Interactive museum
guide: Fast and robust recognition of museum objects. In B. Schiele, L.van
Gool, & L. Paletta (Eds) Proceedings International Workshop on Mobile Vision.
Graz: Joanneum Research.
Bimber, O., F röhlic h, B. , Sch m a lst i eg, D ., & Enca r naçã o , L. M.
(2001). The virtual showcase. IEEE Computer Graphics and Applications.
21(6), 48-55.
Bimber, O. & Raskar, R. (2005). Spatial Augmented Reality: Merging Real and
Virtual Worlds. A K Peters/CRC Press.
Bohdal, R. (2019). Hardware for Augumented & Mixed Reality. History,
overview, basic principles & used technology. A chapter submitted to
(Prodomou, 2019).
Zuzana Berger-Haladová, Andrej Ferko
604
Brzezinski, T. (2018). Augmented Reality: Ideas for Student Explorations.
Retrieved from https://www.geogebra.org/ar (accessed 1 June 2019).
Blythe, M.A., Overbeeke, K., Monk, A.F., Wright, P.C. (Eds.) (2004). Funology,
From Usability to Enjoyment. Dordrec ht : Springer.
Bruns, E., Brombach, B., Zeidler, T., & Bimb er, O. (2007). Enabling
mobile phones to support large-scale museum guidance. Multimedia, IEEE,
14(2):16 25.
Burns, P. (2010). The history of the discovery of cinematography 1860
1869 Retrieved from http://www.precinemahistory.net/1860.htm (accessed 1
June 2019).
Cakmakci, O. & Rolland, J. (2006). Head-worn displays: A review. J.
Display Technol., 2(3):199 216.
Cameron, F. & Kenderdine, S. (2010). Theorizing digital cultural heritage: A
critical discourse. MIT Press.
Caudell, T. & Mizell, D. (1992). Augmented reality: an application of
heads-up display technology to manual manufacturing processes. In J. F.
Nunamaker (jr.) (ed.) System Sciences, 1992. Proceedings of the Twenty-Fifth
Hawaii International Conference on, (pp. 659 669).
Cruz-Neira, C., Sandin, D. J., & DeFanti, T. A. (1993). Surround-screen
projection-based virtual reality: the design and implementation of the CAVE.
In M.C. Whitton (ed.) Proceedings of the 20th annual conference on Computer
graphics and interactive techniques, SIGGRAPH ’93, (pp. 135142), New
York: ACM.
Feiner, S., MacIntyre, B., Haupt, M., & Solomon, E. (1993a). Windows
on the world: 2D windows for 3D augmented reality. In S. Hudson et al.
(Eds) Proceedings of the 6th annual ACM symposium on User interface
software and technology, UIST ’93, (pp. 145155), New York, NY, USA. ACM.
Feiner, S., Macintyre, B., & Seligmann, D. (1993b). Knowledge-based
augmented reality. Commun. ACM, 36:5362.
Föck l er , P. et al. (2005). PhoneGuide: museum guidance supported by on-device
object recognition on mobile phones. In M. Billinghurst (ed.) MUM ’05:
Proceedings of the 4th International Conference, (pp. 310), New York,
USA. ACM.
Forisek, M. & Steinova , M. (2013). Explaining Algorithms Using Metaphors.
Springer.
Glassner, A. (2009). Interactive storytelling: Techniques for 21st century fiction.
AK Peters/CRC Press.
Towards Augmented Reality Educational Authoring
605
Hughes, J. F. e. a. (2013). Computer Graphics: Principles and Practice (3rd
Edition). Addison-Wesley Professional.
Huron, D. (2004). Music-engendered laughter: an analysis of humor devices in
PDQ Bach. Proceedings of the 8th International Conference on Music.
pp. 700704.
Kalman, R. E. (1960). A new approach to linear filtering and prediction
problems. Transactions of the ASME Journal of Basic Engineering, 82 (Series
D):3545.
Kato, H. & Billinghurst, M. (1999). Marker tracking and HMD calibration
for a video-based augmented reality conferencing system. In R. Behringer
Gudrun Klinker (ed.) Augmented Reality, 1999. (IWAR ’99) Proceedings. 2nd
IEEE and ACM International Workshop on, (pp. 85 94).
Kaufmann, H. (2009). Construct3d overview. Retrieved from
https://www.youtube.com/watch?v=QvsN-5GykhA (accessed 1 June 2019).
Kaufmann, H., Schmalstieg, D., & Wagner, M. (2000). Construct3d: a
virtual reality application for mathematics and geometry education. Education
and information technologies, 5(4): 263276.
Klein, G. and Murray, D. (2007). Parallel tracking and mapping for small
AR workspaces. In H. Kato, T. Drummond, M. Livingston (Eds) ISMAR
2007. 6th IEEE and ACM International Symposium on Mixed and Augmented
Reality, 2007. (pp. 225234). DOI:10.1109/ISMAR.2007.4538852
Koestler, A. (1964). The Act of Creation. Macmillan USA.
Kostrub, D. & Ostradicky, P. (2019). The augmented reality teaching process
in it’s subjects interpretations. Qualitative teaching and learning research
based on an augmented reality usage. A chapter submitted to (Prodomou,
2019).
Krueger, M. W., Gionfriddo, T., & Hinrichsen, K. (1985). Videoplace
-an artificial reality. In L.Borman, R. Smith (Eds). Proceedings of the
SIGCHI conference on Human factors in computing systems, CHI ’85, (pp. 35
40), New York, NY, USA. ACM. DOI:10.1145/317456.317463
Kusunoki, F., Sugimoto, M., & Hashizume, H. (2002). Toward an
interactive museum guide system with sensing and wireless network
technologies. In M. Milrad, H. Ulrich Hoppe (Eds) Wireless and Mobile
Technologies in Education, 2002. Proceedings. IEEE International Workshop on,
(pp. 99 102).
Lacko, J. (2019). Cultural Heritage Objects in Education by Virtual and
Augmented Reality. In M. Claudia tom Dieck, T. Jung (Eds) Augmented
Reality and Virtual Reality: The Power of AR and VR for Business (pp. 175-
187). Springer, Cham.
Zuzana Berger-Haladová, Andrej Ferko
606
Lanier, J. (2010). Brief biography of Jaron Lanier Retrieved from.
http://www.jaronlanier.com/general.html (accessed 1 June 2019).
Mann, S. (2004). Continuous lifelong capture of personal experience with eyetap.
In J.Gemmell, H. Sundaram (Eds) Proceedings of the the 1st ACM workshop
on Continuous archival and retrieval of personal experiences, CARPE’04,
(pp. 121), New York, NY, USA. ACM.
Milgram, P., Takemura, H., Utsumi, A., & Kishino, F. (1995).
Augmented reality: a class of displays on the reality-virtuality continuum.
In H. Das (ed.) Society of Photo-Optical Instrumentation Engineers (SPIE)
Conference Series, 2351 of Society of Photo-Optical Instrumentation Engineers
(SPIE) Conference Series, (pp. 282292).
Miyashita, T., Meier, P., Tachikawa, T., Orlic, S., Eble, T., Scholz,
V., Gapel, A., Gerl, O., Arnaudov, S., & Lieberknecht, S. (2008).
An augmented reality museum guide. In T. Drummond, S. Julier (Eds)
Proceedings of the 7th IEEE/ACM International Symposium on Mixed and
Augmented Reality, ISMAR ’08, (pp. 103106), Washington, DC, USA.
IEEE Computer Society.
NuFormer (2011). First 3D video mapping projection. Zierikzee: NuFormer.
Retrieved from http://www.nuformer.com/ (accessed 1 June 2019).
Olsson, T. & Salo, M. (2011). Online user survey on current mobile
augmented reality applications. In M. Wiedmer, V. Lepetit (Eds) Mixed and
Augmented Reality (ISMAR), 2011 10th IEEE International Symposium on, (pp.
7584) Basel: IEEE Computer Society.
Papagiannakis, G. et al. (2018). Mixed reality gamified presence and
storytelling for virtual museums. In: N. Lee (ed.) Encyclopedia of Computer
Graphics and Games. Springer. Cham. https://doi.org/10.1007/978-3-319-
08234-9_249-1
Prodromou, T. (ed.) (2019). Augmented reality in educational settings. Brill |
Sense.
Puentadura, R.R. (2018). SAMR: A contextualized introduction. Retrieved from
http://hippasus.com/rrpweblog/archives/2014/01/15/SAMRABriefContextuali
zedIntroduction.pdf (accessed 1 June 2019).
Rizvic, S., Sadak, A., Ferko, A. et al. 2013. A piece of peace in
sWARajevo: Locally and globally interesting stories for virtual museums. In
A. C. Addison, L. De Luca, G. Guidi, S. Pescarin (Eds) Digital Heritage
International Congress (DigitalHeritage) 2, (pp. 445-445).
Qualcomm (2013). Vuforia Retrieved from https://www.vuforia.com/ (accessed 1
June 2019).
Towards Augmented Reality Educational Authoring
607
Qvortrup, L. e. (2001). Virtual interaction: interaction in virtual inhabited 3D
worlds. Springer.
Raskar, R., van Baar, J., Beardsley, P., Willwacher, T., Rao, S., &
Forlines, C. (2003). iL-amps: geometrically aware and self-configuring
projectors. in ACM Transactions on Graphics (TOG). 22(3), July 2003.
ACM. 809-818
Rekimoto, J. (1996). Transvision: A hand-held augmented reality system for
collaborative design. Retrieved from https://www.researchgate.net/
publication/228929153_Transvision_A_hand-held_augmented_reality_
system_for_collaborative_design (accessed 1 June 2019).
Schmalstieg, D. (2016). Portal ARbook. Presentations. Retrieved from
https://arbook.icg.tugraz.at/ (accessed 1 June 2019).
Schmalstieg, D. & H ö l l e r e r , T. (2016). Augmented reality: principles and
practice. Addison-Wesley Professional.
Shneiderman, B. & Plaisant, C. (2005). Designing the user interface:
strategies for effective human-computer interaction. Pearson Education.
Sparacino, F. (2002). The museum wearable: real-time sensor-driven
understanding of visitors’ interests for personalized visually-augmented
museum experiences. In D. Bearman, J. Trant (Eds) Proceedings of Museums
and the Web (MW2002, (pp. 1720).
Sutherland, I. E. (1965). The ultimate display. In W. A Kalenich (ed.) Proceedings
of the IFIP Congress, (pp. 506508). London : Macmillan and Co., 1965.
Sutherland, I. E. (1968). A head-mounted three dimensional display. Proceedings
of the December 9-11, 1968, fall joint computer conference, part I, AFIPS ’68
(Fall, part I), (pp. 757764), New York: ACM.
Takahashi, D. (2009). Microsoft games exec details how project Natal was born
Retrieved from http://venturebeat.com/2009/06/02/microsoft-games-executive-
describes-origins-of-project-natal-game-controls/ (accessed 1 June 2019).
The macula (2010). The 600 years Retrieved from http://vimeo.com/15749093
(accessed 1 June 2019).
The Spitfire Site (2007). Mark II Gyro Gunsight Retrieved from
http://spitfiresite.com/2007/11/mark-ii-gyro-gunsight.html. (accessed 1 June
2019).
Thomas, B., Close, B., Donoghue, J., Squires, J., De Bondi, P.,
Morris, M., & Piekarski, W. (2000). Arquake: an outdoor/indoor
augmented reality first person application. In A. Rawlinson (ed.) Wearable
Computers, 2000. The Fourth International Symposium on, (pp. 139 146).
Zuzana Berger-Haladová, Andrej Ferko
608
Vsauce (2010). 12 best Kinect hacks 2010 Retrieved from
http://www.youtube.com/watch?v=ho8KVOe _y08. (accessed 1 June 2019).
Weiser, M. (1991). The computer for the 21st century. Scientific American.
265(3), 94-104.
Citation: Berger-Haladová, Z., Ferko, A., (2019). Towards Augmented Reality
Educational Authoring. In E. Smyrnova-Trybulska (Ed.) E- Learning and
STEM Education. E-Learning”, 11, (pp. 587-608) Katowice-Cieszyn: Studio
Noa for University of Silesia.
Article
Full-text available
McGraw-Hill Medical). The magazine Scientific American Mind is an excellent way to keep up with recent discoveries. For high school students, if you find this book interesting we encourage you to consider a career as a neuroscientist. Working neuroscientists often find it helpful to have some background in at least a few of the following areas: biology, chemistry, computer science, engineering, genetics, mathematics, physics, and psychology. Come on in, the water's fine! The following references give more details about topics in the book and can be found at http://welcometoyourbrain.com.
Chapter
In this paper, we focus on the use of cultural heritage objects (historical, natural and technical) in the education process. We create universal content that can be used in education in schools, but also at home with the use of virtual and augmented reality. We discuss the potential of the VR and AR tools for better memorising of the information based on the virtual objects and scenes. We describe the selection of individual objects in order to be able to communicate information about individual objects in pairs across the Slovak-Ukrainian border and to use them appropriately on the tools created in the InovEduc project to improve the education at secondary schools on subjects like history, geography, religion, regional education, civics and languages. We also focus on creating content so that it can be used in other areas such as tourism, museums and other cultural institutions.
Book
Lars Qvortrup The world of interactive 3D multimedia is a cross-institutional world. Here, researchers from media studies, linguistics, dramaturgy, media technology, 3D modelling, robotics, computer science, sociology etc. etc. meet. In order not to create a new tower of Babel, it is important to develop a set of common concepts and references. This is the aim of the first section of the book. In Chapter 2, Jens F. Jensen identifies the roots of interaction and interactivity in media studies, literature studies and computer science, and presents definitions of interaction as something going on among agents and agents and objects, and of interactivity as a property of media supporting interaction. Similarly, he makes a classification of human users, avatars, autonomous agents and objects, demon­ strating that no universal differences can be made. We are dealing with a continuum. While Jensen approaches these categories from a semiotic point of view, in Chapter 3 Peer Mylov discusses similar isues from a psychological point of view. Seen from the user's perspective, a basic difference is that between stage and back-stage (or rather: front-stage), i. e. between the real "I" and "we" and the virtual, representational "I" and "we". Focusing on the computer as a stage, in Chapter 4 Kj0lner and Lehmann use the theatre metaphor to conceptualize the stage phenomena and the relationship between stage and front-stage.