Gamiﬁed Augmented Reality Training for An
Assembly Task: A Study About User Engagement
74081 Heilbronn, Germany
74081 Heilbronn, Germany
Abstract—Augmented Reality and Gamiﬁcation are displaying
beneﬁcial effects to enhance user experience and performance
in many domains. They are widespread across many areas like
education, industrial training, marketing, and services. However,
the idea of combining the two approaches for an innovative
training instrument is fairly new, especially in assembly training.
Moreover, learning about the effects of gamiﬁcation on human,
user engagement, in particular, is a complicated subject. There
have been several efforts toward this direction, yet the overall
situation is still nascent. In this work, we present a gamiﬁed
augmented reality training for an industrial task and investigate
user engagement effect while training with the gamiﬁed and the
nongamiﬁed system. The result shows that people perform better
and engage to a greater degree in the gamiﬁed design.
I. INT ROD UC TI ON
AUGMENTED Reality (AR) is growing stronger than
ever. Market research predicts a 70 to 75 billion revenue
for AR by 2023  and by 2019 AR for training, in particular,
will take place in 20% of large enterprise businesses . AR
is the novel technology which superimposes virtual objects
upon the real world subjects or environment while enabling
real-time interactions . In recent years, AR has captured
the research interests in many areas such as education and
training , , assembly and production operations ,
. As a result, the outcome of teaching and learning, skill
acquisition and development as well as user experience have
shown outstanding beneﬁcial effects.
Gamiﬁcation, on the other hand, is the term for adapting the
design elements which commonly characterize entertainment
games into other settings but gaming. While the academic
world is still debating on the consensus of deﬁnition and
scope, the beneﬁts that gamiﬁcation brings are undenialble.
It is not uncommon to say that games are addictive, yet
beyond entertainment purposes, they are believed to better
life in many aspects . Gamiﬁcation’s ultimate goal is to
simulate the fun elements that enhance the user experience,
improve worker productivity or advance student engagement.
Since gamiﬁcation is often mistaken with the meaning of the
“serious game,” which is any full-ﬂedged game that used for
other purposes exceeding pure entertainment, we limit the
work in this paper to the most widely accepted deﬁnition of
Fig. 1. The GAR design with gamiﬁcation elements: points, progress bar and
“Gamiﬁcation is the use of game design elements in non-
Since both AR and gamiﬁcation already have their certain
contribution into the education ﬁeld, in the context of training
especially, it is surprising that gamiﬁed AR systems have
not been popular for training in the production environment.
Accountable for this probably is the ﬁne line between making
work fun and making fun of work . Due to the nature of
productional work, the misuse of the gamiﬁed systems could
take away the user’s focus attention and result in damages or
even injuries. Therefore, here we attempt to form a gamiﬁed
AR application for an assembly training task following special
design requirements for a production environment. Our focus
is on the user engagement aspect because it is an important
factor contributes to the effectiveness of training.
II. RE LATE D WOR K
Although the term “gamiﬁcation” is relatively new, since
around 2003, its applications have already widespread across
many industrial as well as scholarly ﬁelds. Recently in the
Gamiﬁcation 2020 report, Gartner predicted that gamiﬁcation
in combination with emerging technologies will create a
signiﬁcant impact on several ﬁelds including the design of
employee performance and customer engagement platform .
In this context, there are numerous examples of studies for
Proceedings of the Federated Conference on
Computer Science and Information Systems pp. 901–904
ISSN 2300-5963 ACSIS, Vol. 18
IEEE Catalog Number: CFP1985N-ART c
2019, PTI 901
either AR training or gamiﬁed training, yet there was hardly
any work on the combination of those.
A recent survey of Seaborn et al.  provides a good
overview of gamiﬁcation from a Human-Computer-Interaction
perspective in both theoretical and practical lights. The work
showed that gamiﬁcation is primarily practiced in the domain
of education, e-learning especially. In the theoretical founda-
tions, there was a dynamic movement towards carving the
boundaries between gamiﬁcation and other similar concepts.
The applied research, meanwhile, painted a positive-leaning
but mixed picture about the effectiveness of gamiﬁed systems.
Despite usual expectation, similar gamiﬁed designs under
different settings returned clashing result over user experience
along with performance. The reason was believed to be highly
context-speciﬁc requirements. Furthermore, learning about the
effects of gamiﬁcation on the human is a complicated subject.
The overall effort toward this direction is still nascent.
While the gamiﬁed system was well accepted in business
contexts, it is not necessarily the case in production training,
left alone Augmented Reality training. K. Lee  showed that
AR for education and training innovation was leaning towards
the “serious game” pole while gamiﬁcation was left outside
of the picture. According to Lee, AR games were particularly
interested in by both “educators and corporate venues.” A role-
playing game for teaching history , for example, proved
the beneﬁt of enabling students for problem-solving, increas-
ing collaboration and exploration via the virtual identities.
However, whether we like it or not, production training is
different from traditional classroom training. When transform-
ing the operational work into a game, a serious game, there
will always be a risk of taking the focus away from the task at
hand. This is when gamiﬁcation comes to play as integrating
gamiﬁcation can provide the fun aspect while still keeping the
workers’ full attention on the operative job .
Probably the most well-known gamiﬁcation in production
is a series of works from Korn et al. , , , .
The center of his works is to evaluate users’ acceptance of
gamiﬁcation in modern production environments. Different
designs, “Circles & Bars” and “Pyramid,” were proposed .
Both designs were used to visualize work steps as well as their
sequences. Color-coded from dark green to yellow, orange and
read is employed to indicate user speciﬁc time progression.
Later on, they were projected into users’ working space as
an assistive application for impaired individuals. The result
indicated a good acceptance level for gamiﬁcation designs
and the “Pyramid” approach was favorable in general. While
the study showed a promising outcome, it focused on user
acceptance and did not measure the quantitative factor of
gamiﬁcation on task completion time and error rates.
III. IMP LE ME NTATION
In this section, we present the implementation of the
application under study. A process of replacing the battery
for a robot arm was implemented based on the instruction
manual of the Mitsubishi Industrial Robot RV-2F Series .
The application ran on the Microsoft HoloLens . Two
Fig. 2. The NGAR design with no gamiﬁcation elements. Only text instruction
prototypes were made, one with the gamiﬁcation design and
the other without. The designs were named Gamiﬁcation AR
(GAR) and Non-Gamiﬁcation AR (NGAR) according to their
characteristics. Due to Microsoft HoloLens small ﬁeld of
view, around 35 degrees, here we provide the user interfaces
captured from Unity Editor to showcase the whole scene setup.
Figure 1 and Figure 2 illustrate the GAR and NGAR design
A. The application
The process for changing the battery was identically built
for both prototypes. There were 21 actions made up 10
steps. Disassembling the cover of the battery compartment,
for example, included two steps of removing the screws and
removing the cover. While removing each of the screws was
counted as an action.
For navigating the process, we augmented the instruction
text for each step as a head-up display which was always
facing the user at the top right corner of the user view. An
instruction manager was used to control the ﬂow of text
visualization. The requirement from the instruction manual
speciﬁed that the steps of the process had to be performed
in a ﬁxed order that’s why only one instruction was displayed
at a time. The next instruction triggered when the user carried
the current step correctly.
Two main interaction types were used to simulate different
interactions. Air tap  was used for interacting with static
objects (e.g. pressing a button) while we utilized drag and drop
for assembling actions (e.g. removing the screw). Similar to
the real working space, disassembled objects were designed to
be placed at a speciﬁc location. For instance, the screws needed
to be placed inside a designated tray instead of dropped on the
To simulate a sense of reality, sounds such as robot arm
were running or turned off were used.
B. Gamiﬁcation Design
The game design elements were implemented only for the
GAR version. It allows to isolate and analyze the effect
902 PROCEEDINGS OF THE FEDCSIS. LEIPZIG, 2019
of gamiﬁed system on the user. This could be reﬂected by
comparing the outcome of the two experiments.
As a result of Korn’s investigation , gamiﬁcation in the
production environment has its own speciﬁc requirements. To
avoid resistance from users or the potential of taking away
their main focuses, we followed the identiﬁed requirements
in designing gamiﬁed application for production settings.
First, “keep the visualization of gamiﬁcation simple.” This
focuses mainly on avoiding animation, moving elements and
using complex graphical structures. The second and third
requirements come together as “avoid explicit interaction with
gamiﬁcation elements” and “support implicit interaction with
gamiﬁcation elements.” For that matter, in our designs we did
not ask for any user’s effort to direct input or reach out to the
1) Point System: The point system was built based on users’
actions. There was a maximum of 21 points according to 21
actions. Points were rewarded to the user when the action was
done. As the ﬁrst attempt to study the effect of gamiﬁcation
design on user engagement, we did not implement a complex
point system with losing points or rewarding extra points at
2) Progress Bar: While the points were based on actions,
progress bar visualized the steps. As stated as one of the
requirements, the user interface was intentionally kept simple
with only one color. Additional text was in place for indicating
3) Signposting: Signposting aims to direct the user in the
right direction. While users without background knowledge
could be confused with the mechanical part names (e.g. Con-
troller box), signposting highlighted the part corresponding to
the currently displayed instruction. It provided the “just-in-
time” hints for the trainees, especially the totally beginner one.
IV. EXP ER IM EN T DES IG N
The experiment was conducted to investigate how gamiﬁca-
tion in AR training impacts user engagement and performance.
The studies for both conditions (GAR and NGAR) took place
in the same room at our research laboratory. To avoid the
learning effect, we employed the between-group design in
which each participant randomly exposed to only one design,
either GAR or NGAR.
Due to the fact that Microsoft HoloLens requires speciﬁc
hand gestures for interaction, the participants were asked if
they have experience with this device. In the case of none,
the participant used the default HoloLens “Learn gesture”
application. This was especially important because the main
task could not be carried on without this step. Before the
experiment, regardless of the HoloLens experience, we re-
peated the main information about the interactive gestures to
Once the participants were conﬁdent interacting with the
device, the main experiment task proceeded. When the user
hit the “Start” button at the ﬁrst scene of the application, the
timer for measuring task completion time was started until the
last step completed.
As we focused on the user engagement we used a post-
study questionnaire with the reﬁned User Engagement Scale
(UES) . UES is a ﬁve-point rating scale: strongly disagree,
disagree, neither disagree nor agree, agree and strongly agree,
respectively from 1 to 5 point. Given the task was not
complicated, the level of fatigue after that was expected not
to be high so that we decided to use the UES long form (UES
- LF). The UES - LF consists of 30 items covering 4 factors:
1) FA: Focused Attention
2) PU: Perceived Usability
3) AE: Aesthetic Appeal
4) RW: Reward Factor
As constructed in the guide to use of UES, all items were
randomized and the indicators (e.g. AE.1) were not visible to
V. RE SU LTS
Most of the participants reported having little or none expe-
rience with AR technology, in particular, Microsoft HoloLens,
before this experiment. So, a potential novelty effect when
initially establishing interaction with new technology might
inﬂuence the research result. The test population was 22 par-
ticipants with 11 regarding each condition. Participants ages
vary from 18 to 34 years old, 15 male and 7 female subjects.
Although some unease and uncertainty were expressed at the
beginning, all participants were more certain after the learning
Figure 3 displays that the GAR design was rated better
in all sub categories. In general, it was clearly preferred to
the NGAR approach. The overall Engagement score was 15.2
(SD=1.8) in GAR and 13.3 (SD=3.5) in NGAR. However, this
did not make up a statistically signiﬁcant difference between
the two groups. Table I provides the results in more detail,
looking at the average score, standard deviation and also the
result of a t-test for both the overall engagement score and its
The standard deviation in the overall user engagement
score was much lower in the GAR design (SD=1.8), versus
SD=3.5 in NGAR, which shows that the GAR subjects more
homogenously perceived the result throughout the group. This
tendency, lower standard deviation, remained true for all four
subfactors in the GAR design as shown in Figure 3. On the
other side, the opinions of NGAR subjects seem to be more
Looking at the training performance, the difference regard-
ing average task completion time (in seconds) between the two
study conditions is statistically signiﬁcant. The t-test resulted
in p < 0.032. The average time was 306.9 (SD=123.2) and
439.5 (SD=134.4) for GAR and NGAR groups respectively.
This positive outcome probably directly inﬂuenced by the
signposting design element.
VI. DI SC US SI ON A ND FU TU RE WORK
As a preliminary result, this work demonstrates the potential
of gamiﬁed AR training for assembly tasks in improving user
engagement and performance. Nevertheless, there is a need for
DIEP NGUYEN, GERRIT MEIXNER: GAMIFIED AUGMENTED REALITY TRAINING FOR AN ASSEMBLY TASK 903
Fig. 3. User Engagement Score as a bar chart with indicated standard
COM PARI SIO N OF USE R ENG AGE MEN T SCO RE
Factor Mean Score (SD) p value
Design GAR NGAR GAR vs. NGAR
Focused Attention 3.5 (0.6) 3.2 (0.8) 0.418
Perceived Usability 3.7 (0.5) 3.4 (0.7) 0.281
Aesthetic Appeal 3.9 (0.7) 3.3 (1.2) 0.162
Reward Factor 4.0 (0.5) 3.4 (1.2) 0.128
Overall Score 15.2 (1.8) 13.3 (3.5) 0.153
further investigation focusing on both short-term and long-
term training effectiveness. A consideration over skills and
knowledge acquisition should be taken into account. To serve
this goal more complex tasks should be implemented with
a higher level of gamiﬁcation, different training levels and
challenges design for individual speciﬁc demands for example.
As we focused on the improvement of user engagement
in gamiﬁed AR training, we did not take in to account the
isolated effect of how each game design elements affects the
user. As mentioned in the Related Work, gamiﬁcation design is
highly context-speciﬁc so that the next important step will be
a qualitative study on how the users perceive different design
elements and their impacts.
VII. CON CL US IO N
The use of gamiﬁcation in combination with AR for pro-
duction training is still new and its potential needs further
exploration. In this paper, we developed a gamiﬁed training
for an assembly task in AR setting and studied its effects on
The result showed that the users displayed a higher level of
engagement as well as better performance with the support of
gamiﬁed AR training. The statistical analysis, though, did not
indicate a signiﬁcant difference.
While the implementation of gamiﬁcation may not yet
fully integrate into the training process, this work certainly
contributes to the existing knowledge body of gamiﬁed AR
training for production domain. This research area also needs
a greater amount of works to identify its beneﬁts alongside
with how to tackle its challenges.
REF ER EN CE S
 T. R. Azuma, “A Survey of Augmented Reality,” Presence: Teleoperators
and Virtual Environments, vol. 6, no. 4, 1997, pp. 355–385.
 M. Billinghurst, H. Kato and I. Poupyrev, “The Magic Book Moving
seamlessly between reality and virtuality,” IEEE Computers, Graphics
and Applications, vol. 21, no. 3, May/June 2001, pp. 2–4.
 S. K. Ong , M. L. Yuan and A. Y. C. Nee, “Augmented reality applica-
tions in manufacturing: a survey,” International Journal of Production
Research, vol. 46, 2008, pp. 2707–2742.
 W. Friedrich, “ARVIKA: Augmented Reality for Development, Pro-
duction and Service,” The 1st International Symposium on Mixed and
Augmented Reality (ISMAR), 2002, pp. 3–4.
 For AR/VR 2.0 to live, AR/VR 1.0 must die, Digi-
for-ar-vr- 2-0- to-live-ar- vr-1-0-must-die/. [Retrieved May 2019]
 Transform Business Outcomes With Immersive Technology,
Smarter with Gartner, https://www.gartner.com/smarterwithgartner/
 S. Dale, “Gamiﬁcation: Making work fun, or making fun of work?,”
Business Information Review, vol. 31, no. (2), 2014, pp. 82–90.
 Gamiﬁcation 2020: What Is the Future of Gamiﬁcation?, https://www.
gartner.com/en/documents/2226015. [Retrieved May 2019]
 J. McGonigal, “Reality is broken: Why games make us better and how
they can change the world,” New York: Penguin, 2011.
 S. Deterding, D. Dixon, R. Khaled and L. Nacke, “From Game Design
Elements to Gamefulness: Deﬁning Gamiﬁcation,” Proceedings of the
15th International Academic MindTrek Conference: Envisioning Future
Media Environments, MindTrek 2011, vol. 11, 2011, pp. 9–15.
 K. L. Schrier, “Revolutionizing history education : using augmented
reality games to teach histories,” Institute of Technology. Dept. of
Comparative Media Studies, Massachusetts, 2005.
 O. Korn, M. Funk and A. Schmidt, “Design approaches for the
gamiﬁcation of production environments,” in Proceedings of the 8th
International Conference on PErvasive Technologies Related to Assistive
Environments, ACM, New York, NY, USA, 2015, pp. 1–7.
 K. Lee, “Augmented Reality in Education and Training,” TechTrends,
vol. 56, 2012, pp. 13–21, https://doi.org/10.1007/s11528-012-0559- 3
 K. Seaborn and D. I. Fels, “Gamiﬁcation in theory and action: A
survey,” International Journal of Human Computer Studies, vol. 74,
2015, pp. 14–31.
 K. Oliver, “Industrial Playgrounds. How Gamiﬁcation Helps to Enrich
Work for Elderly or Impaired Persons in Production,” In: Proceedings of
the 4th ACM SIGCHI Symposium on Engineering Interactive Computing
Systems, New York, 2012, pp. 313-316.
 O. Korn, M. Funk, S. Abele, A. Schmidt and T. Hörz, “Context-
aware Assistive Systems at the Workplace. Analyzing the Effects of
Projection and Gamiﬁcation,” In: PETRA 14 Proceedings of the 7th
International Conference on PErvasive Technologies Related to Assistive
Environments, ACM, New York, NY, USA, 2014.
 O. Korn, M. Funk and A. Schmidt, “Towards a Gamiﬁcation of Industrial
Production. A Comparative Study in Sheltered Work Environments,”
In: Proceedings of the 7th ACM SIGCHI Symposium on Engineering
Interactive Computing Systems, ACM, New York, NY, USA, 2015.
 Mitsubishi Industrial Robot RV-2F Series: Instruction Manual Robot
Arm Setup & Maintenance, www.geva-roboter.at/ﬁles/rv-2f_series_
robot_arm_setup___maintenance.pdf. [Retrieved May 2019]
 Introduction to the HoloLens, https://msdn.microsoft.com/en-us/
magazine/mt788624.aspx. [Retrieved May 2019]
 H. L. O’Brien, P. Cairns and M. Hall, “A practical approach to measuring
user engagement with the reﬁned user engagement scale (UES) and new
UES short form,” International Journal of Human Computer Studies,
vol. 112, 2018, pp. 28–39.
904 PROCEEDINGS OF THE FEDCSIS. LEIPZIG, 2019