![Evaluation framework [11]](https://www.researchgate.net/profile/Michael-Spitzer-2/publication/335790516/figure/fig4/AS:802606012182528@1568367593197/Evaluation-framework-11_Q320.jpg)
Content uploaded by Michael Spitzer
Author content
All content in this area was uploaded by Michael Spitzer on Sep 13, 2019
Content may be subject to copyright.
A Research Agenda to Deploy Technology Enhanced
Learning with Augmented Reality in Industry
Michael Spitzer∗
michael.spitzer@v2c2.at
Virtual Vehicle Research Center
Graz, Styria, Austria
Inge Gsellmann
inge.gsellmann@v2c2.at
Virtual Vehicle Research Center
Graz, Styria, Austria
Matthias Hebenstreit
matthias.hebenstreit@v2c2.at
Virtual Vehicle Research Center
Graz, Styria, Austria
Stelios Damalas
steliosandreas.damalas@v2c2.at
Virtual Vehicle Research Center
Graz, Styria, Austria
Martin Ebner
martin.ebner@tugraz.at
Graz University of Technology
Graz, Styria, Austria
ABSTRACT
To apply Technology Enhanced Learning (TEL) with Aug-
mented Reality (AR) in industry, a suitable methodology is
necessary. This work focuses on how to deploy and evalu-
ate AR learning scenarios in industrial environments. The
methodology evolved within the two EU projects
FACTS4WORKERS and iDEV40 and has been improved it-
eratively. The rst step is to investigate the use case at the
industry partner. Then the appropriate concept is dened.
The next step is to develop a rst prototype. This proto-
type is then improved during several iterations according to
the feedback of the industry partner. When the prototype
reaches an appropriate Technology Readiness Level (TRL), a
nal evaluation is carried out to verify the software artifact
against the gathered requirements.
KEYWORDS
Augmented Reality, AR, Technology Enhanced Learning,
TEL, Problem-based Learning, On-the-job Training
1 INTRODUCTION AND MOTIVATION
Industry is changing rapidly at the moment. IT-driven, often
highly disruptive changes are aecting many businesses, es-
pecially, but not exclusively, in the manufacturing industry.
This process is often referred to as Industry 4.0 [
12
]. New de-
vices and technologies addressed at both private customers
∗Main Author
Permission to make digital or hard copies of part or all of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for prot or commercial advantage and that copies
bear this notice and the full citation on the rst page. Copyrights for third-
party components of this work must be honored. For all other uses, contact
the owner/author(s).
MuC’19 Workshops, Hamburg, Germany
©
Proceedings of the Mensch und Computer 2019 Workshop on Smart
Collaboration - Mitarbeiter-zentrierte Informationssysteme in der Produk-
tentstehung. Copyright held by the owner/author(s).
https://doi.org/10.18420/muc2019-ws- 300-05
and industry are emerging on the market. Digitizing compa-
nies and keeping pace with new devices, technologies and
software is a great challenge. In order for companies to adapt
quickly, it is very important to train and teach employees
how to use these new technologies. Bridging the gap between
new technologies and their application in industrial and ed-
ucational settings, however, can be a very demanding task.
Therefore, a methodology is needed to integrate new tech-
nologies in educational and training-on-the-job scenarios.
Especially after smart glasses, such as Vuzix, Google Glass,
Microsoft HoloLens and Magic Leap, were rolled out, there
were several attempts to apply these technologies in indus-
try. Stocker et al. [
14
] deployed a software demonstrator for
smart glasses, which presents a checklist in the user’s eld
of vision to help workers doing maintenance and assembly
tasks in the automotive domain. The assistance tool was eval-
uated by experts in automotive production. Although they
were skeptical about interaction techniques such as touch
and speech recognition, the evaluation clearly showed the
potential of these devices and technologies. The context of
use was rated as promising if the quality of the devices were
to improve. The current generation of smart glasses oers
distinct improvements in display quality, speech recognition
and input options. Therefore, it is necessary not to focus
on the specic device but more on the context of use and
whether the smart glasses scenario makes sense. The hard-
ware is rapidly improving and smart glasses manufacturers
are launching new devices at short intervals.
2 TERMINOLOGY
Technology Enhanced Learning
Browne, Hewitt, and Walker [
5
] dene TEL as any online
facility or system which supports learning and teaching. A
great challenge for TEL is that simply digitizing analog learn-
ing material is not sucient. Teaching methodologies also
need to change [
10
]. This does not only apply to education in
general but also to educational scenarios in industry settings.
MuC’19 Workshops, Hamburg, Germany Spitzer et al.
Augmented Reality
An early AR survey was carried out by Azuma [
2
], who
describes AR as a variation of Virtual Reality (VR): While the
user is fully immersed in VR and cannot see the real world
around him or her, AR allows the user to see the real world
with virtual objects superimposed upon or overlaid with the
real world.
3 RESEARCH AGENDA
We developed a research agenda to plan, conduct and eval-
uate TEL AR projects in industry. This research agenda
evolved during two EU research projects, namely
FACTS4WORKERS and iDEV40. In FACTS4WORKERS, we
developed a prototype to help maintenance workers to clean
the lens of a laser cutter [
13
]. In iDEV40, we are develop-
ing an AR-based assembly support system prototype in a
special purpose engineering domain. In FACTS4WORKERS,
we gathered the requirements for a TEL AR system from
the industry partner as a rst step. During an on-site visit,
personas and problem scenarios were identied. The AS-IS
situation was analyzed and a TO-BE situation was developed
[
6
]. After identifying the requirements, we dened a suitable
didactic concept. The main use case focus for this research
agenda is on maintenance and assembly tasks. A signicant
challenge when learning how to do maintenance tasks is that
workers usually do their training directly on the machine.
This means that during this training phase, the machine can-
not be used to produce anything.
The challenge is not only to create a didactic concept which
reduces the downtime of the machine, but also to ensure
learning success. The next step after creating the didactic
concept was developing the rst prototype. This step is very
important in order to verify the gathered requirements. At
the beginning, the actual user requirements are often unclear
and only become more dened once users from the target
group have started testing the rst prototype to get an initial
idea of the features [
1
]. The next step is then to gather feed-
back from the industry partner, especially from users from
the target group. After that, the qualitative feedback from the
users is considered for the next iteration of the prototype. To
improve the prototype iteratively, common agile software de-
velopment practices are followed [
4
]. After several iterations,
when the prototype reaches the desired Technology Readi-
ness Level (TRL), a nal evaluation is carried out to validate
the software artifact against the predened requirements.
The validation framework that was used was developed to t
various kinds of use cases and the dierent industry partners
[8]. Figure 1 summarizes the research agenda.
Figure 1: Research agenda for TEL AR in industry
Use case investigation
In the rst phase, an on-site visit at the industry partner is
necessary to gather the real-world context of the use case.
Current processes and challenges are identied using inter-
views and by observing users from the target group. Another
important aspect of the on-site visit is to get into direct con-
tact with the workers to build trust and to involve them in
the development of the TEL AR prototype.
Didactic concept
With the advent of Industry 4.0, the role of humans in
production-related elds is changing. Workers are confronted
with new challenges, meaning that they need new skills. One
possible solution how to train and teach employees to master
these new challenges is to apply and further develop Tech-
nology Enhanced Learning in industry. One particular issue
is how to support the digitization of domain-specic knowl-
edge without losing any of it. [
7
]. We use a problem-based
learning setting as a starting point [
3
]. Additionally, we con-
nect a virtual classroom scenario with an in-situ learning
scenario at the workplace. The didactic concept is separated
into two TEL learning scenarios. First, participants study
learning material such as manuals or other documentation.
The rst TEL phase is a virtual learning scenario in a virtual
learning environment where the training object is displayed
as a hologram in the eld of view of the learner. The mainte-
nance or assembly artifact can be investigated in 3D from
dierent angles and viewing directions. An animation shows
the steps that are necessary to complete the learning scenario.
The hologram is enriched with symbols and text annotations.
Working with teaching material in 3D has the advantage
A Research Agenda to Deploy Technology Enhanced Learning with Augmented Reality in Industry MuC’19 Workshops, Hamburg, Germany
that the 3D animation can be observed from dierent angles
which could improve the learning experience. The success
of such learning scenario depends on the spatial abilities of
the learner. Learners with low spatial capabilities tend to
be cognitively overloaded by the learning situation [
9
]. In
this case other learning materials and scenarios should also
be considered. The classroom scenario works in any envi-
ronment. The real production machine, or any other tools
or equipment are not needed. After the classroom training,
a feedback and reection phase follows to ensure that the
virtual learning situation was successful. The same situation
is then played out again for training at the machine or as-
sembly station. This training-on-the-job scenario is the nal
step before the participants are able to perform the task by
themselves. Figure 2 summarizes the didactic concept. The
TEL software used is the same in both scenarios. In the rst
scenario, the TEL software is a virtual setting. For the sec-
ond scenario, AR is used to overlay additional information
and training material above real-world objects. To enable an
easy transition between the rst and the second scenario,
the same user interface (UI) is used within the TEL software.
Figure 2: Didactic concept
AR Prototype
In both projects, the Microsoft HoloLens is used to implement
the prototypes. The generic methodology can be applied to
any kind of AR device. An advantage of using smart glasses
is that the users have their hands free to conduct the learning
situation. The key feature of the TEL AR software artifact
is that the UI is consistent for both TEL scenarios. Figure 3
shows the UI of the application. The UI controls the mainte-
nance or assembly instruction animations. The animations
can be paused, forwarded, rewound and repeated. The UI is
based on the UI of commodity video players so that it can
be easily recognized by the users. The menu can be freely
positioned in space to prevent it from blocking the line of
sight of the user. The 3D cube can represent any kind of
3D object, such as a machine that needs maintenance or an
actual product that is being assembled. The opacity of the
computer-generated object can be changed using the cube
symbols at the bottom.
Industry partner feedback
The AR prototype is tested at the industry partner and is
then improved according to the feedback provided by users
from the target group from industry. Several iterations may
be necessary before an appropriate TRL level is reached and
before the AR prototype is ready for the nal evaluation.
Figure 3: AR Prototype - UI controls
Final Evaluation
Within the FACTS4WORKERS project, an evaluation frame-
work was developed to validate the software prototypes
deployed at the industry partners. The framework separates
the evaluation into two dierent strategies. The rst strat-
egy is an impact analysis. The second strategy is a qual-
ity validation. Both strategies are divided in human-driven
approaches and data-driven approaches. Human-driven ap-
proaches are surveys, interviews and observations. Data-
driven approaches include log analysis and application data.
Data-driven approaches are often available at a higher level
of maturity of the software artifact. Human-driven approaches
also work during the very early stages of the project [
8
]. Fig-
ure 4 summarizes the evaluation methodology. We gathered
feedback from several industry partners according to which
the FACTS4WORKERS prototypes are a good t for the de-
ned use cases. As a result, a start-up that is now using the
prototype to build products has emerged from the research
project.
4 CONCLUSION
To implement TEL with AR in industry, it is necessary to
have an adequate research agenda. The research agenda
presented here was used in the FACTS4WORKERS project to
deploy several TEL AR prototypes at several industry partner
sites and is now also to be used in the iDEV40 project. The
prototypes were evaluated by users from the target group
and were improved iteratively. During the nal evaluation
in the FACTS4WORKERS project, the software prototype
MuC’19 Workshops, Hamburg, Germany Spitzer et al.
Figure 4: Evaluation framework [11]
could be validated against the requirements dened in the
rst phase of the project.
ACKNOWLEDGMENTS
The project FACTS4WORKERS has received funding from
the European Union’s Horizon 2020 research and innovation
programme under grant agreement No 636778. The project
iDEV40 has received funding from the Electronic Component
Systems for European Leadership Joint Undertaking under
grant agreement No 783163. This Joint Undertaking receives
support from the European Union’s Horizon 2020 research
and innovation programme and Austria, Spain, Finland, Ire-
land, Sweden, Germany, Poland, Portugal, Netherlands, Bel-
gium, Norway. The publication was written at VIRTUAL
VEHICLE Research Center in Graz and partially funded by
the COMET K2 - Competence Centers for Excellent Tech-
nologies Programme of the Federal Ministry for Transport,
Innovation and Technology (bmvit), the Federal Ministry
for Digital, Business and Enterprise (bmdw), the Austrian
Research Promotion Agency (FFG), the Province of Styria
and the Styrian Business Promotion Agency (SFG).
REFERENCES
[1]
Maryam Alavi. 1984. An Assessment of the Prototyping Approach to
Information Systems Development. Commun. ACM 27, 6 (June 1984),
556–563. https://doi.org/10.1145/358080.358095
[2]
Ronald T Azuma. 1997. A survey of augmented reality. Presence:
Teleoperators & Virtual Environments 6, 4 (1997), 355–385.
[3]
Howard S Barrows, Robyn M Tamblyn, et al
.
1980. Problem-Based
Learning - An Approach to Medical Education. Springer Publishing
Company, New York.
[4]
Kent Beck, Mike Beedle, Arie Van Bennekum, Alistair Cockburn, Ward
Cunningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew
Hunt, Ron Jeries, et al
.
2001. Manifesto for agile software develop-
ment. (2001).
[5]
Tom Browne, Roger Hewitt, and Richard Walker. 2008. 2008
Survey of Technology Enhanced Learning for higher education
in the UK. https://www.ucisa.ac.uk/publications/~/media/
290DD5217DA5422C8775F246791F5523.ashx
[6]
Jonathan Denner, Peter Heinrich, Constantin Heldman, and Alexander
Richter. 2015. First version of requirements of workers and organisa-
tions. Project Report-FACTS4WORKERS: Worker-Centric Workplaces in
Smart Factories (2015).
[7]
Andreas Engelmann, Peter Heinrich, and Gerhard Schwabe. 2017. Mo-
biles Lernen für Industrie 4.0: Probleme, Ziele, Lernarrangements. In
Towards Thought Leadership in Digital Transformation: 13. Interna-
tionale Tagung Wirtschaftsinformatik, WI 2017, St.Gallen, Switzerland,
February 12-15, 2017., Jan Marco Leimeister and Walter Brenner (Eds.).
https://aisel.aisnet.org/wi2017/track07/paper/9
[8]
Lea Hannola, FJ Lacuera-Pérez, Melanie Steinhüser, Kirsi Kokkonen,
Ville Ojanen, and Marlene Schaer. 2017. An evaluation framework
for worker-centric solutions in production environments. DEStech
Transactions on Engineering and Technology Research icpr (2017).
[9]
Thomas Huk. 2006. Who benets from learning with 3D models? The
case of spatial ability. Journal of computer assisted learning 22, 6 (2006),
392–404.
[10]
Adrian Kirkwood and Linda Price. 2014. Technology-enhanced learn-
ing and teaching in higher education: what is ’enhanced’ and how do
we know? A critical literature review. Learning, Media and Technol-
ogy 39, 1 (2014), 6–36. https://doi.org/10.1080/17439884.2013.770404
arXiv:https://doi.org/10.1080/17439884.2013.770404
[11]
Francisco Lacueva-Pérez, Lea Hannola, Jan Nierho, Stelios Damalas,
Soumyajit Chatterjee, Thomas Herrmann, and Marlene Schaer. 2018.
Comparing Approaches for Evaluating Digital Interventions on the
Shop Floor. Technologies 6, 4 (2018), 116.
[12]
Heiner Lasi, Peter Fettke, Hans-Georg Kemper, Thomas Feld, and
Michael Homann. 2014. Industry 4.0. Business & Information Systems
Engineering 6, 4 (2014), 239–242. https://doi.org/10.1007/s12599- 014-
0334-4
[13]
Michael Spitzer, Marlene Schaer, and Matjaž Milfelner. 2018. Seamless
learning in the production. Mensch und Computer 2017-Workshopband:
Spielend einfach interagieren 13 (2018), 211.
[14]
Alexander Stocker, Michael Spitzer, Christian Kaiser, Manfred Rosen-
berger, and Michael Fellmann. 2017. Datenbrillengestützte Checklisten
in der Fahrzeugmontage. Informatik-Spektrum 40, 3 (2017), 255–263.
