ArticlePDF Available

Augmented Reality Application to Support Remote Maintenance as a Service in the Robotics Industry

Authors:
  • University of Patras Lab. For Manufacturing Systems and Automation (LMS)
  • Laboratory for Manufacturing Systems and Automation

Abstract and Figures

Maintenance of manufactured products is among the most common services in industry and its cost often exceed 30% of the operating costs. Modern manufacturing companies are shifting their focus from products to combined ecosystem of Products- Service Systems (PSS). Towards that end, the main objective of this work is to develop a cloud-based service-oriented system that implements AR technology for remote maintenance by enabling cooperation between the on- spot technician and the manufacturer. The proposed system includes the methods for the record of the malfunction by the end user, the actions required by the expert so as to provide instructions in an Augmented Reality application for maintenance, as well as the cloud- based platform that will allow their communication and the exchange of information. In addition to the above, the proposed system consists of smart assembly/disassembly algorithms for automated generation of assembly sequences and AR scenes and improved interface, aiming to maximize existing knowledge usage while creating vivid AR service instructions. The proposed system is validated in a real-life case study following the requirements of a robotics SME.
Content may be subject to copyright.
Available online at www.sciencedirect.com
2212-8271 © 2017 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
Peer-review under responsibility of the scientifi c committee of The 50th CIRP Conference on Manufacturing Systems
doi: 10.1016/j.procir.2017.03.154
Procedia CIRP 63 ( 2017 ) 46 51
ScienceDirect
The 50th CIRP Conference on Manufacturing Systems
Augmented reality application to support remote maintenance as a service
in the Robotics industry
D. Mourtzisa*, V. Zogopoulosa, E. Vlachoua
aLaboratory for Manufactu ring Systems and Automation, Depa rtment of Mechanical Enginee ring and Aeronautics, University of Patras, 26500, Rio Patras,
Greece
* Corresponding author, e-mail: mourtzis@lms.mech.upatras.gr, Tel: +30 2610 910160, Fax: +30 2610 997744
Abstract
Maintenance of manufactured products is among the most common services in industry and its cost often exceed 30% of the operating costs.
Modern manufacturing companies are shifting their focus from products to combined ecosystem of Products- Service Systems (PSS). Towards
that end, the main objective of this work is to develop a cloud-based service-oriented system that implements AR technology for remote
maintenance by enabling cooperation between the on- spot technician and the manufacturer. The proposed system includes the methods for the
record of the malfunction by the end user, the actions required by the expert so as to provide instructions in an Augmented Reality application
for maintenance, as well as the cloud- based platform that will allow their communication and the exchange of information. In addition to the
above, the proposed system consists of smart assembly/disassembly algorithms for automated generation of assembly sequences and AR scenes
and improved interface, aiming to maximize existing knowledge usage while creating vivid AR service instructions. The proposed system is
validated in a real-life case study following the requirements of a robotics SME.
© 2017 The Authors. Published by Elsevier B.V.
Peer-review under responsibility of the scientific committee of The 50th CIRP Conference on Manufacturing Systems.
Keywords: Augmented Reality; Remote Maintenan ce; Product Service System
1. Introduction
In the era of mass production and in the emerging era of mass
personalization [1][2] with increased competition between
manufacturers, it has become really important to come up with
new ways to improve customer satisfaction. Many
manufacturers attempt to achieve that by offering high- quality
Product Service Systems (PSS) throughout the product’s life
cycle [3]. PSS is a value proposition strategy that offers
products-services and is designed to be: competitive, satisfy
customer needs, and have a lower environmental impact than
traditional business models. Maintenance service is one of the
most commonly used, especially in products that require
maintenance frequently. Towards supporting this interface
between IT systems and avoiding isolated work, Cloud
Manufacturing has been regarded as an enabler and has already
formed the basis for new business models [4]. Furthermore, this
approach takes into account the usage of CAx systems in
manufacturing and the need to develop new instructions
providing systems that are more intriguing and more accurate
than the traditional ones. Augmented Reality (AR) is a rapidly
evolving technology that is used more and more in different
manufacturing fields in the last few years [5], [6]. Using the
CAD three-dimensional geometries and assembly information,
enriched with intelligence, the manufacturer can create a series
of AR scenes to support service sequence.
Targeting the Cloud manufacturing and remote
maintenance, the main objective of this work is to develop an
internet-based, service-oriented system that implements AR
technology for enabling tele- maintenance by cooperation of the
end user and the manufacturer.
2. State of the art
Maintenance is a core activity of the production lifecycle
since it accounts for as much as 60 to 70% of its total costs [7].
This has led to increased need for maintenance planning
through product’s life cycle and the implementation of more
© 2017 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
Peer-review under responsibility of the scientifi c committee of The 50th CIRP Conference on Manufacturing Systems
47
D. Mourtzis et al. / Procedia CIRP 63 ( 2017 ) 46 – 51
and more new technologies (cloud manufacturing [4],
Dynamically Adaptive Systems for self- maintenance [8],
machine monitoring [7]). Despite the effort to limit machine
downtime [9], most of service impact on productivity accounts
for unexpected breakdown, as it cannot be predicted and
quantified in terms of time and required effort. And despite
some effort on the field, it requires a time-consuming process
which has a negative impact on machine availability.
Augmented Reality (AR) is another enabling technology used
for dealing with the increasingly complex maintenance
procedures [10]. Either by using head-mounted displays [11],
[12] or portable devices [13] a number of solutions have
emerged, testing various ways of AR system- user interaction
(voice commands, gestures, devices- hosted menus). The
potential of the newly introduced technology in supporting
maintenance tasks is renowned even by large manufacturing
companies (BMW [14], Bosch [15]). More recently, the
concept of enabling communication between an expert and the
on- spot technician (tele- maintenance) has arisen [16], [17]
delivering some promising results in synchronous and
asynchronous information exchange.
Apart from maintenance, AR has found fertile ground in
other fields of manufacturing. Firstly, AR can be used as a
mean to vividly project the current status of a warehouse or a
production line, allowing constantly monitoring its current
status, communication and planning [18], [19]. Secondly,
Augmented and Virtual Reality have been proved valuable
tools in prototyping and collaborative design [20]. They allow
the fast and costless creation of visual prototypes that can be
manipulated by more than one user and also, overlaid on top of
existing products so as to facilitate customization and reusable
engineering [21], [22]. Moreover, Virtual and Augmented
Reality have been widely used as means of training technicians
in performing assembly tasks [23]. Those technologies provide
an intriguing experience for the technician [24] and thus, they
are more efficient than the traditional methods [25], [26].
Meaningful information generated by IT tools should be
seamlessly shared across the enterprise in order to support
different business functions. Cloud manufacturing enables this
ubiquitous information provision and enables the creation of
intelligent factory networks reshaping the manufacturing
business models. Cloud computing systems and cloud
manufacturing may play a critical role in the realisation of
“Design Anywhere Manufacture Anywhere” philosophy [4].
Another recent study [27] presented the key benefits to
manufacturing as a result of adopting the Cloud technology
such as scalability to business size and needs, ubiquitous
network access [28], and visualisation. However, security and
data protection are still challenges that need to be addressed
[29]. Issues, such as the resource location multi-tenancy and
authentication also need to be tackled in a combined way [30].
The literature review makes apparent that Augmented
Reality systems are welcomed in manufacturing. A field that
already has implemented cloud features in many of its
applications. The contribution of this development compared
to existing approaches is the creation of an asynchronous AR
remote maintenance support system that implements cloud-
based communication between the end user and the
manufacturer that facilitates the reuse of existing knowledge.
In addition to that, the developed internet-based and service-
oriented system is supported by an assembly/disassembly
algorithm that enables the automated generation of the AR
scenes and increases the level of automation. The implemented
platform is designed to be provided for Product-Service
support throughout a product’s life cycle so as to reduce the
impact of Mean Time to Repair in machine availability,
especially in unexpected breakdowns, where external expert
contribution in malfunction detection and service sequence
may be needed. The developed platform utilizes a cloud
database that enables ubiquitous data access and permits the
technician to upload the malfunction report and receive the
corresponding service sequence in AR scenes by the
manufacturer easily. Moreover, the cloud platform facilitates
the supervising mechanic in AR scene creation by keeping
record of older service sequence that can be fully or partially
reused.
3. Architecture design of tele- maintenance system
This paper proposes an innovative Product- Service System
that enables tele- maintenance support through Augmented
Reality scenes. The platform includes the deployment of a
cloud system that will facilitate the communication between
the on- spot technician and the expert by enabling feedback
reports and maintenance instructions exchange. Fig. 1 presents
the architecture of the proposed system and the data exchange
between the technician and the manufacturer representative
mechanic. In order to achieve the proper functionality of the
proposed platform, some key features are established. Each
time the remote maintenance framework is called, a three-step
procedure is required: (i) malfunction report composition, (ii)
diagnosis and AR maintenance instruction generation and, (iii)
maintenance and evaluation.
The first step in the proposed system is registering the
malfunction report. Whenever a service is required, regardless
Fig. 1. Architecture of the developed framework
Fig
1
Architecture of th e developed framework
48 D. Mourtzis et al. / Procedia CIRP 63 ( 2017 ) 46 – 51
if concerning scheduled maintenance or unexpected
malfunction, a service report is created prior to any other action.
In traditional methods, this is a written report, created by the
on- spot technician which describes the problem in text, reports
any parts that need to be replaced and, in some cases, defines
all the actions needed to address it. In the proposed approach,
the technician takes advantage of the communications
technologies to create a digitized, data- rich report.
The malfunction report application provides the technician
with a set of features that facilitate accurate malfunction
capturing, which is essential for remote failure diagnosis. The
application is designed to be used through either the proposed
AR hardware setup or solely using a mobile device. When on-
spot, the operator may record the malfunction by writing
explanatory text, which may include sensor data or description
on the technician’s actions to address the issue, by taking
photographs of the malfunctioning or broken down parts, and
by recording sound, which may include voicemails or machine
noise, using the malfunction report application. The data are
then stored locally on the device and uploaded to the cloud
platform. The maintenance support provider is notified that a
new malfunction report is uploaded.
After viewing the malfunction report, the maintenance
support provider begins the main part of the service;
malfunction cause identification and AR instructions
generation. Remote maintenance may be used to cover both
those needs. On the one hand, the AR equipment allows the on-
spot technician to create an enhanced failure report which,
using the cloud-based feedback mechanisms, can be sent to the
manufacturer in short time. In some cases, the expert has to
guide the on-spot technician in order to determine the cause of
the malfunction; a procedure that may require to exchange
feedback on how to gather more information on the problem.
To achieve that, the supervising expert may provide some AR
instructions to the technician on how to gather additional
information and ask for new report. Uncommon failures require
increased amount of time to detect with traditional practice and
may, even, require the expert to perform on- spot inspection.
Thus, remote diagnosis drastically decreases down time.
On the other hand, Augmented Reality is used for visualizing
the repair sequence. The maintenance expert has to create an
AR instruction scene sequence, which is adjusted to the
problem. To achieve that, he takes advantage of two system
features: knowledge reusability and automatization. First of all,
the expert can consult and reuse similar scenes that are already
created for similar tasks, such as lubrication or screw
tightening, which can be stored in the cloud database.
Moreover, available in the cloud database, the maintenance
expert may find the CAD of the product’s parts and a set of
utilities. These utilities include pre-created part movement,
scripts, as well as the GUI that allows the technician to interact
with the system. Additionally, an algorithm that automates
scene creation in assembly procedures [31] has been
implemented. The algorithm receives as input a CAD file of a
mechanism and, through consequently moving each part, finds
the sequence of assembly/disassembly process needed to be
followed. Moreover, the algorithm defines the axis and
direction of assembly and disassembly of each part and assigns
the fasteners (e.g. bolts, screws) to their corresponding parts.
Therefore, the maintenance expert may quickly create AR
instructions, divided in perceivable steps, for the assembly and
disassembly tasks, that account for a large part of the
maintenance procedure. The implementation of these
components in the proposed system aims to improve the
efficiency of providing remote maintenance support by
reducing the required time, while also reducing the effort and
AR expertise required from the maintenance expert for
designing AR scenes and user interfaces.
Aiming to further reduce the response time of the
maintenance support system, each AR maintenance sequence
that is created to address a maintenance task will be stored in
the cloud database. The same malfunction may come up for
more than one customers. Especially scheduled maintenance
tasks, tend to occur more than once in a product’s life cycle. By
introducing proper organizing of the existing AR maintenance
instructions in the cloud database (filing per targeted machine
and naming according to the task), the maintenance support
provider will be available to instantly recall existing AR
sequences. As a result, AR maintenance sequence reuse is
expected to increase efficiency in maintenance tasks and reduce
machine downtime.
After completing the AR maintenance instructions creation,
the maintenance expert sends them to the on-spot technician.
The communication is again realized through the cloud
platform. The technician is notified that the instructions are
available, downloads them and performs the maintenance task.
After completing the task, the technician verifies that the
machine is now fully operational.
4. Implementation of the proposed framework
4.1 Hardware Implementation
Aiming to use a hardware setup that allows the user to move
freely and in the same time enjoy a highly usable interface, the
combined use of AR goggles and mobile device is preferred
[19]. In order to carry out the proposed system, three devices
have been used: a set of optical see-through Augmented Reality
goggles, a laptop PC and a mobile device. Aiming to provide
the end- user with a high quality visual result that also permits
him to maintain visual contact with the potentially dangerous
environment and not occupy his hands, a set of AR goggles
have been used for this system; Vuzix™ Star 1200XL [32].
Secondly, the system described in this paper requires the use of
a laptop PC (host- PC). This computer is responsible for
executing the AR application, communicating with the mobile
device and with the cloud platform. The last part, the mobile
device, hosts an interface that allows the operator to interact
with the AR application through menus. On top of that, it
allows the operator to add information concerning the task. As
stated before, the mobile device can also be used an
independent tool for feedback recording by the operator, which
provides increased mobility. For the needs of this paper,
Nexus 4 Android smartphone was used.
The proposed system requires the devise to work together so
as to provide the desired result. To achieve that, the HMD and
the mobile device need to be wired to the host- PC. The
operator wears the HMD on his head, adjusts the AR goggles
49
D. Mourtzis et al. / Procedia CIRP 63 ( 2017 ) 46 – 51
and straps the mobile device to his arm. This configuration
is designed so as not to limit his mobility. The proposed
hardware setup can be seen in Fig. 2 below.
4.2 Software Implementation
For the development of this system two software packages
were used. In order to achieve high quality geometry rendering,
a commercial cross-platform game creation system was used:
Unity 3D™ [33]. It includes a game engine and integrated
development environment (IDE) which allows scripting in
three programming languages (C# was used for the developed
platform). The criteria that led to this choice are:
x It supports a wide variety of object formats permitting
extracting the 3D geometries from the CAD files.
x It is able to access dynamic file formats, such as .dll and
.xml, and connect to web URLs
x It supports development not only on Windows but also on
Android™ and iOS™ applications, so as to create
applications that can also be implemented directly to mobile
devices
Tracking is a crucial part in Augmented Reality as it
positions the virtual environment in the real surroundings of the
user [34]. To achieve tracking, it uses the HMD’s camera to
recognize predefined frame markers in user’s field of view. The
transformation T between a camera and a marker is:
ݔൌܶܺ
Where: X is a point in world coordinates, xc is its projection in
ideal image coordinates and T is the pose matrix.
Transformation T consists of translation vector t and a rotation
matrix R:
ݔܴȁݐሿൈܺݔݕݖ቉ൌݎݎݎݐ
ݎݎݎݐ
ݎݎݎݐ൩൦ܺ
ܻ
ܼ
ͳ
In addition to that, in order to map between frame marker ideal
image (xc) and observed pixel coordinates (xpix) a camera
calibration matrix K is used so:
ݔ௣௜௫ܭൈݔฺ൭ݔ௣௜௫
ݕ௣௜௫
ͳ൱ൌ݂Ͳ݌
Ͳ
Ͳ݂݌
Ͳ
ͲͲͳͲ
ݔ
ݕ
ݖ
As a result, when the camera detects the frame marker:
ݔܭൈܶൈܺ
Where xi are the positions of the four corners of the marker
in the camera image and Xi are their corresponding world
coordinates (Fig. 3).
The system then calculates M = K × T which is the final
transformation matrix and applies it to all visualized
geometries, so as to estimate their projection on the virtual
environment [35]. With the aim of creating a system whose
tracking is accurate and robust, PTC Vuforia SDK, a
commercial extension for Unity™, was selected [36].
5. Product Service System (PSS) model
The concept of the PSS is a special case of servitization [37].
PSS is a newly introduced strategy of combining the product
and the set of services that come with it, in one bundle. This
way the manufacturers shift from selling just a product to
selling a functionality or a service, which is provided by the
product. The manufacturer is responsible for all the actions that
will secure that the provided service remains adequate
throughout the products’ lifecycle [38].
The proposed platform aims to offer a value- adding,
product-oriented service [38] that will benefit both the
customer, who enjoys robust and high- quality maintenance
support, and the OEM/ PSS provider, who gains a critical
advantage compared to competitor companies as he is more
involved in an extended part of his products’ lifecycle [39].
Thus, the PSS provider, exploiting cloud communication and
data exchange, may easily interact with the customers,
targeting multiple markets with little cost, controlling a large
portion of after sales product-related services, increasing the
profits, gathering crucial product design feedback (design for
maintainability) and creating a stronger connection with the
customers. The increased level of automation and the high
efficiency of maintenance service of the developed platform
allows its implementation in existing business models. Ease of
implementation and low implementation cost was one of the
main concerns of the developed framework. This is highly
important especially for SMEs, as they can increase their
influence in the market by providing their customers with
unified service solutions of high efficiency together with their
products. The implementation of such high-end services allows
Fig. 2. On - spot operators’ hardware setup
Fig. 3. Architecture of the AR scene
50 D. Mourtzis et al. / Procedia CIRP 63 ( 2017 ) 46 – 51
SMEs to increase customers’ satisfaction, withstand
competition pressure and expand their sales networks.
6. Robotics & Automation industrial case
The developed framework was implemented on a case study
provided by a robotics SME (Fig 4). The case concerns a real-
life maintenance sequence, on which the manufacturer is
usually called to deploy specialized personnel that must go on
the spot and maintain the machine. More specifically, the
maintenance case refers to a battery pack replacement of an
industrial robot installed in a Near East country, in a city
1100km away from the use case provider facilities.
This task is crucial for securing robot’s continuous function.
It commonly requires technicians from the robotics SME to go
where the robot cell is installed and perform the tasks. The
selected task requires simple actions by the technician, thus can
be easily explained remotely. At the same time, it includes all
the features that need to be tested. At first, since the
maintenance tasks consisted solely of assembly/ disassembly
tasks, the smart algorithm automatically created most of the AR
scenes. In Table 1, the Excel file generated by the assembly/
disassembly algorithm for this task is presented.
Part Name
+X
-X
+Y
-Y
+Z
-Z
Tier
Num
Type
Robot.1
0
Base
BatteryBase.1
0
1
0
0
0
0
1
Part
Battery.1
0
0
1
0
0
0
2
Part
Battery.2
0
0
1
0
0
0
2
Part
Battery.3
0
0
1
0
0
0
2
Part
Battery.4
0
0
1
0
0
0
2
Part
Table 1. Assembly steps table based on the smart assembly/disassembly
precedence algorit hm
In the first six columns, the axis and direction of part
disassembly is indicated. Moreover, the sequence of
disassembly is presented in the “Tier Number” column.
Exploiting this information, the developed system may quickly
and automatically create the assembly/ disassembly AR scenes,
drastically reducing the time required for creating maintenance
instructions.
Moreover, the interface is used not only to change scenes but
also to input information that cannot be known in AR
instructions creation, such as which battery connector is free.
This is a crucial part of this maintenance task as the new
batteries need to be connected before unplugging the old ones.
Based on that input, the system alters the projected AR
instructions in the next steps. The steps of the procedure were
provided by corresponding industrial partner.
With the current approach, the robotics SME deploys a
maintenance expert to go on-spot to perform the maintenance
task. This procedure costs 1370€ and requires 9 hours from the
time the malfunction is reported until the machine is back
online. Applying the proposed approach, the cost is reduced to
150€ and the required time is decreased to 2 hours (Fig. 5).
Especially in the case that the task has re-occurred in the past
and can be recalled, omitting the time required for creating AR
instructions, the required time is further reduced. Thus, the
positive impact of applying the proposed framework
application is apparent.
Fig. 5. Required cost and time- Comparison of the two methods
7. Conclusions and future work
This paper presents the development and testing of an
Augmented Reality remote maintenance platform that can be
used for providing PSS maintenance service. A cloud platform
was also implemented as a communication enabler and to assist
existing knowledge reuse. This study also takes into account
algorithms which increase the efficiency of the procedure and
Fig. 4. The sustain able Product Service System
51
D. Mourtzis et al. / Procedia CIRP 63 ( 2017 ) 46 – 51
the automation level by reducing the actions and expertise
required by the engineer to create the service sequence
instructions which can be effortlessly perceived by less
experienced technicians. In addition to that, through the
proposed remote approach the required maintenance time and
cost is highly reduced. Finally, manufacturing companies can
provide remote maintenance as a service, increasing their
customer’s satisfaction and delivering added value solutions.
Future work could focus on integrating the existing system
and other systems in an enterprise, increasing interoperability.
In addition to that, it is essential to reduce AR scenes designer’s
cognitive load by increasing the automatization of the
procedure.
Acknowledgements
The work presented in this paper is partially supported by
the European Union’s Horizon 2020 research and innovation
project “Customer-driven design of product-services and
production networks to adapt to regional market requirements-
ProRegio” (GA No: 636966).
References
[1] Chryssolouris G. Manufacturing Systems: Theory and Practice, 2nd Ed.
New York: Springer-Verlag; 2006.
[2] Mourtzis D, Doukas M, The evolution of manufacturing systems: From
craftsmanship to th e era of customisation , Design and Management of
Lean Production Systems, IGI Global, Chapter 1, 2014
[3] Takata S, Kimura F, Van Houten FJAM, Westkämper E, Shpitalni M,
Ceglarek D, Jay Lee. Maintenance: Changing role in life cycle
management, CIRP Annals Manufacturi ng Technology; 2004.
[4] Xu X. From cloud computing to cloud manufacturing , Robot and
Computer-Integrated Manufacturing 2014; 28(1): pp. 75-86
[5] Ong SK, Yuan ML, Nee AYC. Augmented Reality Applications in
Manufacturing: A Survey, Int J. Production Research 2008;46: pp. 2707-
2742.
[6] Nee AYC, Ong SK, Chryssolouris G, Mourtzis D. Augmented Reality
applications in design and manufacturing, CIRP Annals- Manufacturing
Technology 2012;61, pp. 657 679.
[7] Mourtzis D, Vlachou E, Milas N, Xanthopoulos N. A cloud-based approach
for maintenance of machine tools and equipment based on shop-floor
monitoring, Procedia CIRP; pp. 655-660
[8] Zhang J, Goldsby HJ, Cheng BHC . Modular verification of dyna mically
adaptive systems, Proceedings of the 8 th ACM International Conference
on Aspect-Oriented Software Development 2009; pp. 161-172.
[9] Albers A, Schmidt G. Scheduling with Unexpect ed Machine Breakdowns.
Computer Science, Proceedi ngs of the Third Internati onal Workshop on
Approximation Algorithms for Combinatorial Optimization Problems:
Randomization, Approximation, and Combinatorial Algorithms and
Techniques, Springer-Verlag 1999; 1671, pp. 269-280.
[10] Azuma RT. A survey of augmented reality. Presence: Teleoperators and
virtual environments. 1997;6 (4): pp.355-85.
[11] Feiner S, MacIntyre B, Seligmann D. Karm a (knowle dge-based
augmented reality for maintenance assistance); 1993
[12] Schwald B, De Laval B. An augmented reality system for training and
assistance to maintenance in the industrial context; 2003.
[13] Henderson, S., Feiner, S., Augmented Reality for Maintenance and Repair
(ARMAR), Final project report; 2010.
[14] Platonov J, Heibel H, Meier P, Groll mann B. A mobile markerless AR
system for maintenance and repair, Proceedings of the 5th IEEE and ACM
International Symposium on Mixed and Augmented Reality, Washington,
DC, USA 2006; pp 105108.
[15] Common Augmented Reality Platform CAP, Bosch, http://oe.bosch-
automotive.com/en/bosch-automotive-aftermarket
[16] Bottecchia S, Cieutat JM., Jessel JP. T.A.C: Augmented Reality System
for Collaborative Tele-Assistan ce in the Field of Maintenance thr ough
Internet, France 2010; pp.1-7
[17] Alem L, Tecchia F, Huang W. Remote Tele-assistance System for
Maintenance Operators in Mines, 11th Underground Coal Operators'
Conference, University of Wollongong & the Australasian Institute of
Mining and Metallurgy 2011; pp. 171-177.
[18] Kollatsch C, Schumann M, Klimant P, Wittstock V, Putz M. Mobile
augmented reality based monitoring of assembly lines, Procedia CIRP,
2014; 23 (C), pp. 246 -251.
[19] Pintzos G, Rentzos L, Papakostas N, Chryssolouris G. A novel approach
for the combined use of AR goggles and m obile devices as communication
tools on the shopfloor, 8th International Conference on Digital Enterprise
Technology 2014; 25, pp. 132-137.
[20] Regenbrecht HT, Wa gner MT, Baratoff G. Ma gicMeeting: a
Collaborative Tangible Augmented Reality System, Virtual Reality 2002;
6(3): pp. 151166.
[21] Ong SK, Pang Y, Nee AYC, Augmented Reality Aided Assembly Design
and Planning, CIRP Annals - Manufacturing Technology 2007; pp 49-52.
[22] Sa de M, Churchill E. Mobile Augmented Reality: Explorin g Design and
Prototyping Techniques, Mobi leHCI '12 Proceedings of the 14th
international conference on Human-computer interaction with mobile
devices and service 2012; p 221-23.
[23] Boud AC, Haniff DJ, Baber C, Steiner SJ. Virtual reality and augmented
reality as a training tool for assembly tasks, IEEE International Conference
on Information Visualization , Proceedings 1999; pp. 32 -36.
[24] Schwald B, de Laval B, An augmented reality system for training and
assistance to maintenance in the industrial context, International
Conference in Central Europ e on Computer Graphics, Visualiza tion and
Computer Vision. Pr oceedings 2003; 3, pp. 425-432
[25] Rentzos L, Papanastasiou S, Papakostas N, Chryssolouris G. Augmented
reality for human-based assembly: Using product and process semantics,
12th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and
Evaluation of Human - Machine Systems, HMS 2013, 12(1), pp. 98-101.
[26] Zhong XW, Boulanger P, Georganas N. Collaborative Augmented
Reality: A Prototype for Industrial Training, 21th Biennial Symposium on
Communication, Canada; 2002.
[27] Lu Y, Xu X, Xu J. Development of a Hybrid Manufacturing Cloud, J
Manufacturing Systems 2014; 33(4), pp. 551-566.
[28] Mourtzis D, and Vlachou E. C loud-based cyber-physical systems and
quality of services, The TQM Journal, 2016:28, no. 5.
[29] Buckholtz B., Ragai I, Wang L. 2 016,“Remote Equipment Security in
Cloud Manufacturing Systems,” International Journal of Manufacturing
Research, Vol.11, No.2, pp.126-143
[30] Rong C, Nguyen ST, Jaatun MG. Beyond lightning: A survey on security
challenges in c loud computing. Comput Electric Engineer 2013; 39(1): pp.
47-54.
[31] Makris S, Pintzos G, Rentzos L, Chryssolouris G. Assembly support using
AR technology based on automatic sequence generation, CIRP Annals -
Manufacturing Technology 2013; 62(1), pp. 9-12.
[32] Vuzix STAR 1200XL product spec ifications
[33] Unity 3D, Unity Technologies, https://unity3d.com/
[34] Milgram P, Takemura H, Utsumi A, Kishino F. Augmented reality: A
class of displays on the reality-virtuality continuum, Telemanipulator and
Telepresence Technologies, SPIE 1994;2351, pp. 282-292.
[35] Siltanen S. Theory and applications of marker- based augmented reality,
Espoo, VTT. 199 p. + app. 43 p. VTT Science; 201 2
[36] Vuforia, PTC, https://www.vuforia.com
[37] Baines T, Lightfoot HW, Evans S, Neely A, Greenough R, Peppard J,
Wilson H. State-of-the-art in product-service systems, Proceedin gs of the
Institution of Mechanical E ngineers, Part B: journal of en gineering
manufacture 2007; 221(10), pp. 1543-1552.
[38] Tukker A. Product services for a resource-efficient and circular economy
a review, Journal of cleaner production 2015;97:pp. 76-91.
[39] Huang B, Li C, Yin C, Zhao X. Cloud manufacturing servi ce platform for
small-and medium-sized enterprises. The International Journal of
Advanced Manufacturin g Technology. 2013;65 (9-12):pp. 1261-72
... Due to the huge market potential for handheld mobile devices, handheld mobile AR has played a predominant role in present researches [6], [12]. It has found application in tourist industries [27], maintenance [28] and task support [29], education [30], and medical practices [31]- [33] respectively. An essential part of AR applications is the spatially consistent alignment of virtual and real objects [5] that required accurate and efficient 3D object manipulation. ...
... Additionally, the AR instructions offer supplementary guidance, empowering maintenance personnel to rectify their work before transitioning to subsequent tasks. Mourtzis et al. (2017) introduced an AR application that provides maintenance experts with a virtual view of the robot and its surrounding environment, enabling them to easily identify and diagnose issues. Furthermore, AR can be employed to furnish step-by-step instructions to the on-site technician. ...
... As MR has matured, its use has expanded across various sectors. In industry, for example, it is used to simulate complex processes, train workers in hazardous tasks, or perform remote maintenance with real-time visual guides [19]. In medicine, MR has been employed in surgery planning and medical student training, allowing them to practice complex procedures in highly realistic simulations without putting patient lives at risk [20]. ...
Article
Full-text available
In today’s education landscape, there is a lack of effective digital tools for hands-on teaching of PC and smartphone hardware assembly, especially in environments where physical resources are limited. This challenge is critical because practical learning in hardware is essential for developing technical skills in engineering students and related fields. Traditional alternatives, such as simulators or online tutorials, have not managed to provide an immersive and efficient learning experience. To address this need, Build_3D was developed—a mixed reality (MR) application designed for learning PC and smartphone hardware assembly, experienced through the Meta Quest 3 headset. The use of MR in an educational context is a topic of considerable debate, given the necessity for a comprehensive analysis of its potential advantages and disadvantages. This research proposes an evaluation of Build_3D’s usability using the Computer System Usability Questionnaire (IBM-CSUQ), focusing on three key dimensions: interface quality (INTERQUIAL), information quality (INFOQUIAL), and system usability (SYSUSE). The results indicate high user satisfaction, with interface quality rated the highest, followed by information quality and system usability. These findings suggest that this application provides an immersive and effective educational experience, minimizing cognitive load without compromising the quality of practical learning. The implications of this solution point to an improvement in hands-on hardware teaching through MR technologies, offering a viable and scalable alternative in educational environments with limited access to physical labs. Finally, this facilitates autonomous and motivating acquisition of technical skills for students.
... Traditional training approaches often fall short in preparing workers for these hybrid environments due to several critical limitations: a lack of realistic interaction scenarios, insufficient adaptability to individual learning curves, the absence of real-time feedback mechanisms, and limited scalability for complex industrial workflows [1][2][3][4]. ...
Article
Full-text available
This research introduces a virtual reality (VR) training system for improving human–robot collaboration (HRC) in industrial disassembly tasks, particularly for e-waste recycling. Conventional training approaches frequently fail to provide sufficient adaptability, immediate feedback, or scalable solutions for complex industrial workflows. The implementation leverages Quest Pro’s body-tracking capabilities to enable ergonomic, immersive interactions with planned eye-tracking integration for improved interactivity and accuracy. The Niryo One robot aids users in hands-on disassembly while generating synthetic data to refine robot motion planning models. A Robot Operating System (ROS) bridge enables the seamless simulation and control of various robotic platforms using Unified Robotics Description Format (URDF) files, bridging virtual and physical training environments. A Long Short-Term Memory (LSTM) model predicts user interactions and robotic motions, optimizing trajectory planning and minimizing errors. Monte Carlo dropout-based uncertainty estimation enhances prediction reliability, ensuring adaptability to dynamic user behavior. Initial technical validation demonstrates the platform’s potential, with preliminary testing showing promising results in task execution efficiency and human–robot motion alignment, though comprehensive user studies remain for future work. Limitations include the lack of multi-user scenarios, potential tracking inaccuracies, and the need for further real-world validation. This system establishes a sandbox training framework for HRC in disassembly, leveraging VR and AI-driven feedback to improve skill acquisition, task efficiency, and training scalability across industrial applications.
... The robotics sector also shows significant influence in this review. A clear trend is observed in this sector, with AR solutions exclusively aimed at operators, intending to enhance human-robot interaction [46,48,49,[52][53][54][55]. Notably, the operators using these applications are also required to have prior experience in their respective fields. ...
Article
Full-text available
This scoping review investigates the integration of augmented reality within Industry 4.0 and Industry 5.0, focusing on its applications across various industrial sectors. The study analyzes 74 relevant articles published since 2014, coinciding with the emergence of Industry 4.0. Since the introduction of the Industry 5.0 concept in 2021, there has been a noticeable increase in publications, reflecting a shift toward applications that prioritize personalization, sustainability, and human-centric innovations. The findings highlight the versatility of augmented reality in enhancing productivity, efficiency, safety, and work quality across industries such as robotics, automotive, education, and construction. However, the implementation of artificial intelligence within augmented reality applications remains limited, revealing a significant opportunity for future research to optimize their integration and functionality. The review emphasizes the importance of developing augmented reality solutions tailored to specific industrial needs, aligned with the principles of Industry 5.0, and leveraging versatile, cost-effective hardware like mobile devices and smartglasses to ensure accessibility and scalability.
Chapter
This chapter provides a comprehensive examination of Augmented Reality (AR) in manufacturing, covering its introduction, types, implementation strategies, advantages, challenges, applications, and methodologies for implementation. We present a brief overview of AR technology in the context of Industry 4.0 and its significance in manufacturing. It highlights the potential of AR to bridge the gap between the physical and digital worlds. We then discuss the various types of AR technologies relevant to manufacturing along with the working of AR technology. The numerous benefits of adopting AR in manufacturing and the systematic methodology for implementing AR in manufacturing environments is presented. Five applications of AR in manufacturing from real industries have been presented. The advantages, common challenges and barriers to AR implementation in manufacturing are discussed. This research provides a comprehensive roadmap for leveraging AR technology in manufacturing, offering insights into its potential, challenges, and practical implementation strategies.
Article
The rising interest in creating versatile robots to handle multiple tasks in various environments, with humans interacting through immersive interfaces. This survey provides a comprehensive review of extended reality (XR) applications in remote human-robot interaction (HRI). We developed a systematic search strategy based on the PRISMA methodology, focusing on peer-reviewed publications that demonstrate practical implementations of XR in remote robot control, real robot system deployment, and HRI applications, we analyzed research published between January 2013 and December 2023. From the initial 2,561 articles, 100 met our inclusion criteria were included. We categorized and summarized the domain in detail, delving into the methods used in these articles to achieve intuitive and effective remote HRI, highlighting user experience enhancement and interaction designs. This survey identifies research opportunities, particularly emphasizes that future researchers should explore the potential of XR, such as exploring multimodal enhancement techniques that seamlessly integrate visual, haptic, and auditory feedback for more intuitive teleoperation. Our analysis reveals that while XR shows promising potential in remote HRI, there are significant gaps, such as user-centered design. This survey provides a framework for understanding the current state of XR-based remote HRI, establishing a foundation for future research.
Chapter
Facing the needs of service-oriented manufacturing, it is a key task to flexibly allocate production resources to achieve the needs and well-being of different groups of end-users, including customers, operators, and managers. To this end, smart product-service system (smart PSS) has emerged as a novel systematic framework for engineering product development and service innovation in the context of human-cyber-physical system. Different from the traditional PSS, smart PSS focuses on providing the end-user’s requirements-oriented manufacturing services based on the mutual information interactions between physical space and cyberspace. This chapter discusses smart PSS’s features, enabling technologies, development process, and illustrate smart PSS’s process with an example of 3D printer product-service bundle configuration. Finally, some future perspectives of smart PSS are also discussed to welcome further in-depth research and practices.
Article
As remote maintenance becomes increasingly prevalent across various industries, the ability to remotely inspect equipment and guide on-site operations has become crucial. Remote maintenance often relies on 3D models that contain densely packed and interconnected parts, presenting a significant challenge in navigation and comprehension for experts. To address this challenge, we introduce HiNaviMec, a novel technique designed to enhance immersive understanding and navigation of complex 3D mechanical models. HiNaviMec employs a connection detection algorithm for 3D models to establish connections between mechanical parts and generate the mechanical structure graph. The graph is then processed using a weighted graph partitioning algorithm to divide the overall mechanical structure into more manageable sections. For each mechanical part, we calculate the relational distance and disassembly direction based on the mechanical structure graph. Furthermore, HiNaviMec generates a hierarchical structure to facilitate understanding of the mechanical structure. A case study and a user experiment of HiNaviMec show that it improves the user’s ability to understand internal mechanical structures and helps users better understand intricate component relationships.
Conference Paper
Full-text available
Maintenance and its cost continue, over the years, to draw the attention of production management since the unplanned failures decrease the reliability of the system and also the return of investments. Advanced maintenance techniques that capture and process shop-floor information can reduce costs and increase the sustainability of an enterprise. This paper presents a condition-based preventive maintenance approach integrated into a machine monitoring framework. The latter acquires data from shop-floor machine tools and analyses them through an information fusion technique to support the condition-based preventive maintenance operations. The proposed approach is developed into a software service, deployed on a Cloud environment. The service gathers and processes data, such as their actual processing time and machining time per tool, related to the operation of machine tools and equipment and calculates the expected remaining useful life of components. Moreover, it provides notifications to machine tool operators and maintenance departments, as well as it enables the communication among them using mobile technology. The framework is applied to a case study with data obtained from a machining SME.
Article
Full-text available
Purpose – The purpose of this paper is to review and explore the evolution, advances and future trends of cloud manufacturing, placing the focus on the quality of services. Moreover, moving toward the new trend of cyber-physical systems (CPS), a cloud-based cyber-physical system (CBCPS) is proposed combining the key enabling techniques of this decade, namely Internet of Things (IoT), cloud computing, Big Data analytics and CPS. Design/methodology/approach – First, an extensive review is made on cloud computing and its applications in manufacturing sectors, namely product development, manufacturing processes and manufacturing systems management. Second, a conceptual CBCPS which combines key enabling techniques including cloud computing, CPS and IoT is proposed. Finally, a review on the quality of the services (QoS) presented in the second step, along with the main security issues of cloud manufacturing, is conducted. Findings – The findings of this review indicate that the combination of the key enabling techniques presented in the CBCPS will lead to a new manufacturing paradigm capable of facing the new challenges and trends. The opportunities, as well as the challenges and barriers of the proposed framework are presented, concluding that the transition into this whole new era of networked computing and manufacturing has a valuable impact, but also generates several security and quality issues. Originality/value – The paper is the first to specifically study the QoS as a factor in the proposed manufacturing paradigm.
Article
Full-text available
Existing and evolving trends and paradigms in manufacturing, such as mass customisation and personalisation, call for better communication among product-production design and customisation and production execution. Specifically, the lack of feedback from and to the shopfloor can lead to lower product quality and increased production times. In state-of-the-art industrial practices, the most common visual interface devices for communication comprise control unit terminals, TFT monitors mounted over work stations, as well as the growing trend of mobile PCs and tablets. New technologies, such as Augmented Reality (AR), have also been considered in academic research for process simulation and operator guidance and training. This paper proposes a novel use of AR goggles, coupled with other mobile devices for the communication of people, working on the shopfloor and in the engineering offices. The proposed methodology tries to address the challenges, related to the use of both technologies (and their respective interfaces) by presenting an integrated approach: the use of a mobile device as an input device and as a fiducial marker for the positioning of a virtual screen in front of the user. After the presentation of the concept, its advantages and disadvantages, compared with current practices as well with the rest of the relevant academic work, are presented. The technical implementation to be realised, including specific software frameworks that will be used is also described. Special attention is given to the data models that will support the implementation of this approach as well as to show how they can be integrated into the existing systems and practices.
Conference Paper
Full-text available
Showing process-relevant information of assembly lines is an important task in production, e.g. for diagnostic purposes. Accessing the relevant manufacturing values during production requires extensive knowledge of the used machines and their control systems (e.g. PLC, CNC) and could cause a delay of the production process. Therefore, this paper presents an Augmented Reality (AR) based application for mobile devices realizing a user-friendly and problem-oriented visualization of relevant information directly on-site. The AR application uses a uniform graphical user interface. Due to this the user does not need to handle different human machine interfaces.
Article
Full-text available
Cloud manufacturing is emerging as a novel business paradigm for the manufacturing industry, in which dynamically scalable and virtualised resources are provided as consumable services over the Internet. A handful of cloud manufacturing systems are proposed for different business scenarios, most of which fall into one of three deployment modes, i.e. private cloud, community cloud, and public cloud. One of the challenges in the existing solutions is that few of them are capable of adapting to changes in the business environment. In fact, different companies may have different cloud requirements in different business situations; even a company at different business stages may need different cloud modes. Nevertheless, there is limited support on migrating to different cloud modes in existing solutions. This paper proposes a Hybrid Manufacturing Cloud that allows companies to deploy different cloud modes for their periodic business goals. Three typical cloud modes, i.e. private cloud, community cloud and public cloud are supported in the system. Furthermore, it enables companies to set self-defined access rules for each resource so that unauthorised companies will not have access to the resource. This self-managed mechanism gives companies full control of their businesses and boosts their trust with enhanced privacy protection. A unified ontology is developed to enhance semantic interoperability throughout the whole process of service provision in the clouds. A Cloud Management Engine is developed to manage all the user-defined clouds, in which Semantic Web technologies are used as the main toolkit. The feasibility of this approach is verified through a group of companies, each of which has complex access requirements for their resources. In addition, a use case is carried out between customers and service providers. This way, optimal service is delivered through the proposed system.
Article
With the growing development of cloud manufacturing (CM) applications in industry, one concern potential users have is security of the data, remote machines and operators. Potential security risks in communication with remotely operated manufacturing equipment have been of recent interest. The purpose of this paper is to provide an overview of security measures being considered to ensure the protection of data being sent to physical machines in a CM system. Topics covered include: internet of things, remote equipment control, security concerns in remote equipment control, existing proposed security measures for remote equipment control, and the future outlook of remote equipment control and security in CM systems.
Article
Since the 1990s, Product Service Systems (PSS) have been heralded as one of the most effective instruments for moving society towards a resource-efficient, circular economy and creating a much-needed ‘resource revolution’. This paper reviews the literature on PSS in the last decade and compares the findings with those from an earlier review in this journal in 2006. Close to 300 relevant papers were identified, over 140 of which have been referenced in this review. Research in the field of PSS has become more prolific, with the output of refereed papers quadrupling since 2000, while on average scientific output has only doubled. PSS has also become embedded in a wider range of science fields (such as manufacturing, ICT, business management, and design) and geographical regions (Asia now produces more papers than Europe). The literature of the last seven years has refined insights with regard to the design of PSS, as well as their business and environmental benefits, and confirmed the definitions and PSS concepts already available in 2006. A major contribution of the recent literature is research into how firms have implemented PSS in their organization and what the key success factors and issues that require special attention are (such as a focus on product availability for clients; an emphasis on diversity in terms of services provided rather than the range of products; and the need for staff to possess both knowledge of the product and relationship management skills). The reasons why PSS have nonetheless still not been widely implemented, particularly in the B2C context, seem to have already been explained fairly well in the literature available in 2006. For consumers, having control over things, artifacts, and life itself is one of the most valued attributes. PSS are often less accessible, or have less intangible value, than the competing product, in part because PSS usually do not allow consumers as much behavioral freedom or even leave them with the impression that the PSS provider could prescribe how they should behave.
Conference Paper
Augmented Reality (AR) can help close the gap between product development and manufacturing operation, mainly because of its ability to reuse and reproduce digital information and knowledge, while supporting assembly operators. This paper presents a semantic-based AR system used for the integration of the existing information and knowledge, available in CAD/PDM systems, for real-time support of the human operator. This study is supported by an algorithm used for the generation of the AR instructions based on the product and process semantics. The results are demonstrated in an automotive use-case.
Chapter
The transition from local economies to the global competitive landscape and the current fluctuating customer demands drives a never-ending transformation of manufacturing systems. In combination with prominent technological advances and other socio-political reasons of each era, such changes led to the evolution of manufacturing paradigms systems from craftsmanship to automation and decentralisation. This transition created the necessity for new means of designing, planning, and controlling manufacturing companies and networks. In production early stages, manufacturing was viewed as a functional area. On the other hand, current production paradigms are based on the customer focus and view manufacturing as an enterprise. This major diversification led to the gradual reorganisation of manufacturing systems in order to achieve mass customisation of typically mass-produced products. The purpose of this chapter is to specify the evolution of manufacturing paradigms, their basic principles, and the links among them. A discussion on future trends of manufacturing paradigms derives from the review and is finally presented. Chapter Preview Top Evolution Of Manufacturing Paradigms The manufacturing industry, since its birth two centuries ago, has evolved through a number of paradigms. The most prevailing manufacturing paradigms that characterised significant periods of time are chronologically: Craft Production, American Production, Mass Production, Lean Production, Mass Customisation and Global Manufacturing. Each paradigm had its time-span, was supported by a specific type of manufacturing system and operated based on specific business model (Table 1). Purchase this chapter to continue reading all 29 pages >