ArticlePDF Available

Augmented Reality Application to Support Remote Maintenance as a Service in the Robotics Industry

Authors:
  • University of Patras Lab. For Manufacturing Systems and Automation (LMS)
  • Laboratory for Manufacturing Systems and Automation

Abstract and Figures

Maintenance of manufactured products is among the most common services in industry and its cost often exceed 30% of the operating costs. Modern manufacturing companies are shifting their focus from products to combined ecosystem of Products- Service Systems (PSS). Towards that end, the main objective of this work is to develop a cloud-based service-oriented system that implements AR technology for remote maintenance by enabling cooperation between the on- spot technician and the manufacturer. The proposed system includes the methods for the record of the malfunction by the end user, the actions required by the expert so as to provide instructions in an Augmented Reality application for maintenance, as well as the cloud- based platform that will allow their communication and the exchange of information. In addition to the above, the proposed system consists of smart assembly/disassembly algorithms for automated generation of assembly sequences and AR scenes and improved interface, aiming to maximize existing knowledge usage while creating vivid AR service instructions. The proposed system is validated in a real-life case study following the requirements of a robotics SME.
Content may be subject to copyright.
Available online at www.sciencedirect.com
2212-8271 © 2017 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
Peer-review under responsibility of the scientifi c committee of The 50th CIRP Conference on Manufacturing Systems
doi: 10.1016/j.procir.2017.03.154
Procedia CIRP 63 ( 2017 ) 46 51
ScienceDirect
The 50th CIRP Conference on Manufacturing Systems
Augmented reality application to support remote maintenance as a service
in the Robotics industry
D. Mourtzisa*, V. Zogopoulosa, E. Vlachoua
aLaboratory for Manufactu ring Systems and Automation, Depa rtment of Mechanical Enginee ring and Aeronautics, University of Patras, 26500, Rio Patras,
Greece
* Corresponding author, e-mail: mourtzis@lms.mech.upatras.gr, Tel: +30 2610 910160, Fax: +30 2610 997744
Abstract
Maintenance of manufactured products is among the most common services in industry and its cost often exceed 30% of the operating costs.
Modern manufacturing companies are shifting their focus from products to combined ecosystem of Products- Service Systems (PSS). Towards
that end, the main objective of this work is to develop a cloud-based service-oriented system that implements AR technology for remote
maintenance by enabling cooperation between the on- spot technician and the manufacturer. The proposed system includes the methods for the
record of the malfunction by the end user, the actions required by the expert so as to provide instructions in an Augmented Reality application
for maintenance, as well as the cloud- based platform that will allow their communication and the exchange of information. In addition to the
above, the proposed system consists of smart assembly/disassembly algorithms for automated generation of assembly sequences and AR scenes
and improved interface, aiming to maximize existing knowledge usage while creating vivid AR service instructions. The proposed system is
validated in a real-life case study following the requirements of a robotics SME.
© 2017 The Authors. Published by Elsevier B.V.
Peer-review under responsibility of the scientific committee of The 50th CIRP Conference on Manufacturing Systems.
Keywords: Augmented Reality; Remote Maintenan ce; Product Service System
1. Introduction
In the era of mass production and in the emerging era of mass
personalization [1][2] with increased competition between
manufacturers, it has become really important to come up with
new ways to improve customer satisfaction. Many
manufacturers attempt to achieve that by offering high- quality
Product Service Systems (PSS) throughout the product’s life
cycle [3]. PSS is a value proposition strategy that offers
products-services and is designed to be: competitive, satisfy
customer needs, and have a lower environmental impact than
traditional business models. Maintenance service is one of the
most commonly used, especially in products that require
maintenance frequently. Towards supporting this interface
between IT systems and avoiding isolated work, Cloud
Manufacturing has been regarded as an enabler and has already
formed the basis for new business models [4]. Furthermore, this
approach takes into account the usage of CAx systems in
manufacturing and the need to develop new instructions
providing systems that are more intriguing and more accurate
than the traditional ones. Augmented Reality (AR) is a rapidly
evolving technology that is used more and more in different
manufacturing fields in the last few years [5], [6]. Using the
CAD three-dimensional geometries and assembly information,
enriched with intelligence, the manufacturer can create a series
of AR scenes to support service sequence.
Targeting the Cloud manufacturing and remote
maintenance, the main objective of this work is to develop an
internet-based, service-oriented system that implements AR
technology for enabling tele- maintenance by cooperation of the
end user and the manufacturer.
2. State of the art
Maintenance is a core activity of the production lifecycle
since it accounts for as much as 60 to 70% of its total costs [7].
This has led to increased need for maintenance planning
through product’s life cycle and the implementation of more
© 2017 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
Peer-review under responsibility of the scientifi c committee of The 50th CIRP Conference on Manufacturing Systems
47
D. Mourtzis et al. / Procedia CIRP 63 ( 2017 ) 46 – 51
and more new technologies (cloud manufacturing [4],
Dynamically Adaptive Systems for self- maintenance [8],
machine monitoring [7]). Despite the effort to limit machine
downtime [9], most of service impact on productivity accounts
for unexpected breakdown, as it cannot be predicted and
quantified in terms of time and required effort. And despite
some effort on the field, it requires a time-consuming process
which has a negative impact on machine availability.
Augmented Reality (AR) is another enabling technology used
for dealing with the increasingly complex maintenance
procedures [10]. Either by using head-mounted displays [11],
[12] or portable devices [13] a number of solutions have
emerged, testing various ways of AR system- user interaction
(voice commands, gestures, devices- hosted menus). The
potential of the newly introduced technology in supporting
maintenance tasks is renowned even by large manufacturing
companies (BMW [14], Bosch [15]). More recently, the
concept of enabling communication between an expert and the
on- spot technician (tele- maintenance) has arisen [16], [17]
delivering some promising results in synchronous and
asynchronous information exchange.
Apart from maintenance, AR has found fertile ground in
other fields of manufacturing. Firstly, AR can be used as a
mean to vividly project the current status of a warehouse or a
production line, allowing constantly monitoring its current
status, communication and planning [18], [19]. Secondly,
Augmented and Virtual Reality have been proved valuable
tools in prototyping and collaborative design [20]. They allow
the fast and costless creation of visual prototypes that can be
manipulated by more than one user and also, overlaid on top of
existing products so as to facilitate customization and reusable
engineering [21], [22]. Moreover, Virtual and Augmented
Reality have been widely used as means of training technicians
in performing assembly tasks [23]. Those technologies provide
an intriguing experience for the technician [24] and thus, they
are more efficient than the traditional methods [25], [26].
Meaningful information generated by IT tools should be
seamlessly shared across the enterprise in order to support
different business functions. Cloud manufacturing enables this
ubiquitous information provision and enables the creation of
intelligent factory networks reshaping the manufacturing
business models. Cloud computing systems and cloud
manufacturing may play a critical role in the realisation of
“Design Anywhere Manufacture Anywhere” philosophy [4].
Another recent study [27] presented the key benefits to
manufacturing as a result of adopting the Cloud technology
such as scalability to business size and needs, ubiquitous
network access [28], and visualisation. However, security and
data protection are still challenges that need to be addressed
[29]. Issues, such as the resource location multi-tenancy and
authentication also need to be tackled in a combined way [30].
The literature review makes apparent that Augmented
Reality systems are welcomed in manufacturing. A field that
already has implemented cloud features in many of its
applications. The contribution of this development compared
to existing approaches is the creation of an asynchronous AR
remote maintenance support system that implements cloud-
based communication between the end user and the
manufacturer that facilitates the reuse of existing knowledge.
In addition to that, the developed internet-based and service-
oriented system is supported by an assembly/disassembly
algorithm that enables the automated generation of the AR
scenes and increases the level of automation. The implemented
platform is designed to be provided for Product-Service
support throughout a product’s life cycle so as to reduce the
impact of Mean Time to Repair in machine availability,
especially in unexpected breakdowns, where external expert
contribution in malfunction detection and service sequence
may be needed. The developed platform utilizes a cloud
database that enables ubiquitous data access and permits the
technician to upload the malfunction report and receive the
corresponding service sequence in AR scenes by the
manufacturer easily. Moreover, the cloud platform facilitates
the supervising mechanic in AR scene creation by keeping
record of older service sequence that can be fully or partially
reused.
3. Architecture design of tele- maintenance system
This paper proposes an innovative Product- Service System
that enables tele- maintenance support through Augmented
Reality scenes. The platform includes the deployment of a
cloud system that will facilitate the communication between
the on- spot technician and the expert by enabling feedback
reports and maintenance instructions exchange. Fig. 1 presents
the architecture of the proposed system and the data exchange
between the technician and the manufacturer representative
mechanic. In order to achieve the proper functionality of the
proposed platform, some key features are established. Each
time the remote maintenance framework is called, a three-step
procedure is required: (i) malfunction report composition, (ii)
diagnosis and AR maintenance instruction generation and, (iii)
maintenance and evaluation.
The first step in the proposed system is registering the
malfunction report. Whenever a service is required, regardless
Fig. 1. Architecture of the developed framework
Fig
1
Architecture of th e developed framework
48 D. Mourtzis et al. / Procedia CIRP 63 ( 2017 ) 46 – 51
if concerning scheduled maintenance or unexpected
malfunction, a service report is created prior to any other action.
In traditional methods, this is a written report, created by the
on- spot technician which describes the problem in text, reports
any parts that need to be replaced and, in some cases, defines
all the actions needed to address it. In the proposed approach,
the technician takes advantage of the communications
technologies to create a digitized, data- rich report.
The malfunction report application provides the technician
with a set of features that facilitate accurate malfunction
capturing, which is essential for remote failure diagnosis. The
application is designed to be used through either the proposed
AR hardware setup or solely using a mobile device. When on-
spot, the operator may record the malfunction by writing
explanatory text, which may include sensor data or description
on the technician’s actions to address the issue, by taking
photographs of the malfunctioning or broken down parts, and
by recording sound, which may include voicemails or machine
noise, using the malfunction report application. The data are
then stored locally on the device and uploaded to the cloud
platform. The maintenance support provider is notified that a
new malfunction report is uploaded.
After viewing the malfunction report, the maintenance
support provider begins the main part of the service;
malfunction cause identification and AR instructions
generation. Remote maintenance may be used to cover both
those needs. On the one hand, the AR equipment allows the on-
spot technician to create an enhanced failure report which,
using the cloud-based feedback mechanisms, can be sent to the
manufacturer in short time. In some cases, the expert has to
guide the on-spot technician in order to determine the cause of
the malfunction; a procedure that may require to exchange
feedback on how to gather more information on the problem.
To achieve that, the supervising expert may provide some AR
instructions to the technician on how to gather additional
information and ask for new report. Uncommon failures require
increased amount of time to detect with traditional practice and
may, even, require the expert to perform on- spot inspection.
Thus, remote diagnosis drastically decreases down time.
On the other hand, Augmented Reality is used for visualizing
the repair sequence. The maintenance expert has to create an
AR instruction scene sequence, which is adjusted to the
problem. To achieve that, he takes advantage of two system
features: knowledge reusability and automatization. First of all,
the expert can consult and reuse similar scenes that are already
created for similar tasks, such as lubrication or screw
tightening, which can be stored in the cloud database.
Moreover, available in the cloud database, the maintenance
expert may find the CAD of the product’s parts and a set of
utilities. These utilities include pre-created part movement,
scripts, as well as the GUI that allows the technician to interact
with the system. Additionally, an algorithm that automates
scene creation in assembly procedures [31] has been
implemented. The algorithm receives as input a CAD file of a
mechanism and, through consequently moving each part, finds
the sequence of assembly/disassembly process needed to be
followed. Moreover, the algorithm defines the axis and
direction of assembly and disassembly of each part and assigns
the fasteners (e.g. bolts, screws) to their corresponding parts.
Therefore, the maintenance expert may quickly create AR
instructions, divided in perceivable steps, for the assembly and
disassembly tasks, that account for a large part of the
maintenance procedure. The implementation of these
components in the proposed system aims to improve the
efficiency of providing remote maintenance support by
reducing the required time, while also reducing the effort and
AR expertise required from the maintenance expert for
designing AR scenes and user interfaces.
Aiming to further reduce the response time of the
maintenance support system, each AR maintenance sequence
that is created to address a maintenance task will be stored in
the cloud database. The same malfunction may come up for
more than one customers. Especially scheduled maintenance
tasks, tend to occur more than once in a product’s life cycle. By
introducing proper organizing of the existing AR maintenance
instructions in the cloud database (filing per targeted machine
and naming according to the task), the maintenance support
provider will be available to instantly recall existing AR
sequences. As a result, AR maintenance sequence reuse is
expected to increase efficiency in maintenance tasks and reduce
machine downtime.
After completing the AR maintenance instructions creation,
the maintenance expert sends them to the on-spot technician.
The communication is again realized through the cloud
platform. The technician is notified that the instructions are
available, downloads them and performs the maintenance task.
After completing the task, the technician verifies that the
machine is now fully operational.
4. Implementation of the proposed framework
4.1 Hardware Implementation
Aiming to use a hardware setup that allows the user to move
freely and in the same time enjoy a highly usable interface, the
combined use of AR goggles and mobile device is preferred
[19]. In order to carry out the proposed system, three devices
have been used: a set of optical see-through Augmented Reality
goggles, a laptop PC and a mobile device. Aiming to provide
the end- user with a high quality visual result that also permits
him to maintain visual contact with the potentially dangerous
environment and not occupy his hands, a set of AR goggles
have been used for this system; Vuzix™ Star 1200XL [32].
Secondly, the system described in this paper requires the use of
a laptop PC (host- PC). This computer is responsible for
executing the AR application, communicating with the mobile
device and with the cloud platform. The last part, the mobile
device, hosts an interface that allows the operator to interact
with the AR application through menus. On top of that, it
allows the operator to add information concerning the task. As
stated before, the mobile device can also be used an
independent tool for feedback recording by the operator, which
provides increased mobility. For the needs of this paper,
Nexus 4 Android smartphone was used.
The proposed system requires the devise to work together so
as to provide the desired result. To achieve that, the HMD and
the mobile device need to be wired to the host- PC. The
operator wears the HMD on his head, adjusts the AR goggles
49
D. Mourtzis et al. / Procedia CIRP 63 ( 2017 ) 46 – 51
and straps the mobile device to his arm. This configuration
is designed so as not to limit his mobility. The proposed
hardware setup can be seen in Fig. 2 below.
4.2 Software Implementation
For the development of this system two software packages
were used. In order to achieve high quality geometry rendering,
a commercial cross-platform game creation system was used:
Unity 3D™ [33]. It includes a game engine and integrated
development environment (IDE) which allows scripting in
three programming languages (C# was used for the developed
platform). The criteria that led to this choice are:
x It supports a wide variety of object formats permitting
extracting the 3D geometries from the CAD files.
x It is able to access dynamic file formats, such as .dll and
.xml, and connect to web URLs
x It supports development not only on Windows but also on
Android™ and iOS™ applications, so as to create
applications that can also be implemented directly to mobile
devices
Tracking is a crucial part in Augmented Reality as it
positions the virtual environment in the real surroundings of the
user [34]. To achieve tracking, it uses the HMD’s camera to
recognize predefined frame markers in user’s field of view. The
transformation T between a camera and a marker is:
ݔൌܶܺ
Where: X is a point in world coordinates, xc is its projection in
ideal image coordinates and T is the pose matrix.
Transformation T consists of translation vector t and a rotation
matrix R:
ݔܴȁݐሿൈܺݔݕݖ቉ൌݎݎݎݐ
ݎݎݎݐ
ݎݎݎݐ൩൦ܺ
ܻ
ܼ
ͳ
In addition to that, in order to map between frame marker ideal
image (xc) and observed pixel coordinates (xpix) a camera
calibration matrix K is used so:
ݔ௣௜௫ܭൈݔฺ൭ݔ௣௜௫
ݕ௣௜௫
ͳ൱ൌ݂Ͳ݌
Ͳ
Ͳ݂݌
Ͳ
ͲͲͳͲ
ݔ
ݕ
ݖ
As a result, when the camera detects the frame marker:
ݔܭൈܶൈܺ
Where xi are the positions of the four corners of the marker
in the camera image and Xi are their corresponding world
coordinates (Fig. 3).
The system then calculates M = K × T which is the final
transformation matrix and applies it to all visualized
geometries, so as to estimate their projection on the virtual
environment [35]. With the aim of creating a system whose
tracking is accurate and robust, PTC Vuforia SDK, a
commercial extension for Unity™, was selected [36].
5. Product Service System (PSS) model
The concept of the PSS is a special case of servitization [37].
PSS is a newly introduced strategy of combining the product
and the set of services that come with it, in one bundle. This
way the manufacturers shift from selling just a product to
selling a functionality or a service, which is provided by the
product. The manufacturer is responsible for all the actions that
will secure that the provided service remains adequate
throughout the products’ lifecycle [38].
The proposed platform aims to offer a value- adding,
product-oriented service [38] that will benefit both the
customer, who enjoys robust and high- quality maintenance
support, and the OEM/ PSS provider, who gains a critical
advantage compared to competitor companies as he is more
involved in an extended part of his products’ lifecycle [39].
Thus, the PSS provider, exploiting cloud communication and
data exchange, may easily interact with the customers,
targeting multiple markets with little cost, controlling a large
portion of after sales product-related services, increasing the
profits, gathering crucial product design feedback (design for
maintainability) and creating a stronger connection with the
customers. The increased level of automation and the high
efficiency of maintenance service of the developed platform
allows its implementation in existing business models. Ease of
implementation and low implementation cost was one of the
main concerns of the developed framework. This is highly
important especially for SMEs, as they can increase their
influence in the market by providing their customers with
unified service solutions of high efficiency together with their
products. The implementation of such high-end services allows
Fig. 2. On - spot operators’ hardware setup
Fig. 3. Architecture of the AR scene
50 D. Mourtzis et al. / Procedia CIRP 63 ( 2017 ) 46 – 51
SMEs to increase customers’ satisfaction, withstand
competition pressure and expand their sales networks.
6. Robotics & Automation industrial case
The developed framework was implemented on a case study
provided by a robotics SME (Fig 4). The case concerns a real-
life maintenance sequence, on which the manufacturer is
usually called to deploy specialized personnel that must go on
the spot and maintain the machine. More specifically, the
maintenance case refers to a battery pack replacement of an
industrial robot installed in a Near East country, in a city
1100km away from the use case provider facilities.
This task is crucial for securing robot’s continuous function.
It commonly requires technicians from the robotics SME to go
where the robot cell is installed and perform the tasks. The
selected task requires simple actions by the technician, thus can
be easily explained remotely. At the same time, it includes all
the features that need to be tested. At first, since the
maintenance tasks consisted solely of assembly/ disassembly
tasks, the smart algorithm automatically created most of the AR
scenes. In Table 1, the Excel file generated by the assembly/
disassembly algorithm for this task is presented.
Part Name
+X
-X
+Y
-Y
+Z
-Z
Tier
Num
Type
Robot.1
0
Base
BatteryBase.1
0
1
0
0
0
0
1
Part
Battery.1
0
0
1
0
0
0
2
Part
Battery.2
0
0
1
0
0
0
2
Part
Battery.3
0
0
1
0
0
0
2
Part
Battery.4
0
0
1
0
0
0
2
Part
Table 1. Assembly steps table based on the smart assembly/disassembly
precedence algorit hm
In the first six columns, the axis and direction of part
disassembly is indicated. Moreover, the sequence of
disassembly is presented in the “Tier Number” column.
Exploiting this information, the developed system may quickly
and automatically create the assembly/ disassembly AR scenes,
drastically reducing the time required for creating maintenance
instructions.
Moreover, the interface is used not only to change scenes but
also to input information that cannot be known in AR
instructions creation, such as which battery connector is free.
This is a crucial part of this maintenance task as the new
batteries need to be connected before unplugging the old ones.
Based on that input, the system alters the projected AR
instructions in the next steps. The steps of the procedure were
provided by corresponding industrial partner.
With the current approach, the robotics SME deploys a
maintenance expert to go on-spot to perform the maintenance
task. This procedure costs 1370€ and requires 9 hours from the
time the malfunction is reported until the machine is back
online. Applying the proposed approach, the cost is reduced to
150€ and the required time is decreased to 2 hours (Fig. 5).
Especially in the case that the task has re-occurred in the past
and can be recalled, omitting the time required for creating AR
instructions, the required time is further reduced. Thus, the
positive impact of applying the proposed framework
application is apparent.
Fig. 5. Required cost and time- Comparison of the two methods
7. Conclusions and future work
This paper presents the development and testing of an
Augmented Reality remote maintenance platform that can be
used for providing PSS maintenance service. A cloud platform
was also implemented as a communication enabler and to assist
existing knowledge reuse. This study also takes into account
algorithms which increase the efficiency of the procedure and
Fig. 4. The sustain able Product Service System
51
D. Mourtzis et al. / Procedia CIRP 63 ( 2017 ) 46 – 51
the automation level by reducing the actions and expertise
required by the engineer to create the service sequence
instructions which can be effortlessly perceived by less
experienced technicians. In addition to that, through the
proposed remote approach the required maintenance time and
cost is highly reduced. Finally, manufacturing companies can
provide remote maintenance as a service, increasing their
customer’s satisfaction and delivering added value solutions.
Future work could focus on integrating the existing system
and other systems in an enterprise, increasing interoperability.
In addition to that, it is essential to reduce AR scenes designer’s
cognitive load by increasing the automatization of the
procedure.
Acknowledgements
The work presented in this paper is partially supported by
the European Union’s Horizon 2020 research and innovation
project “Customer-driven design of product-services and
production networks to adapt to regional market requirements-
ProRegio” (GA No: 636966).
References
[1] Chryssolouris G. Manufacturing Systems: Theory and Practice, 2nd Ed.
New York: Springer-Verlag; 2006.
[2] Mourtzis D, Doukas M, The evolution of manufacturing systems: From
craftsmanship to th e era of customisation , Design and Management of
Lean Production Systems, IGI Global, Chapter 1, 2014
[3] Takata S, Kimura F, Van Houten FJAM, Westkämper E, Shpitalni M,
Ceglarek D, Jay Lee. Maintenance: Changing role in life cycle
management, CIRP Annals Manufacturi ng Technology; 2004.
[4] Xu X. From cloud computing to cloud manufacturing , Robot and
Computer-Integrated Manufacturing 2014; 28(1): pp. 75-86
[5] Ong SK, Yuan ML, Nee AYC. Augmented Reality Applications in
Manufacturing: A Survey, Int J. Production Research 2008;46: pp. 2707-
2742.
[6] Nee AYC, Ong SK, Chryssolouris G, Mourtzis D. Augmented Reality
applications in design and manufacturing, CIRP Annals- Manufacturing
Technology 2012;61, pp. 657 679.
[7] Mourtzis D, Vlachou E, Milas N, Xanthopoulos N. A cloud-based approach
for maintenance of machine tools and equipment based on shop-floor
monitoring, Procedia CIRP; pp. 655-660
[8] Zhang J, Goldsby HJ, Cheng BHC . Modular verification of dyna mically
adaptive systems, Proceedings of the 8 th ACM International Conference
on Aspect-Oriented Software Development 2009; pp. 161-172.
[9] Albers A, Schmidt G. Scheduling with Unexpect ed Machine Breakdowns.
Computer Science, Proceedi ngs of the Third Internati onal Workshop on
Approximation Algorithms for Combinatorial Optimization Problems:
Randomization, Approximation, and Combinatorial Algorithms and
Techniques, Springer-Verlag 1999; 1671, pp. 269-280.
[10] Azuma RT. A survey of augmented reality. Presence: Teleoperators and
virtual environments. 1997;6 (4): pp.355-85.
[11] Feiner S, MacIntyre B, Seligmann D. Karm a (knowle dge-based
augmented reality for maintenance assistance); 1993
[12] Schwald B, De Laval B. An augmented reality system for training and
assistance to maintenance in the industrial context; 2003.
[13] Henderson, S., Feiner, S., Augmented Reality for Maintenance and Repair
(ARMAR), Final project report; 2010.
[14] Platonov J, Heibel H, Meier P, Groll mann B. A mobile markerless AR
system for maintenance and repair, Proceedings of the 5th IEEE and ACM
International Symposium on Mixed and Augmented Reality, Washington,
DC, USA 2006; pp 105108.
[15] Common Augmented Reality Platform CAP, Bosch, http://oe.bosch-
automotive.com/en/bosch-automotive-aftermarket
[16] Bottecchia S, Cieutat JM., Jessel JP. T.A.C: Augmented Reality System
for Collaborative Tele-Assistan ce in the Field of Maintenance thr ough
Internet, France 2010; pp.1-7
[17] Alem L, Tecchia F, Huang W. Remote Tele-assistance System for
Maintenance Operators in Mines, 11th Underground Coal Operators'
Conference, University of Wollongong & the Australasian Institute of
Mining and Metallurgy 2011; pp. 171-177.
[18] Kollatsch C, Schumann M, Klimant P, Wittstock V, Putz M. Mobile
augmented reality based monitoring of assembly lines, Procedia CIRP,
2014; 23 (C), pp. 246 -251.
[19] Pintzos G, Rentzos L, Papakostas N, Chryssolouris G. A novel approach
for the combined use of AR goggles and m obile devices as communication
tools on the shopfloor, 8th International Conference on Digital Enterprise
Technology 2014; 25, pp. 132-137.
[20] Regenbrecht HT, Wa gner MT, Baratoff G. Ma gicMeeting: a
Collaborative Tangible Augmented Reality System, Virtual Reality 2002;
6(3): pp. 151166.
[21] Ong SK, Pang Y, Nee AYC, Augmented Reality Aided Assembly Design
and Planning, CIRP Annals - Manufacturing Technology 2007; pp 49-52.
[22] Sa de M, Churchill E. Mobile Augmented Reality: Explorin g Design and
Prototyping Techniques, Mobi leHCI '12 Proceedings of the 14th
international conference on Human-computer interaction with mobile
devices and service 2012; p 221-23.
[23] Boud AC, Haniff DJ, Baber C, Steiner SJ. Virtual reality and augmented
reality as a training tool for assembly tasks, IEEE International Conference
on Information Visualization , Proceedings 1999; pp. 32 -36.
[24] Schwald B, de Laval B, An augmented reality system for training and
assistance to maintenance in the industrial context, International
Conference in Central Europ e on Computer Graphics, Visualiza tion and
Computer Vision. Pr oceedings 2003; 3, pp. 425-432
[25] Rentzos L, Papanastasiou S, Papakostas N, Chryssolouris G. Augmented
reality for human-based assembly: Using product and process semantics,
12th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and
Evaluation of Human - Machine Systems, HMS 2013, 12(1), pp. 98-101.
[26] Zhong XW, Boulanger P, Georganas N. Collaborative Augmented
Reality: A Prototype for Industrial Training, 21th Biennial Symposium on
Communication, Canada; 2002.
[27] Lu Y, Xu X, Xu J. Development of a Hybrid Manufacturing Cloud, J
Manufacturing Systems 2014; 33(4), pp. 551-566.
[28] Mourtzis D, and Vlachou E. C loud-based cyber-physical systems and
quality of services, The TQM Journal, 2016:28, no. 5.
[29] Buckholtz B., Ragai I, Wang L. 2 016,“Remote Equipment Security in
Cloud Manufacturing Systems,” International Journal of Manufacturing
Research, Vol.11, No.2, pp.126-143
[30] Rong C, Nguyen ST, Jaatun MG. Beyond lightning: A survey on security
challenges in c loud computing. Comput Electric Engineer 2013; 39(1): pp.
47-54.
[31] Makris S, Pintzos G, Rentzos L, Chryssolouris G. Assembly support using
AR technology based on automatic sequence generation, CIRP Annals -
Manufacturing Technology 2013; 62(1), pp. 9-12.
[32] Vuzix STAR 1200XL product spec ifications
[33] Unity 3D, Unity Technologies, https://unity3d.com/
[34] Milgram P, Takemura H, Utsumi A, Kishino F. Augmented reality: A
class of displays on the reality-virtuality continuum, Telemanipulator and
Telepresence Technologies, SPIE 1994;2351, pp. 282-292.
[35] Siltanen S. Theory and applications of marker- based augmented reality,
Espoo, VTT. 199 p. + app. 43 p. VTT Science; 201 2
[36] Vuforia, PTC, https://www.vuforia.com
[37] Baines T, Lightfoot HW, Evans S, Neely A, Greenough R, Peppard J,
Wilson H. State-of-the-art in product-service systems, Proceedin gs of the
Institution of Mechanical E ngineers, Part B: journal of en gineering
manufacture 2007; 221(10), pp. 1543-1552.
[38] Tukker A. Product services for a resource-efficient and circular economy
a review, Journal of cleaner production 2015;97:pp. 76-91.
[39] Huang B, Li C, Yin C, Zhao X. Cloud manufacturing servi ce platform for
small-and medium-sized enterprises. The International Journal of
Advanced Manufacturin g Technology. 2013;65 (9-12):pp. 1261-72
... [12] developed an AR-based assistance system for assembly tasks; Mourtzis et.al. [13] proposed IT platform able to connect the manufacturer with the field operator and provide AR procedures for remote maintenance; finally, Konstantinidis et.al. [14] presented an AR assistant for low-skilled operators to guide them in maintenance procedures via smartphones. ...
Chapter
In recent decades industrial development has led to increasingly sophisticated machinery and systems, which require complex maintenance routines. Consequently, maintenance operators may not have the sufficient skills to perform recovery procedures properly and quickly, so that the need of assistance from the manufacturer’s after-sales service or companies specialized in maintenance services. Such actions usually lead to very long recovery times, high maintenance costs, and a temporary drop in production. In this scenario, we should consider that Industry 4.0 is making available innovative technologies, such as Augmented Reality (AR), suitable for improving the skills and competencies of operators without burdening their cognitive load, and consequently wellbeing. However, technologies must be selected, designed, and used according to the users’ needs to be effective and useful. The paper presents a user experience (UX)-driven methodology for designing user-centric AR applications for complex maintenance procedures. The methodology was applied to a real industrial case concerning the management of CNC machines in a plant producing tractors components, where a smartphone-based AR application was designed and tested with users. The satisfactory results highlighted the potential benefits of AR in industry and specifically in maintenance.
... [12] developed an AR-based assistance system for assembly tasks; Mourtzis et.al. [13] proposed IT platform able to connect the manufacturer with the field operator and provide AR procedures for remote maintenance; finally, Konstantinidis et.al. [14] presented an AR assistant for low-skilled operators to guide them in maintenance procedures via smartphones. ...
Conference Paper
In recent decades industrial development has led to increasingly sophisticated machinery and systems, which require complex maintenance routines. Consequently, maintenance operators may not have the sufficient skills to perform recovery procedures properly and quickly, so that the need of assistance from the manufacturer's after-sales service or companies specialized in maintenance services. Such actions usually lead to very long recovery times, high maintenance costs, and a temporary drop in production. In this scenario, we should consider that Industry 4.0 is making available innovative technologies, such as Augmented Reality (AR), suitable for improving the skills and competencies of operators without burdening their cognitive load, and consequently wellbeing. However, technologies must be selected, designed, and used according to the users' needs to be effective and useful. The paper presents a user experience (UX)-driven methodology for designing user-centric AR applications for complex maintenance procedures. The methodology was applied to a real industrial case concerning the management of CNC machines in a plant producing tractors components, where a smartphone-based AR application was designed and tested with users. The satisfactory results highlighted the potential benefits of AR in industry and specifically in maintenance.
... The benefit of using AR in the industrial context can reduce error rates by up to 82% [3,4], reduce costs by 20% [5], reduce assembly time by 50%, and reduce content creation and maintenance by 90% [6]. In addition, it can improve speed [4], optimize the system, reduce costs, and improve efficiency with hands-free operation [1]. ...
Chapter
EXtended Reality (XR) has numerous applications in industrial contexts, including to assist in activities such as preventive maintenance and/or reactive interventions in machinery, wherein time-effectiveness is required to mitigate the costs of inactive production lines or the need to subcontract specialized technicians to perform those operations. However, developing an Augmented Reality (AR) application capable of being user-friendly and truly serviceable for aiding in the referred context requires keeping in mind human factors as key aspects and, as such, interaction must be a concern to address, namely, considering user feedback. Therefore, starting from a previous AR system oriented to industrial interventions that was developed and assessed with users and considering the feedback of the participant, the duly alterations were done and a new test was conducted, with a total of 27 participants, divided into 2 groups: 11 users who participated in the first experience and 16 general users who had never contacted with the solution before. A qualitative improvement in the user experience was noticed from the questionnaires feedback. Points such as fluidity, responsiveness, ease, speed, and intuitiveness were highlighted. Voice commands matched the tap commands in terms of user preferences.
... On the other hand technologies such as Virtual Reality (VR) that immerses users in a computer-generated world, and Augmented Reality (AR), overlaying digital information onto the physical world are spreading in the manufacturing sector [10], [11]. Augmented ...
Conference Paper
Full-text available
The purpose of this paper is to establish the relative maturity of industry 4.0 implementation by the composites manufacturing industry in the UK. The overall goal of this research is to establish Industry 4.0 implementation methodology for composites manufacturing SMEs, and the maturity measure is an important indicator of the current or “as is” status of the composites companies concerning Industry 4.0 adoption.
... Cloud-based assembly platform provides flexible, high-performance, and universal capabilities. It works with big data to support communication, share data, and answer engineer questions through information provided by the manufacturer and reliable sources [84,85]. ...
Chapter
Full-text available
The concept of Augmented Reality (AR) has existed in the field of aerospace for several decades in the form of Head-Up Display (HUD) or Head-Worn Display (HWD). These displays enhance Human-Machine Interfaces and Interactions (HMI2) and allow pilots to visualize the minimum required flight information while seeing the physical environment through a semi-transparent visor. Numerous research studies are still being conducted to improve pilot safety during challenging situations, especially during low visibility conditions and landing scenarios. Besides flight navigation, aerospace engineers are exploring many modern cloud-based AR systems to be used as remote and/or AI-powered assist tools for field operators, such as maintenance technicians, manufacturing operators, and Air Traffic Control Officers (ATCO). Thanks to the rapid advancement in computer vision and deep neural network architectures, modern AR technologies can also scan or reconstruct the 3D environment with high precision in real time. This feature typically utilizes the depth cameras onboard or independent from the AR devices, helping engineers rapidly identify problems during an inspection and implement the appropriate solutions. Some studies also suggest 3D printing of reconstructed models for additive manufacturing. This chapter covers several aspects and potentials of AR technology in the aerospace sector, including those already adopted by the companies and those currently under research.
... Rhetorically, companies utilising advanced PdM solutions face difficulties in managing assets, and report high amount of failures. This prooved to be a business opportunity for companies and maintenance experts in engaging and providing outsourced maintenance services with in-door solutions, which is why many engage with MaaS (Maintenance as a Service) concepts [31]. Also, the results show that 13.1% of companies outsource their maintenance activities, while 50.4% of rely on external experts or companies to perform failure analysis of their equipment. ...
Preprint
Full-text available
As a consequence of applying advanced maintenance practices, the theoretical probability of failures is relatively low. However, observation of low market intelligence and maintenance management has been reported. The experimental investigation is supported by findings from a survey targeting asset-intensive companies applying hydraulic power systems. Next, the study incorporates multidimensional data analysis using CA-AHC (Correspondance Analysis with Agglomerative Hierarchical Clustering) approach. The non-parametric machine learning models are used from generated feature subspace to extract features affecting maintenance performance indicators. The results support empirical evidence that equipment age increases the probability of failures. However, the novel findings show that number of maintenance personnel, equipment size measured by nominal working energy consumption, and activities dedicated to maintaining fluid cleanliness impact regression results of companies utilising hydraulic applications.
... AR systems enhance the physical world by overlaying virtual information on it without compromising the user's ability to perceive reality (2). In the manufacturing industry, AR has proven to be a powerful tool for generating virtual assembly instructions (3,4), quality inspection (5), maintenance (6), training (7), remote assistance (8), safety (9), etc. AR applications on the shop floor provide operators with real-time information, schematics, and guidance on maintenance procedures while performing their daily tasks. This increases operator accuracy and efficiency and leads to reduced downtime. ...
Chapter
Full-text available
Augmented Reality (AR) applications are beginning to see rapid adaptation in the manufacturing sector. AR systems in the manufacturing industry can assist operators in simple manufacturing tasks, such as virtual instructions, as well as complex remote assistance and maintenance tasks. There are many ways to interact with AR content, commonly known as interaction modalities. These modalities are classified into three categories: touch, touchless, and wearable, or sometimes a combination of more than one modality known as multimodal interfaces. However, in the manufacturing industry, there is no one-size-fits-all solution regarding the interaction with AR content. Current research explores various input interaction modalities in AR and assesses the advantages & disadvantages associated with them in the manufacturing context. We adopt both touch and touchless modalities in our case study of assembling a planetary gear and present the merits and demerits of each modality assessing the extent of natural user interaction with AR content. Interaction with a wearable AR system is not part of this investigation.
Chapter
Additive manufacturing (AM) is a revolutionary approach to industrial production that enables manufacturing lighter, more significant components and systems by directing hardware to deposit material in precise geometric shapes, layer upon layer, using data from computer-aided design (CAD) software or 3D object scanners. Design for additive manufacturing (DfAM) refers to creating a product design that takes advantage of the unique capabilities of AM. This study investigates the role of Augmented Reality (AR) as a tool for enhancing the learning experience of DfAM using Head-mounted Displays (HMDs). An AR application for Microsoft HoloLens 2 is created to allow the exploration of selected DfAM models and their industrial use cases. We analyse how the design thinking technique is implemented in creating this study, from the stage of understanding and defining the problem to the final solution. Further, we discuss the process of creating the innovative interaction design through the cross-platform game engine Unity and examine the structure of the application AR DfAM Explorer as well as the relevance of DfAM models chosen for the visualization. A questionnaire was prepared, and a real-time hands-on validation of the developed prototype was carried out with experts from the field of AM as well as students learning AM. Moreover, a pilot study assessed the effect of learning using HMD compared to the paper variant on students’ knowledge change in DfAM principles and related AM topics. The results obtained from the validation and pilot study were scrutinized and hence the advantages and limitations of the HMD visualization were discussed. Finally, an outlook for future developments and further extensions of this idea are provided.
Thesis
Full-text available
Technical drawing training has an important place among basic engineering trainings. Especially due to the lack of technical drawing reading skills of the new graduates, the number of discarded products in the machinery and manufacturing sector increases while the productivity in production decreases. In order to eliminate this important problem and meet the expectations of the sector in this field, the ability to read Technical Drawing should be gained during vocational and engineering education. In this study; supportive virtual (VR) and augmented reality (AR) applications have been developed in order to meet the educational material needs of trainers and to improve the learning performance of the students and to meet the expectations of the sector. The AR is also a way to help individuals learn lifelong in terms of installation, use and sustainability, and to provide a more understandable transfer of knowledge and skills. In this study, AR applications have been developed in all subject titles, including surface roughness and geometric tolerance information, in order to support the correct application of the details on the drawings in technical drawing reading and manufacturing, which are seriously lacking in the machinery manufacturing sector. While developing its applications, content and material development studies were conducted in line with needs analysis and expectations studies. While developing AR applications, the parts to be transferred to Unity3D environment have been converted to obj or fbx file format with optimal mesh structure. Google ARCore library to develop AR application in Untiy3D environment and related library for QR code scanning have also been added to Unity3D environment. The animations were designed in accordance with the scenarios and the interface was designed to manage these animations.
Conference Paper
Full-text available
Maintenance and its cost continue, over the years, to draw the attention of production management since the unplanned failures decrease the reliability of the system and also the return of investments. Advanced maintenance techniques that capture and process shop-floor information can reduce costs and increase the sustainability of an enterprise. This paper presents a condition-based preventive maintenance approach integrated into a machine monitoring framework. The latter acquires data from shop-floor machine tools and analyses them through an information fusion technique to support the condition-based preventive maintenance operations. The proposed approach is developed into a software service, deployed on a Cloud environment. The service gathers and processes data, such as their actual processing time and machining time per tool, related to the operation of machine tools and equipment and calculates the expected remaining useful life of components. Moreover, it provides notifications to machine tool operators and maintenance departments, as well as it enables the communication among them using mobile technology. The framework is applied to a case study with data obtained from a machining SME.
Article
Full-text available
Purpose – The purpose of this paper is to review and explore the evolution, advances and future trends of cloud manufacturing, placing the focus on the quality of services. Moreover, moving toward the new trend of cyber-physical systems (CPS), a cloud-based cyber-physical system (CBCPS) is proposed combining the key enabling techniques of this decade, namely Internet of Things (IoT), cloud computing, Big Data analytics and CPS. Design/methodology/approach – First, an extensive review is made on cloud computing and its applications in manufacturing sectors, namely product development, manufacturing processes and manufacturing systems management. Second, a conceptual CBCPS which combines key enabling techniques including cloud computing, CPS and IoT is proposed. Finally, a review on the quality of the services (QoS) presented in the second step, along with the main security issues of cloud manufacturing, is conducted. Findings – The findings of this review indicate that the combination of the key enabling techniques presented in the CBCPS will lead to a new manufacturing paradigm capable of facing the new challenges and trends. The opportunities, as well as the challenges and barriers of the proposed framework are presented, concluding that the transition into this whole new era of networked computing and manufacturing has a valuable impact, but also generates several security and quality issues. Originality/value – The paper is the first to specifically study the QoS as a factor in the proposed manufacturing paradigm.
Article
Full-text available
Existing and evolving trends and paradigms in manufacturing, such as mass customisation and personalisation, call for better communication among product-production design and customisation and production execution. Specifically, the lack of feedback from and to the shopfloor can lead to lower product quality and increased production times. In state-of-the-art industrial practices, the most common visual interface devices for communication comprise control unit terminals, TFT monitors mounted over work stations, as well as the growing trend of mobile PCs and tablets. New technologies, such as Augmented Reality (AR), have also been considered in academic research for process simulation and operator guidance and training. This paper proposes a novel use of AR goggles, coupled with other mobile devices for the communication of people, working on the shopfloor and in the engineering offices. The proposed methodology tries to address the challenges, related to the use of both technologies (and their respective interfaces) by presenting an integrated approach: the use of a mobile device as an input device and as a fiducial marker for the positioning of a virtual screen in front of the user. After the presentation of the concept, its advantages and disadvantages, compared with current practices as well with the rest of the relevant academic work, are presented. The technical implementation to be realised, including specific software frameworks that will be used is also described. Special attention is given to the data models that will support the implementation of this approach as well as to show how they can be integrated into the existing systems and practices.
Conference Paper
Full-text available
Showing process-relevant information of assembly lines is an important task in production, e.g. for diagnostic purposes. Accessing the relevant manufacturing values during production requires extensive knowledge of the used machines and their control systems (e.g. PLC, CNC) and could cause a delay of the production process. Therefore, this paper presents an Augmented Reality (AR) based application for mobile devices realizing a user-friendly and problem-oriented visualization of relevant information directly on-site. The AR application uses a uniform graphical user interface. Due to this the user does not need to handle different human machine interfaces.
Article
Full-text available
Cloud manufacturing is emerging as a novel business paradigm for the manufacturing industry, in which dynamically scalable and virtualised resources are provided as consumable services over the Internet. A handful of cloud manufacturing systems are proposed for different business scenarios, most of which fall into one of three deployment modes, i.e. private cloud, community cloud, and public cloud. One of the challenges in the existing solutions is that few of them are capable of adapting to changes in the business environment. In fact, different companies may have different cloud requirements in different business situations; even a company at different business stages may need different cloud modes. Nevertheless, there is limited support on migrating to different cloud modes in existing solutions. This paper proposes a Hybrid Manufacturing Cloud that allows companies to deploy different cloud modes for their periodic business goals. Three typical cloud modes, i.e. private cloud, community cloud and public cloud are supported in the system. Furthermore, it enables companies to set self-defined access rules for each resource so that unauthorised companies will not have access to the resource. This self-managed mechanism gives companies full control of their businesses and boosts their trust with enhanced privacy protection. A unified ontology is developed to enhance semantic interoperability throughout the whole process of service provision in the clouds. A Cloud Management Engine is developed to manage all the user-defined clouds, in which Semantic Web technologies are used as the main toolkit. The feasibility of this approach is verified through a group of companies, each of which has complex access requirements for their resources. In addition, a use case is carried out between customers and service providers. This way, optimal service is delivered through the proposed system.
Article
With the growing development of cloud manufacturing (CM) applications in industry, one concern potential users have is security of the data, remote machines and operators. Potential security risks in communication with remotely operated manufacturing equipment have been of recent interest. The purpose of this paper is to provide an overview of security measures being considered to ensure the protection of data being sent to physical machines in a CM system. Topics covered include: internet of things, remote equipment control, security concerns in remote equipment control, existing proposed security measures for remote equipment control, and the future outlook of remote equipment control and security in CM systems.
Article
Since the 1990s, Product Service Systems (PSS) have been heralded as one of the most effective instruments for moving society towards a resource-efficient, circular economy and creating a much-needed ‘resource revolution’. This paper reviews the literature on PSS in the last decade and compares the findings with those from an earlier review in this journal in 2006. Close to 300 relevant papers were identified, over 140 of which have been referenced in this review. Research in the field of PSS has become more prolific, with the output of refereed papers quadrupling since 2000, while on average scientific output has only doubled. PSS has also become embedded in a wider range of science fields (such as manufacturing, ICT, business management, and design) and geographical regions (Asia now produces more papers than Europe). The literature of the last seven years has refined insights with regard to the design of PSS, as well as their business and environmental benefits, and confirmed the definitions and PSS concepts already available in 2006. A major contribution of the recent literature is research into how firms have implemented PSS in their organization and what the key success factors and issues that require special attention are (such as a focus on product availability for clients; an emphasis on diversity in terms of services provided rather than the range of products; and the need for staff to possess both knowledge of the product and relationship management skills). The reasons why PSS have nonetheless still not been widely implemented, particularly in the B2C context, seem to have already been explained fairly well in the literature available in 2006. For consumers, having control over things, artifacts, and life itself is one of the most valued attributes. PSS are often less accessible, or have less intangible value, than the competing product, in part because PSS usually do not allow consumers as much behavioral freedom or even leave them with the impression that the PSS provider could prescribe how they should behave.
Conference Paper
Augmented Reality (AR) can help close the gap between product development and manufacturing operation, mainly because of its ability to reuse and reproduce digital information and knowledge, while supporting assembly operators. This paper presents a semantic-based AR system used for the integration of the existing information and knowledge, available in CAD/PDM systems, for real-time support of the human operator. This study is supported by an algorithm used for the generation of the AR instructions based on the product and process semantics. The results are demonstrated in an automotive use-case.
Chapter
The transition from local economies to the global competitive landscape and the current fluctuating customer demands drives a never-ending transformation of manufacturing systems. In combination with prominent technological advances and other socio-political reasons of each era, such changes led to the evolution of manufacturing paradigms systems from craftsmanship to automation and decentralisation. This transition created the necessity for new means of designing, planning, and controlling manufacturing companies and networks. In production early stages, manufacturing was viewed as a functional area. On the other hand, current production paradigms are based on the customer focus and view manufacturing as an enterprise. This major diversification led to the gradual reorganisation of manufacturing systems in order to achieve mass customisation of typically mass-produced products. The purpose of this chapter is to specify the evolution of manufacturing paradigms, their basic principles, and the links among them. A discussion on future trends of manufacturing paradigms derives from the review and is finally presented. Chapter Preview Top Evolution Of Manufacturing Paradigms The manufacturing industry, since its birth two centuries ago, has evolved through a number of paradigms. The most prevailing manufacturing paradigms that characterised significant periods of time are chronologically: Craft Production, American Production, Mass Production, Lean Production, Mass Customisation and Global Manufacturing. Each paradigm had its time-span, was supported by a specific type of manufacturing system and operated based on specific business model (Table 1). Purchase this chapter to continue reading all 29 pages >