Information Exchange Process for AR based Smart Facility Maintenance System Using BIM Model

Full text

36th International Symposium on Automation and Robotics in Construction (ISARC 2019)
Information Exchange Process for AR based Smart Facility
Maintenance System Using BIM Model
S.W. Chunga, S.W. Kwonb, D.Y. Moona, K.H. Leea, and J.H. Shina
aDepartment of Convergence Engineering for Future City, SungKyunKwan University, Republic of Korea
bSchool of Civil, Architectural Engineering and Landscape Architecture, Sungkyunkwan University, Republic of
Korea (corresponding author)
E-mail: suwanx@nate.com, swkwon@skku.edu, yoshy17@nate.com, kiwijoa@skku.edu, kyup@skku.edu
Abstract –
In this study, we propose information exchange
process for the effective integration of buil ding
information modeling (BIM) into an augmented
reality (AR)-based smart facilities maintenance
(SFM) system. The proposed SFM system refers to a
system that combines technologies such as AR and
IoT sensors in the field maintenance work. This
requires the acquisition of data from various sources
followed by transformation of these data into an
appropriate format. Construction operation building
information exchange (COBie) is widely used as a
means to effectively integrate and utilize information
for maintenance. Therefore, SFM system has a
requirement attribute information system with
reference to COBie. But this information should be
linked to the maintenance work procedures in the
actual use case scenario and i t is necessary to define
the information exchange process. To solve this
problem, we uses the following methods to enable
SFM system development with applications for BIM
and AR technologies in the FM of the building sector
of public facilities. First, it analyzes the previous
studies on BIM-based maintenance works and AR
technology. Second, it divides the SFM work process
utilizing the BIM-based COBie system, and it defines
the COBie data required for each work phase. Third,
it develops a scenario-based business process
modeling notation (BPMN) for the SFM system
prototype. Finally, it proposes an implementation
method of SFM system architecture.
Keywords –
Building information model; Facility
maintenance; Augmented reality; Business process
modeling notation
1 Introduction
1.1 Background and Objectives of the
Research
One of main purposes of facility management (FM)
is to realize the expected asset value by maintaining or
complementing qualities of facilities planned in the
design phase and secured in the construction phase. FM
phase accounts for the longest proportion in the building
life cycle, and the cost consumed in this phase occupies
85% of the life cycle cost (LCC) [1]. Facility
information generated in the design and construction
phases is utilized as basic implementation data of the
facility management system (FMS) for effective and
systematic operation and maintenance, but it takes a
considerable amount of time and money to implement
basic data in the FMS through defining and verifying
the required information in the operation and
maintenance phases. To resolve difficulties in current
FMS implementation process, building information
modeling (BIM) technology has been introduced, which
can act as an alternative system to systematically and
efficiently operate and maintain the facilities. BIM is
structured with objects, attributes, and relationships.
Since all attributes and relationships are defined based
on objects, BIM can manage the required information
for operation and maintenance by function, space, and
use purpose.
Building o wners and facility maintenance mangers
have an opportunity to reduce a large amount of cost
during a much longer maintenance period than that of
design or construction phase by utilizing BIM-based
information. However, although utilization of BIM
information in the facility maintenance phase has many
advantages, BIM information has not yet been utilized
in on-site maintenance work. This is because a variety
of information required for facility maintenance has not
been defined, and, in many cases, this information is not
included in BIM-based design.
595

36th International Symposium on Automation and Robotics in Construction (ISARC 2019)
To solve this problem, a large number of studies
have been conducted to develop a system to transfer
BIM information generated over the design and
construction phases into the facility maintenance phase.
As a result, data exchange format called construction
operation building information exchange (COBie) was
developed by public institutions, which took project
ordering tasks of major facilities in the USA.
Meanwhile, AR technology has been utilized in
various areas, such as product review and production
system design. More recently, with the advancement of
smart devices, users can now employ AR more easily
than ever before. The construction industry is also
actively introducing on-site facility data visualization
and real-time collaboration features by utilizing AR
technology. However, few studies have been conducted
on the utilization of AR technology in the maintenance
phase as compared to the design and construction
phases, even though economic benefits would be the
largest in terms of LCC.
Thus, as one of the few studies on SFM for
automation management and efficient execution of
facility maintenance, this study aims to discuss the
current trend of BIM and AR technology; it measures of
how to link information and proposes an AR-based
SFM system. In addition, this study sets the COBie
system as a bench marking target—which has been
developed as a system prototype in the USA and is now
widely recognized in the world—to define the required
information exchange process for implementation.
1.2 Study Method and Procedure
This study uses the following methods to enable
SFM system development with applications for BIM
and AR technologies in the FM of the building sector of
public facilities. First, it analyzes previous studies on
BIM-based maintenance works and AR technology.
Second, it divides maintenance work process utilizing a
BIM-based COBie system, and it defines COBie data
required for each work phase. Third, it develops a
scenario-based business process modeling notation
(BPMN) for the SFM system prototype. Finally, it
proposes an implementation method of SFM system
architecture.
2 Analysis of Previous Studies
2.1 Building Maintenance Using BIM
When BIM is applied in the maintenance phase, it
can be utilized in analysis data for the purchase
procurement of machine equipment, control systems,
and other products; it can also check whether all
systems are correctly operating after buildings are
complete, which is the advantage of BIM to the
maintenance phase. It can also act as an interface for
sensors; it can remotely operate the management of
facilities and provide real-time monitoring of control
systems through providing accurate information about
space and systems [2]. BIM for FM provides a manual
for building users, which can help to locate models
visually, such as facilities, attachments, and furniture.
BIM also supports contingency plans, security
management, and scenario planning [3]. As described
above, several studies have been conducted actively to
show the advantages of a BIM application to FM.
A study by Yu developed a data model, Facility
Management Classes (FMC), which can extract required
information from Industry Foundation Classes (IFC) for
information compatibility of BIM data into FM [4]. A
study by Mendez R. proposed functions that can be
utilized by BIM in the maintenance phase; the study
developed a web-based prototype to improve the
utilization of BIM information [5]. East and Brodt
developed a data exchange format called COBie based
on a spreadsheet to solve inefficiencies due to
unnecessary information in maintenance among
information sets created during the design and
construction phases [6]. A study by Burcin defined a
level of BIM recognition in the maintenance phase
through expert interviews, applicable areas, required
data, and processes to implement BIM successfully [7].
Lee et al. attempted to improve a facility maintenance
system through benchmarking the COBie system, and
Choi et al. conducted a study to implement a FM system
for sewage treatment based on the COBie [8][9].
One study proposed software architecture to
integrate BIM with geographic information system
(GIS)-based FMS effectively [10]. Another study
proposed integrating BIM-based Mechanical, Electrical,
and Plumbing (MEP) information in the operation and
maintenance phases [11].
Most of the previous studies on FM utilizing BIM
focus on the interoperability with FM system and a
measure to link data produced during design and
construction phases as well as a direction to application
of BIM to FM systems and commercialization has been
chosen.
2.2 Utilization of AR Technologies in
Building Industry
The trend of utilizing AR technology in building
industry has showed that earlier studies focused on the
analysis of applicability of virtualization technology
Hammad presented the applicability of AR technology
to field works of infrastructure, and Koo considered the
development of a wireless technology-based, AR-
applied system in terms of display, tracking, and servers.
Afterward, a study on the presentation of an AR system
596

36th International Symposium on Automation and Robotics in Construction (ISARC 2019)
prototype for the inspection of steel column
construction was conducted [12] [13].
For studies on AR authoring tools, AR modeling
through capturing and rendering of exterior structures in
real time and 3D modeling can be found. A tool called
“Tinmith” was manufactured to interact between users
and AR directly [14]. Moon et al. implemented an AR
prototype that can select the position of a tower crane
appropriately in a high-rise building construction site
and conducted a verification procedure through actual
building drawings [15].
More recently, studies on the application of AR to
building maintenance have been conducted in various
ways. A study by Bae et al. proposed a novel method
that supported on-site construction and facility
management works by documenting on-site problems
and progress visually by on-site staff and by making
recent project information accessible as a form of AR
overlay [16]. Another study proposed a natural marker-
based AR framework that can support facility managers
digitally when they search FM items or perform
maintenance and repair works [17]. Kwon et al.
proposed a SFM system that improved existing
maintenance work; thy studied a novel maintenance
work process and an implementation method to apply
AR technology [18].
The analysis results of previous studies have showed
that the utilization of AR technology in the building
industry has enabled efficient construction management
by recognizing design drawings intuitively by users.
Four-dimensional (4D) BIM based on existing virtual
environments has lacked reflection about the actual
construction site conditions. Thus, if AR technology
that reduces a cognitive resistance is applied to
maintenance sites, its value and utility would be
significant in terms of productivity improvements.
3 SFM and COBie System
3.1 Overview of SFM
A SFM system proposed in this study aims to
support tasks of workers and managers by introducing
Internet of things (IoT)/VR/AR technologies to existing
on-site maintenance works. The base data to combine
the new technologies are BIM data. To do this, a study
on improved process of sequential and repetitive task
elements was conducted by applying AR/VR
technologies to existing maintenance tasks [18].
Data such as drawings, historical information, and
related documents stored in a distributed manner in the
facility maintenance tasks are linked and integrated with
BIM data to overcome the limitations of simple
repetitive tasks and inefficient data utilization.
Figure 1. Configuration of proposed system
As shown in Figure 1, variably distributed data can
be gathered, and each request can be processed as a hub
through the AR-based smart maintenance platform,
which manages the facility management database in the
center. This study's scope is limited to building interior
constructions (floor, wall, ceiling, window, door etc.)
among the maintenance target facilities.
3.2 SFM Work Process
A BIM attribute information and the historical
management information required by task units in the
maintenance work may be different from each other. To
define the information clearly and store them in a
database for the utilization of AR visualization, task
units were classified based on maintenance work
scenarios.
Figure 2. Scenario-based workflow
597

36th International Symposium on Automation and Robotics in Construction (ISARC 2019)
As shown in Figure 2, an on-site inspector checks
the inspection-required facility location and recognizes
the AR marker attached to the inspection target space in
the daily inspection task scenario. Inspection-required
members and checklists in the corresponding space are
visualized through the markers by which the inspector
can perform visual inspection and photo shooting. In
this process, video information through the inspector's
camera is shared in real time so that various decision-
making participants can judge together whether there
are defects or not. If decision making determines that
there is a defect through the inspection, historical
information, such as installation, warranty, and
management companies, is visualized, and the status
information can be sent to the corresponding companies
directly from the site. After this, a maintenance manager
decides whether repair work is needed. If repair work is
done, task-related guidance and progress are monitored
in the construction process, and stored in the FMS.
The core technologies of AR visualization in this
process are the following: 1) AR-based surveillance task
support technology, 2) 3D information plus AR
overlaying technology, and 3) 3D data conversion
technology for AR/VR.
Figure 3. AR visualization process in inspection
work
3.3 Definition of the COBie System
To implement the AR-based smart maintenance
process proposed in this study, COBie is benchmarked
and utilized. COBie is an information exchange data
format based on a spread sheet jointly developed by
public institutions that take project orders of major
facilities in the USA led by the US army corps of
engineers. It was developed as a means to replace a
large amount of documents related to facility
management at the time of completing the construction
of a project. COBie includes definitions and formats of
information required during the maintenance phase
among a BIM information created in a series of phases
during construction projects : planning, design,
construction, operation, and maintenance.
Table 1. COBie data sheets and contents by phase
As presented in Table 1, COBie consists of 18 data
sheets, and the related detailed contents can be
expanded through expandable templates. In the data
sheets that can be utilized in the whole phases of
construction projects, contract-related items of
construction participants, drawings, coordination
systems, and building completion-related issues are
included. In the initial design phase, facility information
Phase Sheet Contents
All
Contact People and Companies
Document All applicable document
references
Attribute Properties of referenced
item
Coordinate Spatial locations in box,
line, or point format
Issue Other issues remaining
at handover.
Early
Design
Facility Information on Facilities
and Standards
Floor Vertical levels and
exterior areas
Space Spaces
Zone Sets of spaces sharing a
specific attribute
Type Types of equipment,
products, and materials
Detailed
Design
Component Individually named or
schedule items
System Sets of components
providing a service
Assembly Constituents for Types,
Components and others
Connection Logical connections
between components
Impact
Economic,
Environmental and
Social Impacts at various
stages in the life cycle
O&M
Spare Onsite and replacement
parts
Resource Required materials,
tools, and training
Job PM, Safety, and other
job plans
598

36th International Symposium on Automation and Robotics in Construction (ISARC 2019)
and floors, spaces, and material types are included, and
in the detailed design phase, the connection of all
members and assembly, system configurations,
processes, and costs are added. In operation and
maintenance (O&M) phase, replacement and repair
items and required resources and tasks are included. In
addition, COBie is configured to expand the required
attribute information constantly.
Figure 4. Smart maintenance tasks and COBie
data mapping
All of these data sheets are not needed in the
COBie system definition, which were required for the
previously proposed maintenance work by the scenarios,
but it is important to utilize only minimum information
considering the link between attribute information and
data lightweight, which are essential for AR
visualization. Thus, the information required for the
previously proposed maintenance work by the scenarios
and COBie data sheets was mapped. This process was
conducted through prior literature reviews and through
interviews and surveys with on-site maintenance
workers, as shown in Figure 4.
Through this process, document, issue, type, and job
data sheets were found to include attribute information
required for the on-site maintenance work. However,
excessive attribute information was required in the
document and job sheet among these four data sheets,
and unnecessary information was also included
depending on maintenance work participants. Thus,
warranty and plan data sheets were additionally created
and used for optimization of AR-based visualization
information.
Table 2. COBie data sheet for field maintenance
Sheet
Contents
Document
Submittals and approvals, DWG.
Issue Other issues remaining at
maintenance
Type Types of equipment, products,
materials, model
, Warranty data
Job PM, Safety, and other job plans
Contents included in the newly added warranty data
were the product names of the interior finish materials,
repair (warranty) responsible companies, contact details,
and warranty-related document information. Through
this information, which companies are responsible can
be identified when defects occur, and requests for repair
can be achieved rapidly. A plan data contains
information that is utilized when maintenance
construction occurs, which includes construction
progress status, follow-up tasks, and next maintenance
inspection date.
4 Information Exchange Process to
Implement the Prototype
4.1 Smart Maintenance BPMN
To implement a prototype of an AR-based, on-site
support module for the SFM on the basis of
maintenance scenarios proposed in this study, a BPMN,
including the current status of data input and output, is
required. BPMN is a visualized notation method to
represent business processes. It is developed to notate a
message flow between activities over a time flow in the
process. BPMN that defines and represents the data
inputs and outputs required in the maintenance work
process, based on daily inspection scenarios, is shown in
Figure 5.
Most of the 3D models and attribute information of
the inspection-required objects are extracted from a
BIM information and stored in a database as COBie
Excel sheets. Here, the additionally required attribute
information is A R market information, inspection date,
and warranty period, which requires a manual input
process by the maintenance manager or supplier.
The most important thing in the utilization of BIM in
maintenance work is to clarify the definitions of
requirements and use cases. Thus, the proposed BPMN
facilitates detailed implementation of smart
maintenance use case.
599

36th International Symposium on Automation and Robotics in Construction (ISARC 2019)
4.2 Functional Screen Configuration and UI
Basically, an AR system consists of a camera, an
image converter, tracking, and a video background
render. A camera component helps efficient delivery to
the camera frame. A camera frame is provided
automatically according to the device, depending on
image format and size. Once markers are recognized
through the HMD camera, users define the targets that
will be observed through the AR via the pixel format
conversion. Targets defined in the database configured
with COBie system monitors objects through the
tracking module. After detected object's status is
identified, the 3D virtual objects are modeled, and
information is provided on the display. Auxiliary
information about augmented 3D virtual object in
display is managed by the web-based FMS and
augmented through XML. In web service application,
auxiliary information about augmented objects was
composed of a management system and two-way
communication systems. The auxiliary information
management system was configured to provide various
pieces of information rapidly to users and overcome the
existing problem, which only showed 3D object
augmentation. Two-way communication system was
configured to provide information in real time—which
was not provided by the auxiliary information
management system—through two-way communication
between users and managers who use the AR system.
Figure 6. Example of AR-based SFM system UI
UI in the AR application, which emphasizes
interaction functions through real world, requires
different designs from those of existing mobile
applications. Most mobile devices utilized in site are
smart phones and tablets, which support a touch-based
interface. However, since on-site maintenance support
application proposed in this study targeted HMD
devices, existing touch-based operations cannot be used.
Thus, controls through gesture or voice recognition are
needed, which require more user-friendly and intuitive
UI. Consequently, menu button items were removed,
and required attribute information was laid out on the
lower side of the screen. Since augmented reality
interacts with the real world, visualized information
supports a transparency control function to project the
real world.
Figure 5. Daily / regular maintenance scenario based BPMN
600

36th International Symposium on Automation and Robotics in Construction (ISARC 2019)
This feature excludes obstacles in the view when
workers move or perform works, and it helps to verify
the worker's location by positioning a navigation map
inside the building on the right upper side of the screen.
Figure 6 shows the example of the screen configuration
of the prototype implementation, in which site inspector
visualizes specification information about door.
Information about supplier expert who supplied door,
information about temperature and humidity of room,
and information about interior finish inspection can be
found in the lower side of the screen. An inspector
observes target facility, such as door, window, wall, and
column, and can record inspection results about
inspection items through checklist.
4.3 Definition of SFM System Information
Exchange System
Entire information exchange process of the SFM
System described above is shown in Figure 7.
Preliminary data use BIM data at handover. BIM
information at this stage includes more detailed
information of the space and the interior finishing
material, and the space information and the property
information of the space are used in conjunction with
COBie. A mong the information required for AR
visualization, 3D model shape information does not
work direct ly from BIM, so it is processed through
information processing and optimization, and additional
information necessary for maintenance is also included
at this time. Basic information required before
performing all these operations is stored in the Type and
Document sheets of the COBie data.
After that, procedure of performing the work by the
site inspector is moved to the site, the marker
recognition, the space and absence information are
visually confirmed, and the result of inspection is
automatically transmitted. In order to recognize the AR
marker information and visualize necessary information,
modules needed are tracking / rendering / measurement
module and implemented through separate API. In
addition to the AR visualization function, there is an
additional need for indoor navigation, data transmission,
voice recognition, video streaming, automatic
measurement and photo shooting functions, which are
linked to each stage of operation.
When inspection is completed, the result information
is stored in COBie's issue sheet, and it is used when
maintenance work is in progress. As the maintenance
work progresses, newly generated contract information
is automatically saved in the Job Datasheet again, and
COBie repeats this process and updates it.
5 Conclusions
This study proposed that data exchange procces
through data item configuration and functional
definition to apply AR technology to on-site
maintenance works in a BIM-based SFM system.
As a result, this study determined that when
maintenance and management information are
configured based on the COBie system, interoperability
in various software environment can be ensured. In
addition, if AR visualization technology of maintenance
information through overlay with real site circumstances
Figure 7. Information exchange system for smart maintenance
601

36th International Symposium on Automation and Robotics in Construction (ISARC 2019)
is utilized, it can enable rapid information sharing about
on-site working conditions and decision making.
Furthermore, the utilization of BIM datasets generated
in design and construction phases is meaningful in terms
of utilization BIM data throughout the whole life cycle.
However, since the study scope was limited to filed
inspection works, the study results cannot be applied to
all maintenance works in general. This study also needs
some feedback through user satisfaction surveys
regarding the proposed system development and UI
configuration. For a future study, a technology that can
visualize the information of complex facilities which
cannot be identified visually should be developed
through AR technology by expanding the maintenance
target to mechanical, electrical, and plumbing in
addition to the interior finish.
Acknowledgment
This research was supported by a grant (19AUDP-
B127891-03) from the Architecture & Urban
Development Research Program funded by the Ministry
of Land, Infrastructure and Transport of the Korean
government.
References
[1] Teicholz, E. Overview and current state of FM
technology. FACILITY MANAGEMENT—AN,
2004.
[2] Sacks, R., Eastman, C., Lee, G., & Teicholz, P.
BIM Handbook: A Guide to Building Information
Modeling for Owners, Designers, Engineers,
Contractors, and Facility Managers. John Wiley &
Sons, 2018.
[3] Jieun, K., Hyunsang, C., Taewook, K. Derivation
of System Requirements and Implementation of
System Framework for BIM-based Urban Facility
Maintenance System. Journal of the Korean
Content Association, 14(4):397-406, 2014.
[4] Yu, K., Froese, T., & Grobler, F. A development
framework for data models for co mputer-
integrated facilities management. Automation in
construction, 9(2):145-167, 2000.
[5] Mendez, R. O. The building information model in
facilities management. PhD Thesis. Worcester
Polytechnic Institute, 2006.
[6] East, W. E., & Brodt, W. BIM for construction
handover. Journal of Building Information
Modeling, 28-35, 2007.
[7] Becerik-Gerber, B., Jazizadeh, F., Li, N., & Calis,
G. Application areas and data requirements for
BIM-enabled facilities management. Journal of
construction engineering and management,
138(3):431-442, 2011.
[8] Lee, S. G., & Yu, J. H. Prerequisites to utilize BIM
(Building Information Modeling) for Facility
Management. Journal of the Korea Facility
Management Association, 8(1):27-39, 2013.
[9] Choi, J. H., & Um, D. Y. A study on the feasibility
of COBie to the wastewater treatment p lant.
Journal of the Korean Society of Civil Eng ineers,
34(1):273-283, 2014.
[10] Kang, T. W., & Hong, C. H. A study on software
architecture for effective BIM/GIS-based facility
management data integration. Automation in
Construction, 54:25-38, 2015.
[11] Hu, Z. Z., Tian, P. L., Li, S. W., & Zhang, J. P.
BIM-based integrated delivery technologies for
intelligent MEP management in the operation and
maintenance phase. Advances in Engineering
Software, 115:1-16, 2018.
[12] Hammad, A., Garrett, Jr, J. H., & Karimi, H. A.
Potential of mobile augmented reality for
infrastructure field tasks. In Applications of
Advanced Technologies in Transportation pp. 425-
432, 2002.
[13] Koo, B., Choi, H., & Shon, T. Wiva: Wsn
monitoring framework based on 3D visualization
and augmented reality in mobile devices. In
International Conference on Smart Homes and
Health Telematics pp. 158-165, Springer, Berlin,
Heidelberg, 2009.
[14] Piekarski, W., & Thomas, B. H. Tinmith-mobile
outdoor augmented reality modelling
demonstration. In Proceedings of the 2nd
IEEE/ACM International Symposium on Mixed
and Augmented Reality pp. 317, IEEE Computer
Society, 2003.
[15] Moon, S. Y., Yun, S. Y., Kim, H. S., & Kang, L. S.
Improved Method for Increasing Maintenance
Efficiency of Construction Structure Using
Augmented Reality by Marker-Less Method.
Journal of The Korean Society of Civil Engineers,
35(4):961-968, 2015.
[16] Bae, H., Golparvar-Fard, M., & White, J. Rapid
image-based localization using clustered 3d point
cloud models with geo-location data for aec/fm
mobile augmented reality applications. In
Computing in Civil and Building Engineering, pp.
841-849, 2014.
[17] Koch, C., Neges, M., König, M., & Abramovici,
M. Natural markers for augmented reality-based
indoor navigation and facility maintenance.
Automation in Construction, 48:18-30, 2014.
[18] Chung, S. W., Kwon, S. W., Moon, D.Y., & Ko, T.
K. Smart Facility Management Systems Utilizing
Open BIM and Augmented/Virtual Reality. In 35th
International Symposium on Automation and
Robotics in Construction, 2018
602