Content uploaded by Junjie Chen
Author content
All content in this area was uploaded by Junjie Chen on Dec 07, 2022
Content may be subject to copyright.
1
Automated facility inspection using robotics and BIM: A knowledge-driven
1
approach
2
3
Junjie Chen a, Weisheng Lu a, , Yonglin Fu a, Zhiming Dong a
4
a Department of Real Estate and Construction, The University of Hong Kong, Pokfulam Road,
5
Hong Kong, China
6
7
This is the pre-print version of the paper:
Chen, J., Lu, W., Fu, Y., & Dong, Z. (2023). Automated Facility Inspection using robotics
and BIM. Advanced Engineering Informatics, 55, 101838. Doi:
10.1016/j.aei.2022.101838.
The final version of this paper is available at: https://doi.org/10.1016/j.aei.2022.101838.
The use of this file must follow the Creative Commons Attribution Non-Commercial No
Derivatives License, as required by Elsevier's policy.
8
Abstract
9
Facility inspection is crucial for ensuring the performance of built assets. A traditional inspection,
10
characterized by humans’ physical presence, is laborious, time-consuming, and becomes difficult
11
to implement because of travel restrictions amid the pandemic. This laborious practice can be
12
automated by emerging smart technologies such as robotics and building information model (BIM).
13
However, such automated facility inspection (AFI) entails an autonomy of the robots to adaptively
14
response to the complexity of their environments, which, unfortunately, has rarely been
15
documented. The goal of this research is to propose a knowledge-driven approach that can
16
potentially lead to large-scale automation of facility inspection. It equips facility inspection robots
17
with an ability of unambiguous reasoning for independent decision-making. At the core the
18
approach is an integrated scene-task-agent (iSTA) model that formalizes engineering priori in
19
facility management and integrates the rich contextual knowledge from BIM. Experiments
20
demonstrated the applicability of the approach, which can endow robots with autonomy and
21
knowledge to navigate the challenging built environments and deliver facility inspection outcomes.
22
The iSTA model is publicized online, in hope of further extension by the research community and
23
practical deployment to enable AFI.
24
25
Keywords: Facility management; Inspection; Robotic; Building information modeling (BIM);
26
Knowledge formalization; Ontology.
27
28
* Corresponding author.
E-mail address: wilsonlu@hku.hk.
2
1. Introduction
29
Once a built asset is handed over, it is officially put into operation, entering the longest phase of
30
its lifecycle. During this process, facility management is crucial to ensure both functional and
31
structural performance of the facility [1]. Inspection is the cornerstone of facility management
32
[2], which aims to gain up-to-date information of the physical assets to ensure that they are
33
complied with prescribed standards and regulation [3]. To date, facility inspection is conducted
34
manually, where building surveyors or structure and mechanical engineers are dispatched onsite
35
to inspect items indicated by an inspection checklist. This practice is often criticized for its
36
difficult physical presence, low efficiency, and onerous paperwork [3, 4].
37
38
Existing manual facility inspection becomes less and less sustainable as major economies are
39
experiencing shrinking population [5], leading to decreasing workforce in the facility
40
management market. The situation is worsened by the on-going COVID-19 pandemic [6], which
41
makes physical onsite inspection more difficult. The series of challenges have forced the
42
academia and industry to think outside of the box, resulting in many computerized tools to
43
support facility inspection. For example, embedded sensing systems, laser scanning, and Radio
44
Frequency Identification (RFID) technologies have been exploited to expedite facility
45
information collection [7-9]. The use of mobile devices (e.g., smart phones, and tablets) for
46
inspection records documentation has become a new norm, freeing inspectors from
47
overwhelming paperwork [3]. While these technologies have undoubtedly helped inspectors,
48
much of the inspection work still needs to be manually accomplished onsite.
49
50
The rapid advancement of smart technologies provides abundant opportunities for smarter
51
facility inspection. Particularly, the development of artificial intelligence (AI) and robots has
52
been used to replace humans in a wide range of tasks such as floor cleaning and disinfection
53
[10]. Inspired by these applications, pioneering studies have explored the potential of robots in
54
facility inspection. These include the development of robotic systems for post-disaster asset
55
assessment [11], water utility inspection [12], and building facility management [13, 14].
56
However, despite the progress achieved, many of the inspection robots still need to be manually
57
controlled by human operators [11]. Some does have a certain level of autonomy, but they are
58
generally confined to relatively simple tasks, failing to independently respond to the dynamic
59
and complex environments.
60
61
Another highly potential technology is building information modeling/model (BIM). As a digital
62
replica of a built asset, BIM offers a single source of truth wherein all project-related information
63
is stored, processed, and managed in a central hub [15]. Leveraging the rich information in BIM,
64
traditional inspection process has been augmented to assist human decision-making [4, 16, 17].
65
3
BIM can also be linked with robots to allow them better understand of the facility as concerned.
66
Follini et al. [18] leveraged the priori geometric and semantic data in BIM to enable robot
67
perception towards the dynamic and unstructured construction site. Chen et al. [19] proposed a
68
BIM-based global path planning method for ground robot navigation in built environments. Kim
69
et al. [20] studied the viability of using readily-available BIM to model a semantic building
70
world as perceived by a robot. These pioneering studies mainly focuses on devising data
71
interface for construction robot task planning [21]. Nevertheless, much remains unknown on how
72
BIM and robotics can be integrated to equip the robots with a high level of autonomy in facility
73
inspection.
74
75
The automation of facility inspection entails an ability to adaptively response to the complexity
76
posed by the tasks (e.g., “inspection of fire doors in a floor”) and the changing environments
77
(e.g., “encountering human occupants during inspection”). Robots can be equipped with such an
78
ability via a knowledge-driven approach. Tenorth and Beetz [22] stressed the importance of
79
knowledge processing in enabling autonomous robots to do the right thing to the right object in
80
the right way. Thosar et al. [23] performed knowledge-driven reasoning for tool selection in
81
household environments. To facilitate interoperability across robotic platforms and unambiguous
82
reasoning for independent decision-making, a formal representation of knowledge is necessary
83
[24]. Knowledge formalization is a way to structure the unstructured knowledge, which aims to
84
reach a formal, explicit specification of a shared conceptualization, i.e., an ontology [25]. The
85
robotic research community has been active in developing such knowledge representations,
86
resulting in a series of ontologies [26-28]. However, these ontological knowledge models are
87
mainly for industrial applications [27] or household services [22]. Facility inspection has its
88
uniqueness (e.g., the availability of BIM, and the unique workflow of inspection tasks), which
89
calls for a tailor-made knowledge model to drive automated robotized inspection.
90
91
This research aims to propose a knowledge-driven approach that can potentially lead to large-
92
scale automation of facility inspection using robotics and BIM. At the core of the approach is a
93
formalized ontological model encompassing three pillar aspects of facility inspection, namely (a)
94
the scene where a robot operates in, (b) the inspection task, and (3) the robots (agents)
95
themselves. The three aspects of knowledge are seamlessly connected, forming a scalable
96
framework called integrated Scene-Task-Agent (iSTA). The remainder of this paper is organized
97
as follows. Subsequent to this introduction is a literature review. Then, the methodology is
98
introduced in Section 3. Following that, the iSTA knowledge model is presented in Section 4,
99
based on which the knowledge-driven approach for automated facility inspection (AFI) is
100
described in Section 5. The approach is evaluated by experiments in Section 6. Research findings
101
and the strengths and limitations of the study are discussed in Section 7, and conclusions are
102
4
drawn in Section 8.
103
104
2. Related works
105
2.1. BIM and robotics for facility inspection
106
Many studies have applied smart technologies such as BIM and robotics to improve facility
107
inspection productivity. BIM has mainly been explored as an information-rich source to support
108
facility inspection. Liu et al. [4] developed a BIM-augmented system for building inspection,
109
which can help users retrieve project information with ease to assist facility condition
110
assessment. Kopsida and Brilakis [16] proposed a registration method to align reality-captured
111
point cloud with BIM for augmented reality (AR)-based inspection. Baek et al. [17] devised a
112
BIM-integrated AR system for facility management using image-based indoor localization.
113
Despite the supportive roles of BIM by providing on-demand, easy-access, and intuitive project
114
information, the process of facility inspection still needs to be manually implemented.
115
116
To improve efficiency and productivity, robotics is increasingly used in facility inspection. Torok
117
et al. [11] integrated ground robots with computer vision for post-disaster building inspection.
118
Walter et al. [12] developed a robotic system to inspect wastewater treatment facility. Asadi et al.
119
[13] presented a vision-based mobile robots for facility construction inspection. In these
120
applications, the inspection robots need to be controlled by human operators, making physical
121
presence onsite inevitable. Some research efforts have been made to automate the inspection
122
process. For example, Tan et al. [29] proposed an automatic drone-based method for building
123
envelope inspection. Kim et al. [20] explored the applicability of automated robot task
124
planning/execution. However, the achieved automation is usually confined to simple tasks
125
executed in relatively well-controlled environments. To equip robots with high-level autonomy
126
and independent reasoning, a sophisticated knowledge model for facility inspection is necessary.
127
128
2.2. Knowledge-driven robotics and automation
129
Autonomous task implementation can be realized by various approaches. One is to program all
130
detailed activities involved into the robot. Obviously, this approach is not sustainable owing to the
131
onerous programming efforts to cater to every possible environmental change [30]. The other
132
approach is to represent task implementation knowledge in an interoperable and widely accepted
133
format so that the robotic agents can reuse existing knowledge and conduct reasoning to adaptively
134
adjust to the external world [31]. The process of developing a formal, explicit specification of a
135
shared conceptualization is referred to as knowledge formalization [25]. Because of the promise
136
presented by the knowledge-driven approach, the robotics and automation community has been
137
focusing on knowledge formalization in recent years.
138
139
5
Malec et al. [32] proposed to streamline the reconfiguration of manufacturing robots via the
140
knowledge formalization. The research work later evolved into a set of public-available ontologies
141
called ROSETTA [27]. The IEEE-RAS Ontologies for Robotics and Automation Working Group
142
released core ontology for robotics and automation (CORA) [26], which has now become a basic
143
ontology widely used in industrial, surgical, and service robots. Tenorth and Beetz [22] introduced
144
KnowRob (Knowledge processing for Robots), which soon proved itself one of the most influential
145
knowledge processing system for autonomous service robots in household environments. The
146
authors released a second generation of the system called KnowRob 2.0 in 2018 [28]. Other
147
knowledge formalisms have also been developed for domain-specific purposes, e.g., search and
148
rescue [33], and household service [23].
149
150
In the AECO (Architecture, Engineering, Construction and Operation) industry, little work has been
151
done in knowledge formalization for robot autonomy. Neythalath et al. [34] proposed a multi-layer
152
knowledge encapsulation model for adaptive robotic manufacturing, which, however, is primarily for
153
industrial robots. Kim et al. [20] explored the applicability of exploiting an IFC-format BIM for
154
construction robot task planning/execution, of which the effectiveness was evaluated in Gazebo
155
simulation environment. However, they focused more on the data interoperability problem between
156
IFC and unified robot description format (URDF), rather than formalizing a general knowledge
157
model for robotized facility inspection.
158
159
2.3. Limitations of existing studies
160
Our literature review reveals three major knowledge gaps in knowledge-driven AFI:
161
(1) Lack of knowledge formalism for break-down process of facility inspection. Previous efforts
162
mainly focused on representing knowledge of facility management to facilitate data exchange
163
[35], or energy analysis [36]. Only a few has paid attention to activity-level descriptions of
164
facility management tasks, which, however, are either for building renovation [37], or bridge
165
rehabilitation [38]. As a robot needs to understand meaning of task before it can implement it,
166
there is an urgent need to develop a knowledge representation of inspection activities with
167
machine-readable language.
168
(2) Gap between general robot description and domain-specific needs in facility inspection. There
169
are a number of robot knowledge processing systems proposed by the robotics community.
170
Nevertheless, they are either for industrial robots [27], or for general applications of service
171
robots [28]. Facility inspection has its own characteristics that are distinguishing from existing
172
robot ontology, e.g., the availability of BIM, and the unique workflow of inspection process.
173
These domain-specific needs should be considered to extend existing general-purpose robot
174
knowledge representation.
175
(3) Absence of an integrated model to synergize knowledge from the diverse domains of built asset,
176
6
inspection, and robotics. The automation of facility inspection requires robots to have
177
knowledge about the scene they are to explore, awareness of the inspection tasks they are to
178
implement, and self-knowledge on what they are capable of. However, to the best of our
179
knowledge, an integration of the three aspects of knowledge (i.e., scene, task, and agent) to
180
enable AFI has not never been reported in literature.
181
182
3. Methodology
183
This study uses the Methontology approach [39] to developing a knowledge model that will drive
184
AFI using robotics and BIM. Methontology is a methodological paradigm for building ontologies
185
from scratch. It typically involves six steps, namely specification, knowledge acquisition,
186
conceptualization, integration, implementation, and evaluation.
187
188
3.1. Specification
189
The stage of specification aims at specifying general requirements for the ontology to develop, which
190
usually include purpose, scope, source of knowledge, and intended users. Table 1 summarizes the
191
specification of our knowledge model for AFI. It is acknowledged that certain tasks of high
192
complexity (e.g., measuring designated physical quantity in a narrow pump house) are still difficult
193
to fully automate. Therefore, the ontology in this study is focused only on visual inspection tasks.
194
The scope is confined to the inspection of three items, i.e., fire safety, light system, and interior wall.
195
The primary use is to endow robots with autonomy and domain knowledge for facility inspection.
196
However, end-users can also be extended to facility managers/owners who can use the ontology to
197
query robot inspection records. Researchers and robotic developers in the facility management
198
domain are potential beneficiaries as well, who can reuse the ontology to develop robotized facility
199
inspection applications.
200
201
Table 1. iSTA ontology specification
202
Items
Descriptions
Purpose
To formalize concepts and their interrelation concerning
built assets (scene), inspection activities/procedures (task),
and robotic platforms (agents) to enable smart facility
inspection using autonomous service robots
Scope
To focus on visual inspection tasks, including the inspection
of fire resistance system, lighting system, and assurance of
the soundness of interior concrete surface
Users
The primary end-user is service robots to allow them
implement facility inspection autonomously. Other potential
users include facility mangers/owners, and
researchers/developers in the area.
7
Knowledge source
Practitioners, domain experts, facility management
handbook, etc.
203
3.2. Knowledge acquisition
204
The acquisition of knowledge usually follows methods (e.g., interview, text analysis, and survey)
205
developed in social science [40]. This study adopts the following approaches to eliciting knowledge
206
related to RFI. First, we referred to relevant documentation materials for an informal text analysis.
207
The referred documents include building inspection guidelines, ordinance, and facility maintenance
208
handbooks published over the past two decades in Hong Kong. The analysis allows us to have a
209
general understanding of the basic items on facility inspection checklists. Then, professionals from
210
related domains (i.e., facility management and robotics) were interviewed to gain further insights.
211
The interviewees include a real estate manager, two wardens of student residents, and two robotic
212
engineers. A list of questions were prepared based on the disciplinary background of the
213
interviewees, as listed in Table 2.
214
215
It should be noted that specification and knowledge acquisition do not have to be conducted in
216
sequential order. In fact, the two were done simultaneously in this study, where the acquired
217
knowledge can be used to update the specification. For example, the scope initially specified was the
218
general visual inspection tasks. As more and more knowledge solicited (especially via the phone
219
interview with the estate manager), the scope was further refined and updated, which was finally
220
confined to three inspection tasks of “fire safety”, “lighting system” and “interior wall”. We also
221
understand, from the interview that, these tasks were normally performed by the wardens when they
222
do daily patrol in the building. If anomalies are found (e.g., “a flickering light”, “a crack on wall”, or
223
“an unilluminated exit sign”), the wardens should report by taking and uploading photos of the
224
anomalies.
225
226
To execute the tasks by robots, according to the robotics engineers, an inspection should to be
227
assigned to different robots on a floor-by-floor basis. This is because different robots have different
228
locomotion capabilities. To make this assignment possible, knowledge of the robots (e.g., “where
229
they are”) is necessary. In addition, the inspection tasks should be broken down into basic activities
230
such as navigation, obstacle avoidance, and photo taking. To plan the navigation path, position
231
information of the facilities to inspect is needed.
232
233
Table 2. Interview questions for knowledge acquisition.
234
Role
Num. of
interviewees
Questions
8
Real estate
manager
1
• What are the regular inspection items in operation and maintenance
phase of your projects?
• Who perform the mentioned inspection tasks?
• Which manual inspection tasks do you think will be replaced by
robots in the near future?
• What is the inspection frequency?
• How are the inspection results solicited to support maintenance
planning?
Wardens
2
• How do you carry out [xxx] inspection task?
• How often is [xxx] inspection implemented?
• How do you record and report the inspection results?
Robotic
engineers
2
• What information of the scene and the robots would be needed for a
robot to implement [xxx] inspection task?
• How should the [xxx] inspection task be broken down in order to be
implemented by a robot?
* Note: the [xxx] is replaced with specific inspection tasks in the interview.
235
236
3.3. Conceptualization
237
Conceptualization means to structure the obtained knowledge in a conceptual model with a hierarchy
238
of main terms and their relationships [37, 39]. Based on the knowledge acquired by text analysis and
239
interview, this study decided to conceptualize the ontology for AFI from three main branches, that is,
240
the Scene, the Task, and the Agent. Within each branch, corresponding terms and vocabularies are
241
further enumerated to enrich the ontology. For example, under the “Task”, there are the “fmTask”,
242
“fmActivity”, and the “adhocAction”, inter alias; under the “fmTask”, there are then
243
“fmTasFireResist” (fire safety inspection), “fmTasInWallDefect” (interior wall inspection), and
244
“fmTasLightInspect” (lighting system inspection).
245
246
3.4 Integration, implementation and evaluation
247
Integration can facilitate the ontology construction by reusing concepts in existing ontologies.
248
Integration is also an inherent requirement of ontology engineering, which envisions a “shared,
249
common representation and reuse of knowledge” [41]. To take advantages of existing ontologies, we
250
adopted two approaches which are referred to as “vertical integration” and “horizontal integration”,
251
respectively. For vertical integration, basic data schemas, e.g., RDF, RDFS, OWL, XSD, and XML,
252
are incorporated at the bottom to provide basic vocabularies such as the concept of “type”,
253
“property”, and “individual” to support ontology development at higher layers. As for horizontal
254
integration, existing knowledge representations in related domains, e.g., IFC for built environment
255
and CORA for robotic agents, are utilized as backbones of the “Scene” and “Agent” branches [39].
256
257
Implementation refers to the realization of the conceptualized ontology with ontology-editing
258
9
software. In this research, we created the branches of “Task” and “Agent” using the Web Ontology
259
Language (OWL) in Protégé. After an ontology is implemented, next step is to evaluate it. A typical
260
evaluation process includes verification and validation. The former intends to ensure the coherence
261
and correctness of developed ontology, while the latter aims to evaluate whether the ontology can be
262
used to solve the intended engineering problem. After the knowledge model was implemented, we
263
invited the five interviewees at the knowledge acquisition stage to review the ontology for
264
verification. Afterwards, a series of simulation experiments were carried out in a “ROS+Gazebo”
265
environment [41] to validate the effectiveness of the model in enabling AFI.
266
267
4. The developed iSTA knowledge model
268
Using the Methontology approach, a knowledge model is developed for driving automated facility
269
inspection. The model broadly categorizes facility inspection knowledge into three interconnected
270
spheres, i.e., built environments “Scene”, inspection “Tasks”, and robotic “Agents”.
271
272
4.1. Ontology of inspection scene
273
Scene perception is a critical element to form a robot’s autonomy. Such perception is
274
traditionally gained progressively via a “learning by doing” approach as the robot explores its
275
environment. BIM provides an unprecedented source of scene information, which can empower
276
robots for value-added applications such as facility inspection.
277
278
Fig. 1. Graph representation of the iSTA-Scene ontology (A backbone of IFC schema).
279
280
This study adopts the IFC schema, the most recognized knowledge formalization in the built
281
environment, as the scene ontology. An open-source IFC2RDF converter is used to translate the
282
original IFC schema to a knowledge graph format [42]. Fig. 1 shows the backbone structure of
283
the IFC schema that is closely related to the inspection tasks in this study (e.g., fire door and
284
10
lighting inspection). Of the many entities in IFC schema, IfcProduct is of the most interest for
285
robotized inspection (the red boxes in Fig. 1). Under the branch of IfcProduct, the abstract
286
concepts about space (e.g., a floor, or a room) are represented by IfcSpatialStructureElement.
287
This entity is highly relevant, as it will be used to describe the scope of inspection work so that
288
suitable robots can be assigned, and related elements can be retrieved. IfcBuildingElement
289
describes all elements participating in a building system such as walls (IfcWall) and doors
290
(IfcDoor), which will be the entities to inspect in this study. As for the lighting system, we will
291
search the IfcLightFixture for related lighting equipment.
292
293
Positions of the building elements are important information, because the robots rely on them for
294
path planning and navigation (the green boxes in Fig. 1). To retrieve element positions, the
295
IfcLocalPlacement entity of the interested elements will be used to progressively obtain their
296
relative positions. For example, the position of an IfcDoor is not explicitly expressed in IFC;
297
Instead, it is represented as local coordinates (i.e., the IfcAxis2Placement3D) in a recursive
298
manner, e.g., position relative to IfcOpeningElement, then to IfcWall, and IfcBuildingStorey, etc.
299
The relative coordinates at different levels will be used to derive global coordinates to guide the
300
robot navigation.
301
302
Other than position, relationship between building elements and their type properties are also
303
critical (the blue boxes in Fig. 1). As for the former, how building elements are related to each
304
other in a spatial concept (e.g., a room, a storey) will help determine the elements to inspect
305
based on the given scope (e.g., “inspect all the fire doors on the 3rd floor”). Such inclusion
306
relation is encoded in the schema. For the type property, this information will serve as query
307
constraints when retrieving corresponding elements, e.g., to find all the fire doors under the door
308
category. Such information is defined by the IfcTypeObject, and is connected to specific
309
IfcObject through the IfcRelDefinesByType.
310
311
4.2. Ontology of inspection task
312
Fig. 2 shows an overview of the developed task ontology in the iSTA model. Here, a facility
313
inspection task is conceptualized into three interconnected entities. At the top level is the fmTask
314
class, which divides facility inspection tasks into general categories such as the inspection of the
315
fire resisting system, or the inspection of the lighting system. A fmTask can be broken down into
316
multiple fmActivity, e.g., assignment of suitable robotic agents, path planning to navigate to
317
target positions, and taking photos of elements being inspected. Different fmActivity are chained
318
by the “isFollowedBy” property to indicate the implementation sequence. During the execution
319
of an activity, there may be actions the robots need to implement in an ad hoc manner. For
320
example, in the process of navigating to the inspection target, the robot needs to activate collision
321
11
avoidance module when encountering unexpected obstacles. Such actions are represented by the
322
adhocAction entity. All the fmTask, fmActivity and adhocAction are related to properties that
323
define their specific attributes.
324
325
Fig. 2. An overview of the iSTA-task ontology.
326
327
Fig. 3 elaborates the tmTask entity. Subclasses of fmTask represent specific inspection tasks such
328
as the inspection of fire door safety, and lighting system. An inspection task has basic properties
329
such as ID (“hasTasID”), starting time (“hasStartTime”), and finishing time (“hasEndTime”),
330
serving as descriptive information for later enquiry. A task is also related to the sensors needed
331
for the inspection, and the scope of the inspection work. Such information will be initialized
332
when a task is assigned. Last but not least, an inspection task is related to its breakdown
333
fmActivity via the “hasActivities” property. Based on our interview with robotic engineers and
334
estate managers, a taxonomy of typical inspection activities and ad hoc actions is established, as
335
explained in Fig. 4. Several activities are required, including the assignment of robots
336
(fmActAsignRobot), search of building elements to inspect (fmActSearIfcEle), path planning
337
(fmActPathPlan), taking photos of the inspection targets (fmActTakePhoto), and navigation
338
(fmActNavigation). Typical ad hoc actions include collision avoidance and obstacle avoidance,
339
which may need to be activated, respectively, during fmActTakePhoto and fmActNavigation.
340
Notice that each activity/action in the ontology corresponds to a module of python code, which
341
will be executed to drive the robot when a command is issued.
342
343
344
Fig. 3. Ontology entities related to fmTask.
345
346
12
347
Fig. 4. Subclasses of fmActivity, adhocAction and their connections.
348
349
350
Fig. 5 uses the example of fire door inspection to elaborate the iSTA-Task ontology. In the
351
middle is a sequence of the inspection activities involved. The task starts with robot assignment.
352
The assigned robot will be updated to relevant properties (e.g., “isAssignedTo” and “hasAgent”)
353
for later enquiry. The robot assignment activity is followed by a search of fire door IFC elements
354
from the scene ontology based on the given inspection scope (via the “hasSearchScope”
355
property). The retrieved fire door coordinates, along with initial coordinates of the robot, will be
356
forwarded to entity fmActPathPlan, which plans navigation path for the robot to follow. After
357
path planning, the fmActNavigation and the fmActTakePhoto are implemented recursively to
358
navigate the trajectory sections one by one, and take photo of each fire door. The cycle goes on
359
until all trajectory sections are marked as “finished”.
360
361
13
362
Fig. 5. Graph representation of an example inspection task — Fire safety inspection
363
(fmTasFireResist).
364
365
4.3. Ontology of inspection agent
366
To follow the principle of reusability, this study extends existing robotic ontologies to meet the
367
need of facility inspection, as shown in Fig. 6. The Agent ontology is built upon CORA, the core
368
ontology broadly encompassing main notions across the robotics and automation arena [26, 43].
369
CORA is a system comprising modularized ontologies in different levels of axiomatization [24], e.g.,
370
CORA-BARE, CORAX, RPARTS, and SUMO-CORA. Some later ontologies, e.g., ROSETTA, in
371
downstream subdivision are developed based on CORA.
372
373
This research accepts CORA’s definition to consider a robot as both a device and an agent, and
374
borrows the “cora-bare: Robot” as the centered entity (see Fig. 6). New properties are added to
375
describe domain-specific information in facility inspection. For example, the “isStoredAt”
376
property reflects in which space the robots are stored so that the one within the inspection scope
377
can be assigned when a new task is issued. The “isPlacedAt” property, on the other hand, stores
378
the current position coordinates of a robot, which would be used as the starting point for path
379
planning. The geometry of a robot is approximated by the bounding box dimensions of a robot,
380
i.e., the properties of “hasRange_length”, “hasRange_width”, “hasRange_height”. Such self-
381
awareness of geometric information is critical for the robots to avoid collision with objects in the
382
environments.
383
14
384
385
Fig. 6. Graph representation of the iSTA-Agent ontology.
386
387
In the domain of facility inspection, the variation among robot instances is mainly determined by
388
the differences of sensors they equipped. This is because agents with different sensors are
389
suitable for different inspection tasks. For example, a robot with RGB cameras is for fire door
390
inspection (just to take photos of the doors), whereas a robot with infrared thermal sensor is
391
needed to detect concealed defects. The “rparts: robotSensingPart” property from RPARTS is
392
used to delineate the relationship between an agent and its forming sensors. After looking into
393
various robotic ontologies, it is found that ROSETTA has a relatively complete description of
394
different classes of sensors. Therefore, the sensing device entities in ROSETTA are included here
395
to represent different sensors.
396
397
4.4. Integrating the scene-task-agent ontologies
398
The aforementioned ontologies are integrated into a unified knowledge model for AFI. The
399
integration is achieved by reusing well-defined entities from one another. Fig. 7 shows the
400
identified entities that are used across ontologies and their connections. It can be observed that
401
the iSTA-Task has borrowed several concepts related to robot agents and building
402
elements/spaces from iSTA-Agent and iSTA-Scene. In the meantime, iSTA-Agent also reused
403
entities (fmListOfCoor and IfcSpatialStructureElement) defined in its counterparts to describe
404
robot position. To make sense of the indexed entities across ontologies, namespaces (or prefixes)
405
of the origin ontologies need to be cited, e.g., the “core-bare” for Robot entity and the “ifc” for
406
IfcSpatialStructureElement entity.
407
408
15
409
Fig. 7. Schematic diagram showing integration of the scene, task, and agent ontologies.
410
411
5. iSTA-driven framework for automated facility inspection
412
Based on the iSTA knowledge model, an implementation framework for automated facility
413
inspection is developed, as shown in Fig. 8. The framework includes three layers, i.e., the human,
414
the knowledge, and the robot layers. The human layer lies on the top, which is consisted of
415
various actors in facility inspection, i.e., estate managers, inspectors, and engineers. The human
416
staff are not required to carry out the inspection, but only do some periphery works such as
417
setting inspection requirements, and implementing repair works before and after inspection. The
418
robot layer encompasses a variety of robots of different types (e.g., ground robots and drones),
419
which will carry out the inspection. The knowledge layer is made up of iSTA ontologies. It
420
bridges the humans and the robots by receiving inspection instructions on the one hand, and on
421
the other hand, driving the robots to inspect facilities automatically.
422
423
Fig. 8. An implementation framework for automated facility inspection driven by iSTA
424
knowledge model.
425
426
The entire workflow starts by a human facility manager specifying the task type, space scope,
427
and sensors required by the inspection to conduct. The specified task information is forwarded to
428
16
the iSTA-Task ontology in the knowledge layer, where knowledge on the breakdown workflow
429
of the task will be retrieved. The iSTA-Scene and iSTA-Agent complement the iSTA-Task
430
ontology by providing contextual knowledge of the facility and information of the robots. Once
431
proper robots have been assigned based on the given inspection type, scope, and required
432
sensors, the robot will execute inspection activities step by step as indicated by iSTA-Task. As
433
the inspection goes on, the generated inspection data (e.g., inspection ID, assigner, datetime, and
434
photos) will be updated to the iSTA-Task. After finished, the inspection photos will be checked if
435
there is any anomaly of the facilities. The checking can either be down manually or automated
436
with computer vision technologies. If no anomaly is detected, the inspection task is ended, and
437
can be closed. Otherwise, engineers of relevant disciplines should be sent to the site to address
438
the problem (e.g., “to fix a flickering light”) until the anomaly is solved.
439
440
6. Experiments
441
The proposed knowledge-driven approach was evaluated by a series of simulated experiments.
442
The target facility to inspect is a “J” shape, three-floor office building, as shown in Fig. 9 (a).
443
The simulation was implemented in an open-source 3D robotics simulator called Gazebo
444
(version 9.0.0). The Gazebo environment has been integrated with robot operating system (ROS)
445
for robot programming and control. The simulation was run on Lenovo-R720-15IKBN with an
446
Intel Core i5-7300HQ CPU and a Intel HD Graphics 630 GPU.
447
448
Fig. 9. (a) BIM model of the pilot project; (b) The scene model after imported to Gazebo.
449
450
17
451
Fig. 10. Instantiation of the iSTA model based on the case building.
452
453
6.1. Implementation of the iSTA model
454
The proposed iSTA knowledge model was instantiated based on the case building. Fig. 10 shows an
455
overview of the resulted iSTA model. To obtain the iSTA-Scene knowledge base, the Revit model of
456
the case building was first exported to an IFC format (2×3 Coordination View 2.0). The IFC file was
457
then processed and converted to an RDF format [42], which describes an entity as a triple that
458
includes a subject, a predicate, and an object. As for iSTA-Task and iSTA-Agent, we created their
459
representations from scratch in Protégé. Instances of different types of inspection tasks and robotic
460
agents were manually input, which will serve as knowledge bases of the inspection workflow and the
461
available robots for later query operations.
462
463
Note that in iSTA-Task, the instances only store high-level specification of the entities. For example,
464
the workflow for fire door inspection “fmTasFireResist” needs to be specified by instances of
465
18
different “fmActivity” connected by the “isFollowedBy” property. Once put into use, it is expected a
466
further instantiation at lower level is needed, e.g., the inspection task/activity that happened at July
467
30, 2022, or other times. As such instances would continuously accumulated as more and more
468
inspections are carried out, we designate a separate knowledge base called “iSTA-Task-data” to store
469
these instances, which would keep the original iSTA-Task as concise as possible. iSTA-Task-data
470
forms a database wherein all historical inspections are kept in records for future analysis or retrieval.
471
472
6.2. Implementation of iSTA-driven facility inspection
473
Simulated scenarios were carried out to validate the iSTA-driven AFI approach. With the
474
approach, a human expert does not need to be physical onsite or control a robot for the
475
inspection. Rather, he (or she) is only required to designate the scope (e.g., “the 3rd floor”) and
476
the type (e.g., fire safety inspection to ensure all fire doors are closed) of the inspection work.
477
Such scope/task designation can be realized via a computer user interface at a central control
478
room. The designation command will be sent to a central server where the iSTA model is hosted.
479
On receiving the command, knowledge related to the task workflow, inspection scene, and
480
available agents will be extracted to automatically inform the inspect operation without human
481
intervention.
482
483
Suppose a command for “inspecting all fire doors on the 3rd floor” is issued, then the branch of
484
“fmTasFireResist” in the iSTA knowledge graph will be activated. As indicated by the
485
knowledge graph (see Fig. 5), the first activity is to assign the task to a suitable robotic agent.
486
There are three robots in total in our experiments, which are, respectively, stored at the three
487
floors of the building, all equipped with high-resolution cameras. The robotic agent information
488
has been keyed in and represented in the iSTA-Agent graph (as shown in Fig. 11 (c)). According
489
to the required working scope (i.e., “the 3rd floor”) and the needed sensor (i.e., “camera”), the
490
task was assigned to “SahayakBot_01”.
491
19
492
Fig. 11. Implementation of robot assignment: (a) Robots placed at different floors ready for task
493
assignment; (b) A close look of the robot in the third floor; (c) Corresponding graph
494
representation of the robot in iSTA-Agent ontology; (d) Python code for robot assignment.
495
496
Once an inspection robot is assigned, next step is to retrieve information of the elements of
497
interest from the iSTA-Scene ontology. Fig. 12 shows the query code corresponding to the
498
“fmActSearthIfcFire” entity, and the retrieved information of all fire doors on the third floor.
499
There are three fire doors in the range of inspection, of which the coordinates have been
500
retrieved and shown at the button right corner of Fig. 12. Based on the given element coordinates
501
and the robot initial position (i.e., the “isPlacedAt” property), path planning (i.e., the
502
“fmActPathPlan” activity) is then executed to compute the robot navigation trajectory. Fig. 13
503
presents the planned path for the robot to inspect the fire doors one by one. The instantiated
504
“fmTrajSection” entities and their related properties have also been shown in the figure. For
505
example, it can be observed that robot has finished navigating along “fmTrajSection_01”, as its
506
related “ifFinish” is filled with “True”. Similarly, the property “ifBack” of the trajectories
507
indicates that the black dash line is the trajectory path that leads the robot to its original position.
508
20
Such knowledge will inform the inspection robot to execute the planned path step by step, and
509
finally returning to the initial starting point.
510
511
Fig. 12. Implementation of IFC element searching (using fire door search as an example).
512
513
Following the workflow indicated by iSTA-Task, the navigation activity “fmActNavigation” will
514
be activated immediately after the “fmActPathPlan”. Fig. 14 (a) shows that the robot is
515
navigating from fire door ① to fire door ② along the planned trajectory “fmTrajSection_02”.
516
Rviz, a ROS graphical interface, was used to visualize the process from the robot’s perspective.
517
The costmap in Fig. 14 (b) presents a 2D description on the difficulty of traversing different
518
areas of the scene, wherein the pink and wathet regions represent the sensed obstacles and
519
corresponding inflated areas. An inflated area is defined as a buffer zone around the obstacles
520
that should be avoided by the robot planned path. The robot has a depth camera in the front of its
521
base platform, which can scan the environment ahead of the robot (image in the middle of Fig.
522
14 (b)).
523
21
524
Fig. 13. Implementation results of path planning.
525
526
527
Fig. 14. Robot navigation to inspect fire door #2.
528
529
22
530
Fig. 15. Photo taking for visual inspection of (a) fire door, and (b) lighting fixture.
531
532
Once the robot get to the end of a trajectory section, the photo-taking activity (i.e., the
533
“fmActTakePhoto” entity) will be activated to take photo of the target for visual inspection. Fig.
534
15 shows the example scenarios of fire door and lighting fixture inspection. On activation, the
535
“fmActTakePhoto” will first adjust the robot arm’s posture to point the camera towards the target
536
(e.g., a fire door or a lamp). Then, a photograph of the target will be captured and stored.
537
Computer vision algorithms such as deep learning (DL) can be used to process the captured
538
photograph to determine if the inspected elements are compliant with relevant ordinance (e.g.,
539
“the fire door is kept close”). If anomalies are detected, the corresponding competent department
540
should be notified to address the issue in due time.
541
23
542
Fig. 16. Knowledge-driven collision avoidance during robot navigation.
543
544
It is worth-mentioning that the inspection robots are operated in a dynamic environment, with
545
possibility to come across facility occupants. The iSTA knowledge model can inform the robot
546
how to deal with such situation. Fig. 16 simulates a scenario where the robot encounters a human
547
in the corridor. As we mentioned before, the “fmActNavigation” has a property called
548
“hasConcurAction”, which directs to the “ahActAvoidObstacle” entity. This means the collision
549
avoidance will be executed when needed during the navigation. When the robot detects an
550
unexpected obstacle (i.e., a human in this case), the costmap will be updated accordingly, and the
551
collision avoidance mode will be activated. Then, the moving trajectory is re-planned based on
552
the updated costmap to bypass the obstacle. As shown in Fig. 16, the robot is successfully guided
553
by the re-planned trajectory to safety navigate through the human. The autonomy to avoid
554
collision allows the inspection robot to co-exist with humans in dynamic environment.
555
556
7. Discussion
557
As the built environments age [44], the importance of facility inspection has never become so
558
stringent. In face of the global pandemic, traditional manual inspection, characterized by its
559
requirement on physical presence, can no longer sustain itself. Potential automation of facility
560
inspection by the use of robotics and BIM presents a way out. Such automated facility inspection
561
requires the robotic agents to be able to independently react to the changes and complexity of
562
24
their tasks and environments [22]. While pre-programming the robots with “if …, then …” rules
563
can give them a certain level of adaptivity in a controlled environment, it is not suitable in an
564
open, dynamic scenario like facility inspection. The proposed knowledge-driven approach
565
presents an alternative to achieve AFI by equipping the robots with an ability of unambiguous
566
reasoning for independent decision-making.
567
568
The study contributes to the knowledge body from three aspects. First, a knowledge-driven
569
approach is developed to endow robots with knowledge processing and reasoning capability to
570
carry out inspection independently. It opens a new venue to counteract the diminishing
571
productivity in facility management by the application of robotics, BIM and other automation
572
technologies. Second, the developed iSTA framework represents the first of its kind for
573
knowledge modelling in the arena of robotized facility inspection. The iSTA model is a symbolic
574
representation [23] of knowledge covering all spheres (i.e., the facility “scene”, the inspection
575
“task”, and the robotic “agent”) of facility inspection. It is built based upon the reusability
576
principle, and thus can take advantage of existing IFC-formatted BIM to enable scene
577
perception. Third, the iSTA model is made publicly available (github.com/civilServant-
578
666/iSTA). It can help researchers and developers train their robots for automated facility
579
inspection. In addition, it can be further enriched by the research community with formalized
580
knowledge about other tasks in facility inspection.
581
582
Despite the advantages, future research is suggested to further develop the proposed approach.
583
Firstly, the iSTA model represents a knowledge base of high-level concepts (tasks, activities,
584
inspection targets, etc.) in facility inspection. However, not all knowledge in an inspection can be
585
engineered in a “top-down” manner like iSTA modelling. For example, it is difficult to handcraft
586
all features/patterns to teach a robot how to distinguish if a fire door is closed based on the
587
collected photo. Such abilities, nonetheless, can be easily acquired in a “bottom-up” manner by
588
learning from data using deep neural networks. The “top-down” knowledge engineering and
589
“bottom-up” neural nets represent two schools of thoughts in AI, that is, symbolism and the
590
connectionism. There is a growing trend of convergence between the symbolism and
591
connectionism [45] in recent years. For the automation of facility inspection, future research
592
should seek to integrate the symbolic iSTA knowledge model with the connectionism-based DL
593
techniques to make use of advantages of both approaches. Secondly, although effectiveness of
594
the proposed approach has been validated, further efforts are needed to enrich the iSTA model to
595
enable robotic agents to take up more facility inspection tasks in more complicated
596
environments. The iSTA model is intended to provide a high-level conceptual structure upon
597
which further specification and extension can be developed in a relatively straightforward way.
598
Therefore, it is hoped that future research can further develop the iSTA model with more detailed
599
25
task description (e.g., object manipulation), and knowledge on more challenging tasks (e.g.,
600
pump house inspection).
601
602
8. Conclusions
603
In recent years, there is growing momentum to boost facility management productivity by the
604
applications of automation and robotics technologies. Examples of these applications include
605
robots that are increasingly seen in floor cleaning, disinfection, and indoor guidance. In line with
606
the ongoing trend, this research proposes a knowledge-driven approach that can potentially lead
607
to large-scale automation of facility inspection using robotics and BIM. With the Methontology
608
approach, a knowledge model is developed. It encompasses three pillar aspects of facility
609
inspection, i.e., knowledge of the scene where a robot operates, knowledge of the inspection task
610
to carry out, and knowledge of the robots (agents) themselves. BIM is leveraged as a readily-
611
available source of facility information to form the scene knowledge base. The three aspects of
612
knowledge are seamlessly integrated, forming a scalable framework called iSTA. An
613
implementation framework for automated facility inspection is devised based on the iSTA model.
614
615
A series of simulated experiments were carried out to demonstrate the applicability of the
616
proposed approach. It is shown that the iSTA knowledge model can endow robotic agents with
617
autonomy and knowledge to navigate the challenging built environments and deliver facility
618
inspection outcomes. Via the automation based on robotics and BIM, the efficiency and
619
productivity of facility inspection have been improved. We publicized the iSTA model online,
620
hoping that it can be further enriched and can help developers deploy their robotic systems for
621
automated facility inspection.
622
623
Declaration of competing interest
624
The authors declare that they have no known competing financial interests or personal
625
relationships that could have appeared to influence the work reported in this paper.
626
627
References
628
[1] Z. Waheed, S. Fernie, Knowledge based facilities management, Facilities 27(7/8) (2009) 258-266.
629
[2] A. Sharma, How important is inspection module in facility management, 2021.
630
https://factech.co.in/blog/how-important-is-inspection-module-in-facility-management/. (Accessed July
631
21 2022).
632
[3] W. Lu, L. Wu, J. Xu, J. Lou, Construction E-Inspection 2.0 in the COVID-19 Pandemic Era: A
633
Blockchain-Based Technical Solution, Journal of Management in Engineering 38(4) (2022) 04022032.
634
[4] D. Liu, X. Xia, J. Chen, S. Li, Integrating Building Information Model and Augmented Reality for Drone-
635
Based Building Inspection, Journal of Computing in Civil Engineering 35(2) (2021) 04020073.
636
26
[5] F. Tang, China population: workforce to drop by 35 million over next five years as demographic pressure
637
grows, 2021. https://www.scmp.com/economy/china-economy/article/3139470/china-population-
638
workforce-drop-35-million-over-next-five. (Accessed July 21 2022).
639
[6] United Nations Conference on Trade and Development, Impact of the pandemic on trade and development:
640
Transitioning to a new normal, 2020. https://unctad.org/system/files/official-
641
document/osg2020d1_en.pdf. (Accessed July 21 2022).
642
[7] B. Akinci, Using sensor systems and standard project models to capture and model project history for
643
building commissioning and facility management, Proceeding of the Facility Area Network Workshop,
644
2004, pp. 26-27.
645
[8] W. Lu, G.Q. Huang, H. Li, Scenarios for applying RFID technology in construction project management,
646
AUTOMAT CONSTR 20(2) (2011) 101-106.
647
[9] Y.H. Niu, W.S. Lu, D.D. Liu, RFID-Enabled Management System Adoption and Use in Construction:
648
Passing Through the Labyrinth with an Improved Technology Acceptance Model, Springer Singapore,
649
Singapore, 2018, pp. 1251-1258.
650
[10] D. Hu, H. Zhong, S. Li, J. Tan, Q. He, Segmenting areas of potential contamination for adaptive robotic
651
disinfection in built environments, BUILD ENVIRON 184 (2020) 107226.
652
[11] M.M. Torok, M. Golparvar-Fard, K.B. Kochersberger, Image-Based Automated 3D Crack Detection for
653
Post-disaster Building Assessment, Journal of Computing in Civil Engineering 28(5) (2014).
654
[12] C. Walter, J. Saenz, N. Elkmann, H. Althoff, S. Kutzner, T. Stuerze, Design considerations of robotic
655
system for cleaning and inspection of large-diameter sewers, 29(1) (2012) 186-214.
656
[13] K. Asadi, H. Ramshankar, M. Noghabaei, K. Han, Real-Time Image Localization and Registration with
657
BIM Using Perspective Alignment for Indoor Monitoring of Construction, Journal of Computing in Civil
658
Engineering 33(5) (2019) 04019031.
659
[14] K. Asadi, H. Ramshankar, H. Pullagurla, A. Bhandare, S. Shanbhag, P. Mehta, S. Kundu, K. Han, E.
660
Lobaton, T. Wu, Vision-based integrated mobile robotic system for real-time applications in construction,
661
AUTOMAT CONSTR 96 (2018) 470-482.
662
[15] R. Sacks, C. Eastman, G. Lee, P. Teicholz, BIM handbook: a guide to building information modeling for
663
owners, designers, engineers, contractors, and facility managers, John Wiley & Sons, 2018.
664
[16] M. Kopsida, I. Brilakis, BIM registration methods for mobile augmented reality-based inspection, Ework
665
and Ebusiness in Architecture, Engineering and Construction (2016) 201-208.
666
[17] F. Baek, I. Ha, H. Kim, Augmented reality system for facility management using image-based indoor
667
localization, AUTOMAT CONSTR 99 (2019) 18-26.
668
[18] C. Follini, V. Magnago, K. Freitag, M. Terzer, C. Marcher, M. Riedl, A. Giusti, D.T. Matt, BIM-
669
Integrated Collaborative Robotics for Application in Building Construction and Maintenance, Robotics
670
10(1) (2021) 2.
671
[19] Z. Chen, K. Chen, C. Song, X. Zhang, J.C.P. Cheng, D. Li, Global path planning based on BIM and
672
physics engine for UGVs in indoor environments, AUTOMAT CONSTR 139 (2022) 104263.
673
[20] S. Kim, M. Peavy, P.-C. Huang, K. Kim, Development of BIM-integrated construction robot task
674
planning and simulation system, AUTOMAT CONSTR 127 (2021) 103720.
675
[21] Q. Chen, J. Chen, W. Huang, Pathfinding method for an indoor drone based on a BIM-semantic model,
676
ADV ENG INFORM 53 (2022) 101686.
677
27
[22] M. Tenorth, M. Beetz, KNOWROB — knowledge processing for autonomous personal robots, 2009
678
IEEE/RSJ International Conference on Intelligent Robots and Systems, 2009, pp. 4261-4266.
679
[23] M. Thosar, S. Zug, A.M. Skaria, A. Jain, A Review of Knowledge Bases for Service Robots in Household
680
Environments, AIC, 2018, pp. 98-110.
681
[24] S. Fiorini, J. Carbonera, P. Gonçalves, V. Jorge, V. Rey, T. Haidegger, M. Abel, S. Redfield, S.
682
Balakirsky, V.R. Sampath Kumar, Extensions to the core ontology for robotics and automation, Robotics
683
and Computer-Integrated Manufacturing 33 (2014).
684
[25] P. Pauwels, A. Costin, M.H. Rasmussen, Knowledge Graphs and Linked Data for the Built Environment,
685
in: M. Bolpagni, R. Gavina, D. Ribeiro (Eds.), Industry 4.0 for the Built Environment: Methodologies,
686
Technologies and Skills, Springer International Publishing, Cham, 2022, pp. 157-183.
687
[26] E. Prestes, S.R. Fiorini, J. Carbonera, Core ontology for robotics and automation, Proceedings of the 18th
688
Workshop on Knowledge Represen-tation and Ontologies for Robotics and Automation, 2014, p. 7.
689
[27] M. Stenmark, J. Malec, Knowledge-based industrial robotics, Twelfth Scandinavian Conference on
690
Artificial Intelligence, IOS Press, 2013, pp. 265-274.
691
[28] M. Beetz, D. Beßler, A. Haidu, M. Pomarlan, A.K. Bozcuoğlu, G. Bartels, Know Rob 2.0 — A 2nd
692
Generation Knowledge Processing Framework for Cognition-Enabled Robotic Agents, 2018 IEEE
693
International Conference on Robotics and Automation (ICRA), 2018, pp. 512-519.
694
[29] Y. Tan, S. Li, H. Liu, P. Chen, Z. Zhou, Automatic inspection data collection of building surface based on
695
BIM and UAV, AUTOMAT CONSTR 131 (2021) 103881.
696
[30] A. Olivares-Alarcos, D. Beßler, A. Khamis, P. Goncalves, M.K. Habib, J. Bermejo-Alonso, M. Barreto,
697
M. Diab, J. Rosell, J. Quintas, A review and comparison of ontology-based approaches to robot
698
autonomy, The Knowledge Engineering Review 34 (2019).
699
[31] M. Beetz, D. Jain, L. Mosenlechner, M. Tenorth, L. Kunze, N. Blodow, D. Pangercic, Cognition-enabled
700
autonomous robot control for the realization of home chore task intelligence, Proceedings of the IEEE
701
100(8) (2012) 2454-2471.
702
[32] J. Malec, A. Nilsson, K. Nilsson, S. Nowaczyk, Knowledge-based reconfiguration of automation systems,
703
2007 IEEE International Conference on Automation Science and Engineering, IEEE, 2007, pp. 170-175.
704
[33] C. Schlenoff, E. Messina, A robot ontology for urban search and rescue, Proceedings of the 2005 ACM
705
workshop on Research in knowledge representation for autonomous systems, 2005, pp. 27-34.
706
[34] N. Neythalath, A. Søndergaard, J.A. Bærentzen, Adaptive robotic manufacturing using higher order
707
knowledge systems, AUTOMAT CONSTR 127 (2021) 103702.
708
[35] W. Chen, M. Das, K. Chen, J.C. Cheng, Ontology-based data integration and sharing for facility
709
maintenance management, Construction Research Congress 2020: Computer Applications, American
710
Society of Civil Engineers Reston, VA, 2020, pp. 1353-1362.
711
[36] N.M. Tomašević, M.Č. Batić, L.M. Blanes, M.M. Keane, S. Vraneš, Ontology-based facility data model
712
for energy management, ADV ENG INFORM 29(4) (2015) 971-984.
713
[37] J.A.P. Amorocho, T. Hartmann, Reno-Inst: An ontology to support renovation projects planning and
714
renovation products installation, ADV ENG INFORM 50 (2021) 101415.
715
[38] C. Wu, P. Wu, J. Wang, R. Jiang, M. Chen, X. Wang, Ontological knowledge base for concrete bridge
716
rehabilitation project management, AUTOMAT CONSTR 121 (2021) 103428.
717
28
[39] M. Fernández-López, A. Gómez-Pérez, N. Juristo, METHONTOLOGY: From Ontological Art Towards
718
Ontological Engineering, AAAI 1997, 1997.
719
[40] T. Hartmann, A. Trappey, Advanced Engineering Informatics - Philosophical and methodological
720
foundations with examples from civil and construction engineering, Developments in the Built
721
Environment 4 (2020) 100020.
722
[41] N.F. Noy, D.L. McGuinness, Ontology development 101: A guide to creating your first ontology,
723
Stanford knowledge systems laboratory technical report KSL-01-05 and …, 2001.
724
[42] P. Pauwels, W. Terkaj, EXPRESS to OWL for construction industry: Towards a recommendable and
725
usable ifcOWL ontology, AUTOMAT CONSTR 63 (2016) 100-133.
726
[43] E. Prestes, J.L. Carbonera, S. Rama Fiorini, V.A. M. Jorge, M. Abel, R. Madhavan, A. Locoro, P.
727
Goncalves, M. E. Barreto, M. Habib, A. Chibani, S. Gérard, Y. Amirat, C. Schlenoff, Towards a core
728
ontology for robotics and automation, Robotics and Autonomous Systems 61(11) (2013) 1193-1204.
729
[44] T. Hartmann, Keynote Session: Digitizing Building Renovation Projects, 2020.
730
https://www.youtube.com/watch?v=qfZFIGWBm_M. (Accessed Jan. 12 2022).
731
[45] F. Caversan, Symbolism Versus Connectionism In AI: Is There A Third Way?, 2020.
732
https://www.forbes.com/sites/forbestechcouncil/2020/09/01/symbolism-versus-connectionism-in-ai-is-
733
there-a-third-way/?sh=5ed634f37549. (Accessed August 8 2022).
734
735
