An Integrated Solution for Automatic 3D Object-based Information Retrieval

Abstract

As-is building information model (BIM) is regarded as the mainstream solution of the digital twin (DT) for the intelligent building management, especially in the facilities management (FM) phase for the existing buildings. The current automatic scan-to-BIM methods mainly focus on the detailed geometric information modelling. However, the attributes information of the 'secondary' building objects is equally valuable comparing to that of the primary structural objects in the FM workflow. These components may include the light fixture, plumbing and heating terminal and furniture. The knowledge supporting to the FM practice can be extracted based on the geometric and attribute information. Therefore, the 'secondary' building components should be efficiently modelled as the main operation and maintenance targets in the FM phase. This paper proposes an automatic 'secondary' object-based BIM model retrieval method based on segmented point cloud model. The machine learning (ML) supported technical conceptual framework will be introduced in this paper.

Full text

An Integrated Solution for Automatic 3D Object-based Information Retrieval
An Integrated Solution for Automatic 3D Object-based Information Retrieval
Ya W
.
1, Mudan W.1, Mahendrini A.1, Guangming W.1, Ioannis B.1 and Lizhao X.2
1University of Cambridge, Cambridge, United Kingdom, CB2 1PZ; email: yw710@cam.ac.uk
1University of Cambridge, Cambridge, United Kingdom, CB2 1PZ; email: mw923@cam.ac.uk
1University of Cambridge, Cambridge, United Kingdom, CB2 1PZ; email: mfa47@cam.ac.uk
1University of Cambridge, Cambridge, United Kingdom, CB2 1PZ; email: gw462@cam.ac.uk
1University of Cambridge, Cambridge, United Kingdom, CB2 1PZ; email: ib340@cam.ac.uk
2University of Hong Kong, Hong Kong SAR; email: u3008759@connect.hku.hk
ABSTRACT
As-is building information model (BIM) is regarded as the mainstream solution of the digital
twin (DT) for the intelligent building management, especially in the facilities management (FM)
phase for the existing buildings. The current automatic scan-to-BIM methods mainly focus on
the detailed geometric information modelling. However, the attributes information of the
‘secondary’ building objects is equally valuable comparing to that of the primary structural
objects in the FM workflow. These components may include the light fixture, plumbing and
heating terminal and furniture. The knowledge supporting to the FM practice can be extracted
based on the geometric and attribute information. Therefore, the ‘secondary’ building
components should be efficiently modelled as the main operation and maintenance targets in the
FM phase. This paper proposes an automatic ‘secondary’ object-based BIM model retrieval
method based on segmented point cloud model. The machine learning (ML) supported technical
conceptual framework will be introduced in this paper.
INTRODUCTION
Point cloud data (PCD) currently serves as one of the primitive representations to aid in
information modelling of existing building assets. Numerous researchers have developed the
integrated solutions to convert PCD into the DT model. However, the current methods
predominately focus on the geometric information modelling leading to that the value of DTs
cannot be realised. The attached real-world attribute information attached is more important in
the FM applications, such as operation and maintenance (O&M) management, asset management
and energy management (Lu et al., 2020; Wen et al., 2020). There is rare research discusses
about how to link the real-world building information to the created DT to facilitate its practical
value. This leads to a significant gap for practitioners in utilising the constructed DT via the
existing methods due to lack of knowledge output beyond geometric information. To address the

An Integrated Solution for Automatic 3D Object-based Information Retrieval
challenge, this paper proposes an object-based ‘secondary’ building componence information
retrieval method based on the BIM object libraries as key training datasets.
Unlike structural components, these 'secondary' components (Adán et al., 2018), for
example, the attributes of lighting fixtures are useful for the energy consumption calculations.
The BIM object databases are designed to include with these valuable attribute information as
well as the geometric information to meet the multiple management requirements in the FM
phase. By matching PCD with BIM objects, we can add valuable practical to enrich the DT
model. Zeng et al. (2020) developed a user exemplar-based method method to create a query
object and retrieve matching instances based on metric learning, and Adán et al. (2018)
developed a DT reconstruction method using an object database. However, it is still a manual
process to identify and match the corresponding digital object. The objective of this paper is to
develop an integrated object-driven geometric and attribute information retrieval framework to
realise the automatic knowledge modelling of DT. It leads to the research question of how to
realise the object-driven information retrieval based on the collected PCD. The contribution of
this research is to build the conceptual building information retrieval framework to enhance the
practical value of as-is DTs. The other contribution is the integrated neural network to
automatically identify the corresponding BIM object matching with the PCD cluster.
METHODOLOGY
Figure 1 Research methodology
There are three steps involved in this framework (see Figure 1), which are: a. geometric
information retrieval; b. attribute information retrieval; and c. knowledge extraction. The first
step involves using a ML-based network to retrieve geometric information, which is the
foundational stage where the shape and dimensions of the building are identified and captured.
Once the corresponding BIM object is matched, its associated attribute information is retrieved.
The attribute information matching in the next step mainly depends on a project building object
database. It is crucial that the geometric and attribute information are already paired and stored,
typically by estate management or an equivalent department. Finally, the geometric and attribute
information will be function-centred organised in the ontology-based knowledge graph to
significantly improves the practical application of the DT from the point cloud data. In this
paper, the technical solution of the geometric information retrieval using advanced ML
techniques will be mainly described.
FRAMEWORK OF AUTOMATIC 3D OBJECT-BASED INFORMATION RETRIEVAL

An Integrated Solution for Automatic 3D Object-based Information Retrieval
Geometric information retrieval of 3D object. The geometric information retrieval is powered
by the integrated ML network (see Figure 2). The lighting fixtures are listed as the examples
shown in the figure.
Figure 2 Technical pipeline of geometric information retrieval
• Data preparation: Two sets of data should be prepared, which are the Anchor Object
point cloud model (AO) and the BIM object library. AO (AO1, AO2,…, AOn) in the Figure
2 is the target of the retrieval process, which is a specific part of the point cloud model in
a building asset that has been segmented and labelled. The BIM object library consists of
two parts, which are the general BIM object library and project database. The general
library is used for extracting features from specific categories of building elements and its
3D objects need to be converted into point cloud format. The project 3D object library
refers to the restricted dataset which should be provided by property stakeholders,
contains the positive selection of BIM objects (POs) matching with the AOs. It's crucial
for this library to label each pair of PO (POx) and AO (AOx) clearly, which is necessary
for the supervised learning process in the neural network.
• Initial 3D object matching: This step includes two parts: feature extraction using
Multilayer Perceptron (MLP) supported neural network and subsequent similarity
calculation via Cosine Similarity (CS). MLP is a point-based method especially effective
for dealing with non-uniform point density which are widely used in PCD processing
work (Rauch & Braml, 2023). In the MLP neural network, the input layer receives the
point cluster of AOs. The network has multiple hidden layers, where neurons apply
activation functions to the inputs, facilitated by a system of weighted connections. The
feature vectors will be extracted through the iterative feedforward and backpropagation
process (Qi et al., 2017). After extracting the global signature of the point features from
both the AOs and the objects in the BIM object library, the framework employs CS
algorithm to identify the EO that bears the highest similarity to the AO. It is mainly used
for the point-based similarity calculation (Collins et al., 2023; Huang & You, 2012). In
the below equation 1), A and X represents the feature vector extracted from BIM object
library and AO respectively. The framework proposes calculating the output features of

An Integrated Solution for Automatic 3D Object-based Information Retrieval
both datasets (the AO and the BIM objects) rather than directly comparing raw point
cloud data, which is more effective due to the feature vectors are structured.
𝐶𝑜𝑠𝑖𝑛𝑒'𝑆𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦'(𝐴,𝐵)=𝐴∙𝑋
‖𝐴‖‖𝑋‖=∑𝐴!𝑋!
"
!#$
9∑𝐴!
%"
!#$ ∙9∑𝑋!
%"
!#$
' '1)
• Matching refinement: The matching process will be further optimized based on the
Triplet Loss (TL) method, which is a technique initially developed for classifying highly
similar objects, such as in face recognition (Schroff et al., 2015). As shown in the below
equation 2), the process involves: the anchor A, which is AO; the positive sample P, the
matched object PO that corresponds to the AO; and the negative sample N, the EO
identified in the previous step. The Loss Function L measures the distances between these
samples with d(A,N) presenting the distance between the anchor and positive sample, and
d(A,N) the distance between the anchor and negative sample. The ‘margin’ is a
hyperparameter setting the minimum distance between dissimilar pairs. The optimisation
of the feature extraction neural network is achieved through iterative processing using the
gradient descent method on the L until the accuracy achieves a specific threshold for
precise object matching.
𝐿'(𝐴,𝑃,𝑁)=max(𝑑(𝐴,𝑃)−𝑑(𝐴,𝑁)+𝑚𝑎𝑟𝑔𝑖𝑛,0)2)
Attribute information retrieval and knowledge extraction. Once the matched BIM object is
identified, its associated attribute information can be retrieved. The attribute data represents the
real-world attribute of the specific building ‘secondary’ objects. Take the lighting fixture as an
example, the attribute can cover the information about material, power, light source, light colour,
installation method etc. By gathering geometric and attribute data, the knowledge framework can
be user-centred designed, which is based on the integration of industrial knowledge graphs.
Consequently, the practical value of DTs can be enhanced in the real-world scenarios.
DISCUSSION AND CONCLUSION
This paper introduces an innovative framework to match 3D BIM objects for the target point
cloud objects and it faces certain limitations that might need to be addressed. The primary
limitation is the framework dependence on a pre-existing project library of 3D objects for both
training and retrieval proposes. It operates under the assumption that the property stakeholders
will supply the necessary information of the building assets. To tackle this limitation, two
scenarios are proposed: The first scenario is the stakeholders are possessing the complete 3D
BIM object library of their assets, necessitating the inclusion of both general and property-
specific 3D BIM object databases in the training dataset. The second scenario caters to the
situations where stakeholders have only attribute and dimension information without geometric
information. Under this circumstance, the solution might be to develop an information query
framework to locate the object in the general BIM object library with high-similarity geometric
information, subsequently aligning it with the real-world asset's attribute information. This

An Integrated Solution for Automatic 3D Object-based Information Retrieval
solution is proposed since the level of details of the object geometric information is less
important than the attribute information and the current general object libraries typically cover a
wide range of product geometric information and various design styles of common building
components.
This paper proposes an integrated solution for extracting the 3D object-based information
from scanned point cloud data aiming to address the research question. The future research will
concentrate on four main areas: 1. testing and verifying the effectiveness of the neural network in
retrieving information based on the case studies; 2. training the BIM object database for the
neural network; 3. Adjusting the matched BIM object parametric and positioning in the DTs and
4. creating a linked data structure that organises both geometric and attribute information.
REFERENCES
Adán, A., Quintana, B., Prieto, S. A., & Bosché, F. (2018). Scan-to-BIM for ‘secondary’
building components. Advanced Engineering Informatics, 37, 119–138.
https://doi.org/10.1016/j.aei.2018.05.001
Collins, F. C., Braun, A., & Borrmann, A. (2023). Finding Geometric and Topological
Similarities in Building Elements for Large-Scale Pose Updates in Scan-vs-BIM. Lecture
Notes in Civil Engineering, 357(September), 517–530. https://doi.org/10.1007/978-3-031-
35399-4_37
Huang, J., & You, S. (2012). Point cloud matching based on 3D self-similarity. IEEE Computer
Society Conference on Computer Vision and Pattern Recognition Workshops, 41–48.
https://doi.org/10.1109/CVPRW.2012.6238913
Lu, Q., Xie, X., Parlikad, A. K., & Schooling, J. M. (2020). Digital twin-enabled anomaly
detection for built asset monitoring in operation and maintenance. Automation in
Construction, 118, 103277. https://doi.org/https://doi.org/10.1016/j.autcon.2020.103277
Qi, C. R., Yi, L., Su, H., & Guibas, L. J. (2017). PointNet++: Deep Hierarchical Feature
Learning on Point Sets in a Metric Space. http://arxiv.org/abs/1706.02413
Rauch, L., & Braml, T. (2023). Semantic Point Cloud Segmentation with Deep-Learning-Based
Approaches for the Construction Industry: A Survey. Applied Sciences (Switzerland),
13(16). https://doi.org/10.3390/app13169146
Schroff, F., Kalenichenko, D., & Philbin, J. (2015). Facenet: A unified embedding for face
recognition and clustering. Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, 815–823.
Wen, Y., Tang, L. C. M., & Ho, D. C. W. (2020). A BIM-based space-oriented solution for
hospital facilities management. Facilities, 39(11–12), 689–702. https://doi.org/10.1108/F-
10-2019-0105
Zeng, S., Chen, J., & Cho, Y. K. (2020). User exemplar-based building element retrieval
from raw point clouds using deep point-level features. Automation in Construction,
114. https://doi.org/10.1016/j.autcon.2020.103159