Project

Generation of semantically rich as-built Building Information Models (BIMs): A derivative-free optimization (DFO) approach

Goal: There is a dilemma in the generation of semantically rich as-built BIMs: On one hand, it is inherently challenging to segment non-geometric semantics from photos and point clouds; On the other hand, free semantic BIM components are booming on the Internet and elsewhere. This project attempts to solve the dilemma by:

1. To develop a general transformation to model as-built BIM generation as a nonlinear optimization problem of fitting components;
2. To establish a semantic BIM component dataset to integrate open access BIM resources; and
3. To adopt and fine-tune state-of-the-art DFO algorithms for automated generation of semantically rich as-built BIMs.

Methods: OpenCV, Model-Driven Development, Point Clouds, 3D Computer Vision, RMSD, Building Information Modelling, as-built BIM, derivative-free optimization, CMA-ES, DIRECT, optimization by simulation, structural similarity (SSIM), scene understanding, NEWUOA, automatic modeling, semantic enrichment, urban semantics, super voxelization

Date: 1 January 2018 - 31 December 2020

Updates
0 new
1
Recommendations
0 new
0
Followers
0 new
31
Reads
0 new
326

Project log

Fan Xue
added a research item
Coarse registration of 3D point clouds plays an indispensable role for parametric, semantically rich, and realistic digital twin buildings (DTBs) in the practice of GIScience, manufacturing, robotics, architecture, engineering, and construction. However, the existing methods have prominently been challenged by (i) the high cost of data collection for numerous existing buildings and (ii) the computational complexity from self-similar layout patterns. This paper studies the registration of two low-cost data sets, i.e., colorful 3D point clouds captured by smartphones and 2D CAD drawings, for resolving the first challenge. We propose a novel method named `Registration based on Architectural Reflection Detection’ (RegARD) for transforming the self-symmetries in the second challenge from a barrier of coarse registration to a facilitator. First, RegARD detects the innate architectural reflection symmetries to constrain the rotations and reduce degrees of freedom. Then, a nonlinear optimization formulation together with advanced optimization algorithms can overcome the second challenge. As a result, high-quality coarse registration and subsequent low-cost DTBs can be created with semantic components and realistic appearances. Experiments showed that the proposed method outperformed existing methods considerably in both effectiveness and efficiency, i.e., 49.88% less error and 73.13% less time, on average. The RegARD presented in this paper first contributes to coarse registration theories and exploitation of symmetries and textures in 3D point clouds and 2D CAD drawings. For practitioners in the industries, RegARD offers a new automatic solution to utilize ubiquitous smartphone sensors for massive low-cost DTBs.
Fan Xue
added a research item
[ISPRS JPRS' 2020 Featured Articles] Recent advancement of remote sensing technologies has brought in accurate, dense, and inexpensive city-scale Light Detection And Ranging (LiDAR) point clouds, which can be utilized to model city objects (e.g., buildings, roads, and automobiles) for creating Digital Twin Cities (DTCs). However, processing such unstructured point clouds is very challenging, epitomized by high cost, movable objects, limited object classes, and high information inadequacy/redundancy. We noticed that many city objects are not in random shapes; rather, they have invariant cross-sections following the Gestalt design principles, including proximity, connectivity, symmetry, and similarity. In this paper, we present a novel unsupervised method, called Clustering Of Symmetric Cross-sections of Objects (COSCO), to process urban LiDAR point clouds to a hierarchy of objects based on their characteristic cross-sections. First, city objects are segmented as connected patches of proximate 3D points. Then, symmetric cross-sections are detected for symmetric city objects. Finally, the taxonomy and groups of city objects are recognized from a hierarchical clustering analysis of the dissimilarity matrix. Experimental results showed that COSCO detected the correct taxonomy and types of 12 cars from 24,126 LiDAR points in 8.28s. Based on the cross-sections and taxonomy, a digital twin was created by registering online free 3D car models in 29.58s. The contribution of this paper is twofold. First, it presents an effective unsupervised method for understanding and developing DTC objects in LiDAR point clouds by harnessing innate Gestalt design principles. Secondly, COSCO can be an efficient LiDAR pre-processing tool for recognizing symmetric city objects' cross-sections, positions, heading directions, dimensions, and possible types for smart city applications in GIScience, Architecture, Engineering, Construction and Operation (AECO), and autonomous vehicles.
Fan Xue
added a research item
Those attempting to integrate building information modeling (BIM) and blockchain soon encounter the enormous challenge of information redundancy. Storage of duplicated building information in decentralized ledgers already creates redundancy, and this is exacerbated as the BIM model develops and is utilized. This paper presents a novel semantic differential transaction (SDT) approach to minimizing information redundancy in the nascent field of BIM and blockchain integration. Whereas the conventional thinking is to store an entire BIM model or its signature code in blockchain, SDT captures local model changes as SDT records and assembles them into a BIM change contract (BCC). In this way, the version history of a BIM project becomes a chain of timestamped BCCs, and stakeholders can promptly synchronize BIM changes in blockchain. We test our approach in two pilot cases. The results show that SDT captures, in near real time, sequential and simultaneous BIM changes at less than 0.02% of the Industry Foundation Classes file size. We also prove model restoration from the lightweight BCCs in a small-scale BIM project. In addressing the fundamental issue of information redundancy in BIM and blockchain integration, this research can help the industry advance beyond the rhetoric to develop operable blockchain BIM systems.
Fan Xue
added a research item
Although the value of 3D point cloud data (PCD) has been increasingly recognized by the architectural, engineering, construction and facility operations (AECO) sectors, there is much less actual application of PCD in facility management (FM) than other stages. In order to facilitate the exploration of using PCD for FM, this study aims to summarize existing research effort and identify the gaps based on a systematic review of previous studies touching upon the PCD-enabled FM. This review was guided by a conceptual model that consists of four key components associated with PCD application process, including target objects, PCD sensing, model output and applications. 47 papers published in 21 academic journals were collected for the analysis. It was found that Light Detection and Ranging (LiDAR), photogrammetry, and Synthetic Aperture Radar (SAR) were the three mostly used technologies for collecting the PCD. The raw signals, such as fragments of point cloud and photos, collected by these technologies need to be pre-processed for generating the PCD, and segmentation and meshing are two general aspects of PCD post-processing to create models. It was also found that most studies focused on geometric properties, data processing, feature extraction, object recognition, and model generation, seldom would they dig deeper for decision-making support of FM applications. Based on the results, three major gaps of PCD-enabled FM were concluded, including (1) overlooking the valuable non-geometric properties (e.g. specifications of materials, relations between objects); (2) less focusing on providing decision support functions; and (3) hovering at data level rather than information level. Eleven possible research directions including semantics enrichment, real-time model generation, longitudinal analysis, and smart living applications of PCD-enabled FM were thus suggested for future research.
Fan Xue
added a research item
Purpose The practice of facility management (FM) has been evolving with the rapid development of pervasive sensing technologies (PSTs) such as sensors, automatic identification (auto-ID), laser scanning and photogrammetry. Despite the proliferation of research on the use of PSTs for FM, a comprehensive review of such research is missing from the literature. This study aims to cover the knowledge void by examining the status quo and challenges of the selected PSTs with a focus on FM. Design/methodology/approach This paper reviewed 204 journal papers recounting cases of using PSTs for FM. The reviewed papers were extracted from Elsevier Scopus database using the advanced search. Findings Findings of this study revealed that PSTs and FM applications form a many-to-many mapping, i.e. one PST could facilitate many FM applications, and one application can also be supported by various PSTs. It is also found that energy modeling and management is the most referred purpose in FM to adopt PSTs, while space management, albeit important, received the least attention. Five challenges are identified, which include high investment on PSTs, data storage problem, absence of proper data exchange protocols for data interoperability, a lack of mature data processing methods for data utilization and privacy of users. Originality/value This paper paints a full picture of PSTs adoption for FM. It pinpoints the promising explorations for tackling the key challenges to future development.
Fan Xue
added 2 research items
As demanded by smart city applications, the recognition and enrichment of urban semantics from unstructured spatial big data became an emerging trend for the development of building information model (BIM) and city information model (CIM). Rooftop constructs the essential part of BIM and CIM and loads various new application practices and scenarios. The recognition and enrichment of rooftop elements represent the trending requirements. This study develops a new approach for semantic enrichment of aerial Light Detection and Ranging (LiDAR) point clouds. In this paper, machine learning models such as decision tree are applied to predict green roof elements based on the geometry and laser reflectance, and was validated in a pilot zone in the main campus of The University of Hong Kong. The recognized rooftop elements could provide a solid foundation for further research, such as rooftop landscape, rooftop energy, rooftop farming.
In the era of smart city, semantically rich city information models (CIMs) are demanded as a critical information hub. Roof albedo, a semantic property measures how much solar radiation is reflected, is vital to various urban sustainability topics, including heat island, local climate, green roof, and urban morphology. This paper presents an approach that enriches LiDAR-based albedo to rooftop models for CIM. First, we apply Chen et al. (2018)’s method to the reconstruction of the geometries of rooftop elements. Then, albedos of roofs and rooftop elements are estimated from the mean reflectance in LiDAR data. A pilot study was conducted in an urban area in Hong Kong. The results showed that the building models created by the presented approach were satisfactory in terms of rooftop elements and roof albedos. The results from the present approach can provide sustainability study the details of 3D geometries and albedos in an urban area.
Fan Xue
added a research item
Reconstructing semantically rich building information model (BIM) from 2D images or 3D point clouds represents a research realm that is gaining increasing popularity in architecture, engineering, and construction. Researchers have found that architectural design knowledge, such as symmetry, planarity, parallelism, and orthogonality, can be utilized to improve the effectiveness of such BIM reconstruction. Following this line of enquiry, this paper aims to develop a novel semantic registration approach for complicated scenes with repetitive, irregular-shaped objects. The approach first formulates the architectural repetition as the multimodality in mathematics. Thus, the reconstruction of repetitive objects becomes a multimodal optimization (MMO) problem of registering BIM components which have accurate geometries and rich semantics. Then, the topological information about repetition and symmetry in the reconstructed BIM is recognized and regularized for BIM semantic enrichment. A university lecture hall case, consisting of 1.9 million noisy points of 293 chairs, was selected for an experiment to validate the proposed approach. Experimental results showed that a BIM was satisfactorily created (achieving about 90% precision and recall) automatically in 926.6 s; and an even more satisfactory BIM achieved 99.3% precision and 98.0% recall with detected semantic and topological information under the minimal effort of human intervention in 228.4 s. The multimodality model of repetitive objects, the repetition detection and regularization for BIM, and satisfactory reconstruction results in the presented approach can contribute to methodologies and practices in multiple disciplines related to BIM and smart city.
Fan Xue
added a research item
Digital twin city (DTC) is a critical information infrastructure that enables many innovative applications for smart and resilient city development. Thanks to the recent advances in remote sensing and photogrammetry, accurate, dense, and large-scale 3D urban point clouds become increasingly available for many cities for creating and updating their DTCs. Because of the immense amount and the high update frequency of urban point clouds, it is too time-consuming and labor-intensive to create and update DTCs solely by human experts. Researchers have developed a wealth of automatic and semi-automatic methods for processing 3D urban point clouds using expert knowledge of the built environment, supervised learning, and reinforced learning of geometric primitives and components. However, these methods are restricted, ironically by the embedded knowledge, in the scalability to sophisticated scenes and the availability of standardized components. Inspired by the success of Google' unsupervised learning program AlphaZero, this paper proposes a novel hierarchical clustering approach for semantic enrichment of point clouds. Unlike the existing approaches relying on fixed domain knowledge, extra correlational training examples, or available 3D references, the proposed approach exploits the similarities between patches of point clouds without explicit domain knowledge. The proposed approach first segments patches from the input point cloud through the connected subgraphs of voxel grids, then computes the dissimilarity matrix between the patches via iterative optimization. Subsequently, the dissimilarity engenders a hierarchy of clusters for understanding the relatedness between the patches. A pilot study on a real urban scene showed that the proposed approach is feasible and potent to cluster and detect objects automatically. Another experiment showed that the dissimilarity-based clusters and associated transformations can help create semantic objects for DTC, as referential 3D models are available.
Fan Xue
added a research item
Development of semantically rich as-built building information models (BIMs) presents an ongoing challenge for the global BIM and computing engineering communities. A plethora of approaches have been developed that, however, possess several common weaknesses: (1) heavy reliance on laborious manual or semiautomatic segmentation of raw data [e.g., two-dimensional (2D) images or three-dimensional (3D) point clouds]; (2) unsatisfactory results for complex scenes (e.g., furniture or nonstandard indoor settings); and (3) failure to use existing resources for modeling and semantic enrichment. This paper aims to advance a novel, derivative-free optimization (DFO)–based approach that can automatically generate semantically rich as-built BIMs of complex scenes from 3D point clouds. In layman’s terms, the proposed approach recognizes candidate BIM components from 3D point clouds, reassembles the components into a BIM, and registers them with semantic information from credible sources. The approach was prototyped in Autodesk Revit and tested on a noisy point cloud of office furniture scanned via a Google Tango smartphone. The results revealed that the semantically rich as-built BIM was automatically and correctly generated with a root-mean-square error (RMSE) of 3.87 cm in 6.44 s, which outperformed the well-known iterative closest point (ICP) algorithm. The approach was then scaled up to a large auditorium scene consisting of 293 chairs to generate a satisfactory output BIM with a precision of 81.9% and a recall of 80.5%. The semantic registration approach also proved superior to existing segmentation approaches in that it is segmentation-free and capable of processing complex scenes and reusing known information. In addition to these methodological contributions, this approach, properly scaled up, will open new avenues for creation of building/city information models from inexpensive data sources and support profound value-added applications such as smart building or smart city developments.
Fan Xue
added a research item
Symmetry is ubiquitous in architecture, across both time and place. Automated architectural symmetry detection (ASD) from a data source is not only an intriguing inquiry in its own right, but also a step towards creation of semantically rich building and city information models with applications in architectural design , construction management, heritage conservation, and smart city development. While recent advances in sensing technologies provide inexpensive yet high-quality architectural 3D point clouds, existing methods of ASD from these data sources suffer several weaknesses including noise sensitivity, inaccuracy, and high computational loads. This paper aims to develop a novel derivative-free optimization (DFO)-based approach for effective ASD. It does so by firstly transforming ASD into a nonlinear optimization problem involving architectural regularity and topology. An in-house ODAS (Optimization-based Detection of Architectural Symmetries) approach is then developed to solve the formulated problem using a set of state-of-the-art DFO algorithms. Efficiency, accuracy, and robust-ness of ODAS are gauged from the experimental results on nine sets of real-life architectural 3D point clouds, with the computational time for ASD from 1.4 million points only 3.7 seconds and increasing in a sheer logarithmic order against the number of points. The contributions of this paper are threefold. Firstly, formulating ASD as a nonlinear optimization problem constitutes a methodological innovation. Secondly, the provision of up-to-date, open source DFO algorithms allows benchmarking in the future development of free, fast, accurate, and robust approaches for ASD. Thirdly, the ODAS approach can be directly used to develop building and city information models for various value-added applications.
Fan Xue
added a research item
Symmetry is a fundamental phenomenon in not only nature and science but also cities and architectures. Architectural symmetry detection (ASD) from 3D urban point clouds is an essential step in understanding the architectures as well as creating a semantic city/building information model (CIM/BIM) to enable various applications for a smart and resilient future. However, manual segmentation and recognition of 3D urban point clouds are too time-consuming, tedious, and costly, and automatic ASD is very challenging. This paper presents a derivative-free optimization (DFO) approach for automatic ASD from 3D urban point clouds. In this paper, we formulate the problem of ASD as a nonlinear optimization problem by extending the mathematical definition of geometric symmetry with architectural styles. We develop a ‘divide-and-detect’ process to detect the symmetry hierarchy based on the formulation and apply the state-of-the-art DFO algorithms. A pilot study was conducted on a case of the rooftop of a neoclassical building. The proposed approach detected the global reflection from 1.4 million points in 23.5 s, and the whole symmetry hierarchy of reflections in about ten minutes. The detected symmetry hierarchy was applied to a regularity-based rooftop modeling method. The contribution of this paper is twofold. First, this paper exposes the problem of ASD to many mathematical methods through an innovative problem formulation. Secondly, the proposed DFO approach is accurate, efficient, and capable of processing large-scale 3D urban point clouds for semantic CIMs/BIMs.
Fan Xue
added a research item
Over the past decade a considerable number of studies have focused on generating semantically rich as‐built building information models (BIMs). However, the prevailing methods rely on laborious manual segmentation or automatic but error‐prone segmentation. In addition, the methods failed to make good use of existing semantics sources. This article presents a novel segmentation‐free derivative‐free optimization (DFO) approach that translates the generation of as‐built BIMs from 2D images into an optimization problem of fitting BIM components regarding architectural and topological constraints. The semantics of the BIMs are subsequently enriched by linking the fitted components with existing semantics sources. The approach was prototyped in two experiments using an outdoor and an indoor case, respectively. The results showed that in the outdoor case 12 out of 13 BIM components were correctly generated within 1.5 hours, and in the indoor case all target BIM components were correctly generated with a root‐mean‐square deviation (RMSD) of 3.9 cm in about 2.5 hours. The main computational novelties of this study are: (1) to translate the automatic as‐built BIM generation from 2D images as an optimization problem; (2) to develop an effective and segmentation‐free approach that is fundamentally different from prevailing methods; and (3) to exploit online open BIM component information for semantic enrichment, which, to a certain extent, alleviates the dilemma between information inadequacy and information overload in BIM development.
Fan Xue
added a research item
The global construction industry has witnessed the prolific development of radio-frequency identification (RFID), building information modeling (BIM), and most recently, linkage of the two. However, comparatively little attention has been paid to understanding the status quo and development trajectory of such RFID-enabled BIM systems. In view of the proliferation of existing RFID, BIM, and information linkage, practitioners would benefit from guidelines for choosing systems so that their construction engineering and management (CEM) needs can be better met. Accordingly, the study described in this paper has two interconnected research aims: (1) to identify current patterns and development trends in RFID-enabled BIM systems; and (2) to develop guidelines for choosing appropriate solutions for different CEM scenarios. A review of 42 actual cases published in scholarly papers reveals that RFID, used to identify objects and improve real-time information visibility and traceability, is now increasingly linked to BIM as a central information platform. This study provides practitioners with five-step guidelines for linking RFID to BIM for various CEM needs. It also provides researchers with a point of departure for further exploration of approaches to enhancing the value of RFID, BIM, and the integration of one with the other.
Fan Xue
added 3 research items
Many studies have been conducted to create building information models (BIMs) or city information models (CIMs) as the digital infrastructure to support various smart city programs. However, automatic generation of such models for high-density (HD) urban areas remains a challenge owing to (a) complex topographic conditions and noisy data irrelevant to the buildings, and (b) exponentially growing computational complexity when the task is reconstructing hundreds of buildings at an urban scale. This paper develops a method - multi-Source recTification of gEometric Primitives (mSTEP) - for automatic reconstruction of BIMs in HD urban areas. By retrieving building base, height, and footprint geodata from topographic maps, level of detail 1 (LoD1) BIMs representing buildings with flat roof configuration were first constructed. Geometric primitives were then detected from LiDAR point clouds and rectified using architectural knowledge about building geometries (e.g. a rooftop object would normally be in parallel with the outer edge of the roof). Finally, the rectified primitives were used to refine the LoD1 BIMs to LoD2, which show detailed geometric features of roofs and rooftop objects. A total of 1361 buildings located in a four square kilometer area of Hong Kong Island were selected as the subjects for this study. The evaluation results show that mSTEP is an efficient BIM reconstruction method that can significantly improve the level of automation and decrease the computation time. mSTEP is also well applicable to point clouds of various densities. The research is thus of profound significance; other cities and districts around the world can easily adopt mSTEP to reconstruct their own BIMs/CIMs to support their smart city programs.
Fan Xue
added an update
This project was financially supported by Hong Kong Research Grants Council (RGC) General Research Fund (GRF) (No. 17201717). More details are available at:
 
Fan Xue
added a research item
Emerging technologies like massive point cloud from laser scanning and 3D photogrammetry enabled new ways of generating ‘as-built’ building information models (BIM) for existing buildings. It is valuable but also challenging to generate semantic models from point cloud and images in automated ways. In this paper, we present a novel method called Optimization-based Model Generation (OMG) for automated semantic BIM generation. OMG starts from a semantic BIM component dataset and a target measurement such as point cloud, photographs, or floor plans. A fitness function is defined to measure the matching level between an arbitrary BIM model and the target measurement without object recognition. Combinations of digital components are then extensively generated as building models regarding semantic constraints. The fittest model that matches the target measurement best is the result of OMG. The proposed method was demonstrated in reconstructing a 3D model of a demolished building. Advantages of OMG include high-level automation, low requirement on measurement, relationship discovery for components, reusable component libraries, and scalability to new environments.
Fan Xue
added a project goal
There is a dilemma in the generation of semantically rich as-built BIMs: On one hand, it is inherently challenging to segment non-geometric semantics from photos and point clouds; On the other hand, free semantic BIM components are booming on the Internet and elsewhere. This project attempts to solve the dilemma by:
1. To develop a general transformation to model as-built BIM generation as a nonlinear optimization problem of fitting components;
2. To establish a semantic BIM component dataset to integrate open access BIM resources; and
3. To adopt and fine-tune state-of-the-art DFO algorithms for automated generation of semantically rich as-built BIMs.