Project

A derivative-free optimization (DFO) approach to architectural symmetry detection from 3D point clouds

Goal: This proposed research project aims to recognize architectural symmetry from 3D point clouds based on known architectural styles and building technologies. Examples of architectural styles are neoclassical architecture and modernist architecture, which have been reflected in real-life physical architecture and influenced by architecture preferences and construction standards and other factors. Examples of building technologies include popular materials and building codes at the moment of construction.

The key scientific question is “How can the symmetry be efficiently recognized with constraints of architectural styles and knowledge by computer program and then be effectively integrated into as-built model generation methods?”

Date: 1 January 2019 - 31 December 2021

Updates
0 new
5
Recommendations
0 new
0
Followers
0 new
20
Reads
3 new
347

Project log

Fan Xue
added a research item
Coarse registration of 3D point clouds plays an indispensable role for parametric, semantically rich, and realistic digital twin buildings (DTBs) in the practice of GIScience, manufacturing, robotics, architecture, engineering, and construction. However, the existing methods have prominently been challenged by (i) the high cost of data collection for numerous existing buildings and (ii) the computational complexity from self-similar layout patterns. This paper studies the registration of two low-cost data sets, i.e., colorful 3D point clouds captured by smartphones and 2D CAD drawings, for resolving the first challenge. We propose a novel method named `Registration based on Architectural Reflection Detection’ (RegARD) for transforming the self-symmetries in the second challenge from a barrier of coarse registration to a facilitator. First, RegARD detects the innate architectural reflection symmetries to constrain the rotations and reduce degrees of freedom. Then, a nonlinear optimization formulation together with advanced optimization algorithms can overcome the second challenge. As a result, high-quality coarse registration and subsequent low-cost DTBs can be created with semantic components and realistic appearances. Experiments showed that the proposed method outperformed existing methods considerably in both effectiveness and efficiency, i.e., 49.88% less error and 73.13% less time, on average. The RegARD presented in this paper first contributes to coarse registration theories and exploitation of symmetries and textures in 3D point clouds and 2D CAD drawings. For practitioners in the industries, RegARD offers a new automatic solution to utilize ubiquitous smartphone sensors for massive low-cost DTBs.
Fan Xue
added a research item
Although the value of 3D point cloud data (PCD) has been increasingly recognized by the architectural, engineering, construction and facility operations (AECO) sectors, there is much less actual application of PCD in facility management (FM) than other stages. In order to facilitate the exploration of using PCD for FM, this study aims to summarize existing research effort and identify the gaps based on a systematic review of previous studies touching upon the PCD-enabled FM. This review was guided by a conceptual model that consists of four key components associated with PCD application process, including target objects, PCD sensing, model output and applications. 47 papers published in 21 academic journals were collected for the analysis. It was found that Light Detection and Ranging (LiDAR), photogrammetry, and Synthetic Aperture Radar (SAR) were the three mostly used technologies for collecting the PCD. The raw signals, such as fragments of point cloud and photos, collected by these technologies need to be pre-processed for generating the PCD, and segmentation and meshing are two general aspects of PCD post-processing to create models. It was also found that most studies focused on geometric properties, data processing, feature extraction, object recognition, and model generation, seldom would they dig deeper for decision-making support of FM applications. Based on the results, three major gaps of PCD-enabled FM were concluded, including (1) overlooking the valuable non-geometric properties (e.g. specifications of materials, relations between objects); (2) less focusing on providing decision support functions; and (3) hovering at data level rather than information level. Eleven possible research directions including semantics enrichment, real-time model generation, longitudinal analysis, and smart living applications of PCD-enabled FM were thus suggested for future research.
Fan Xue
added a research item
[Outstanding Paper Award] Window view is an intimate medium between occupants and nature, especially in high-density cities like Hong Kong; and thus belongs to the quality of a house or apartment. In literature, researchers found that window views of nature are vital to the occupants’ physical and psychological health and productivity improvement. Understanding the view situation at the urban level can facilitate urban environment optimization, urban planning and development policies, and smart city management. Currently, views of nature have been quantitatively studied in satellite images and cars’ cameras at a macro or micro level, respectively. However, as an essential supplement to the greenery view information hub at a mesoscale, few studies on efficient visualization and classification of window views at the urban level seem available. This paper presents an automatic approach that captures and classifies photo-realistic views at the windows in a 3D photogrammetric city model. First, by triangulating the window geometries from geo-matched 3D photogrammetric and 2D digital maps, the rich window semantics are registered to the 3D models. Then, the similar window views are visualized in batch with an appropriate focal length and field of view. Finally, the view at each window is analyzed and classified through transfer learning automatically. We applied the proposed approach to the 3D model of Hong Kong Island and found satisfactory results for identifying nature scenes or urban scenes. Once massively adopted, the presented approach can offer novel geographic indicators for billions of urban inhabitants and the Architecture, Engineering, Construction, and Operation (AECO) industry.
Fan Xue
added a research item
[ISPRS JPRS' 2020 Featured Articles] Recent advancement of remote sensing technologies has brought in accurate, dense, and inexpensive city-scale Light Detection And Ranging (LiDAR) point clouds, which can be utilized to model city objects (e.g., buildings, roads, and automobiles) for creating Digital Twin Cities (DTCs). However, processing such unstructured point clouds is very challenging, epitomized by high cost, movable objects, limited object classes, and high information inadequacy/redundancy. We noticed that many city objects are not in random shapes; rather, they have invariant cross-sections following the Gestalt design principles, including proximity, connectivity, symmetry, and similarity. In this paper, we present a novel unsupervised method, called Clustering Of Symmetric Cross-sections of Objects (COSCO), to process urban LiDAR point clouds to a hierarchy of objects based on their characteristic cross-sections. First, city objects are segmented as connected patches of proximate 3D points. Then, symmetric cross-sections are detected for symmetric city objects. Finally, the taxonomy and groups of city objects are recognized from a hierarchical clustering analysis of the dissimilarity matrix. Experimental results showed that COSCO detected the correct taxonomy and types of 12 cars from 24,126 LiDAR points in 8.28s. Based on the cross-sections and taxonomy, a digital twin was created by registering online free 3D car models in 29.58s. The contribution of this paper is twofold. First, it presents an effective unsupervised method for understanding and developing DTC objects in LiDAR point clouds by harnessing innate Gestalt design principles. Secondly, COSCO can be an efficient LiDAR pre-processing tool for recognizing symmetric city objects' cross-sections, positions, heading directions, dimensions, and possible types for smart city applications in GIScience, Architecture, Engineering, Construction and Operation (AECO), and autonomous vehicles.
Fan Xue
added a research item
Despite the extensive use of positioning and navigation in outdoor space, indoor positioning and navigation systems, essential for intelligent building and smart city services, are unsatisfactory in either performance or price, sometimes in both. This paper analyzes and compares the performances and prices of existing indoor positioning technologies that are categorized into a few classes according to their spatial sensing and referencing methods. Based on the previous work in walkability, this paper proposes a novel Walkability network-based Augmented Reality (WaNAR) method using smartphones with AR positioning function for positioning and navigation. In WaNAR, drifting of the AR positioning signals are corrected continuously by the ground-truth 3D indoor/outdoor walkability network (e.g., nobody is supposed to walk through a wall) in a 3D model. The error at the vertical axis of the walking direction is corrected continuously and that of the walking direction is compensated at every turn. WaNAR can be used in both indoor and outdoor navigation, its performance and price are proved to be largely improved compared to existing technologies. The only investment for a typical building is a 3D drawing of indoor walkable space in a few staff-hours. WaNAR has broad application prospects at various positioning and navigation scenarios.
Fan Xue
added a research item
In the era of smart city, semantically rich city information models (CIMs) are demanded as a critical information hub. Roof albedo, a semantic property measures how much solar radiation is reflected, is vital to various urban sustainability topics, including heat island, local climate, green roof, and urban morphology. This paper presents an approach that enriches LiDAR-based albedo to rooftop models for CIM. First, we apply Chen et al. (2018)’s method to the reconstruction of the geometries of rooftop elements. Then, albedos of roofs and rooftop elements are estimated from the mean reflectance in LiDAR data. A pilot study was conducted in an urban area in Hong Kong. The results showed that the building models created by the presented approach were satisfactory in terms of rooftop elements and roof albedos. The results from the present approach can provide sustainability study the details of 3D geometries and albedos in an urban area.
Fan Xue
added a research item
Reconstructing semantically rich building information model (BIM) from 2D images or 3D point clouds represents a research realm that is gaining increasing popularity in architecture, engineering, and construction. Researchers have found that architectural design knowledge, such as symmetry, planarity, parallelism, and orthogonality, can be utilized to improve the effectiveness of such BIM reconstruction. Following this line of enquiry, this paper aims to develop a novel semantic registration approach for complicated scenes with repetitive, irregular-shaped objects. The approach first formulates the architectural repetition as the multimodality in mathematics. Thus, the reconstruction of repetitive objects becomes a multimodal optimization (MMO) problem of registering BIM components which have accurate geometries and rich semantics. Then, the topological information about repetition and symmetry in the reconstructed BIM is recognized and regularized for BIM semantic enrichment. A university lecture hall case, consisting of 1.9 million noisy points of 293 chairs, was selected for an experiment to validate the proposed approach. Experimental results showed that a BIM was satisfactorily created (achieving about 90% precision and recall) automatically in 926.6 s; and an even more satisfactory BIM achieved 99.3% precision and 98.0% recall with detected semantic and topological information under the minimal effort of human intervention in 228.4 s. The multimodality model of repetitive objects, the repetition detection and regularization for BIM, and satisfactory reconstruction results in the presented approach can contribute to methodologies and practices in multiple disciplines related to BIM and smart city.
Fan Xue
added a research item
Digital twin city (DTC) is a critical information infrastructure that enables many innovative applications for smart and resilient city development. Thanks to the recent advances in remote sensing and photogrammetry, accurate, dense, and large-scale 3D urban point clouds become increasingly available for many cities for creating and updating their DTCs. Because of the immense amount and the high update frequency of urban point clouds, it is too time-consuming and labor-intensive to create and update DTCs solely by human experts. Researchers have developed a wealth of automatic and semi-automatic methods for processing 3D urban point clouds using expert knowledge of the built environment, supervised learning, and reinforced learning of geometric primitives and components. However, these methods are restricted, ironically by the embedded knowledge, in the scalability to sophisticated scenes and the availability of standardized components. Inspired by the success of Google' unsupervised learning program AlphaZero, this paper proposes a novel hierarchical clustering approach for semantic enrichment of point clouds. Unlike the existing approaches relying on fixed domain knowledge, extra correlational training examples, or available 3D references, the proposed approach exploits the similarities between patches of point clouds without explicit domain knowledge. The proposed approach first segments patches from the input point cloud through the connected subgraphs of voxel grids, then computes the dissimilarity matrix between the patches via iterative optimization. Subsequently, the dissimilarity engenders a hierarchy of clusters for understanding the relatedness between the patches. A pilot study on a real urban scene showed that the proposed approach is feasible and potent to cluster and detect objects automatically. Another experiment showed that the dissimilarity-based clusters and associated transformations can help create semantic objects for DTC, as referential 3D models are available.
Fan Xue
added an update
This project was financially supported by Hong Kong Research Grants Council (RGC) General Research Fund (GRF) (No. 17200218). More details are available at:
 
Fan Xue
added a research item
Symmetry is ubiquitous in architecture, across both time and place. Automated architectural symmetry detection (ASD) from a data source is not only an intriguing inquiry in its own right, but also a step towards creation of semantically rich building and city information models with applications in architectural design , construction management, heritage conservation, and smart city development. While recent advances in sensing technologies provide inexpensive yet high-quality architectural 3D point clouds, existing methods of ASD from these data sources suffer several weaknesses including noise sensitivity, inaccuracy, and high computational loads. This paper aims to develop a novel derivative-free optimization (DFO)-based approach for effective ASD. It does so by firstly transforming ASD into a nonlinear optimization problem involving architectural regularity and topology. An in-house ODAS (Optimization-based Detection of Architectural Symmetries) approach is then developed to solve the formulated problem using a set of state-of-the-art DFO algorithms. Efficiency, accuracy, and robust-ness of ODAS are gauged from the experimental results on nine sets of real-life architectural 3D point clouds, with the computational time for ASD from 1.4 million points only 3.7 seconds and increasing in a sheer logarithmic order against the number of points. The contributions of this paper are threefold. Firstly, formulating ASD as a nonlinear optimization problem constitutes a methodological innovation. Secondly, the provision of up-to-date, open source DFO algorithms allows benchmarking in the future development of free, fast, accurate, and robust approaches for ASD. Thirdly, the ODAS approach can be directly used to develop building and city information models for various value-added applications.
Fan Xue
added a research item
Symmetry is a fundamental phenomenon in not only nature and science but also cities and architectures. Architectural symmetry detection (ASD) from 3D urban point clouds is an essential step in understanding the architectures as well as creating a semantic city/building information model (CIM/BIM) to enable various applications for a smart and resilient future. However, manual segmentation and recognition of 3D urban point clouds are too time-consuming, tedious, and costly, and automatic ASD is very challenging. This paper presents a derivative-free optimization (DFO) approach for automatic ASD from 3D urban point clouds. In this paper, we formulate the problem of ASD as a nonlinear optimization problem by extending the mathematical definition of geometric symmetry with architectural styles. We develop a ‘divide-and-detect’ process to detect the symmetry hierarchy based on the formulation and apply the state-of-the-art DFO algorithms. A pilot study was conducted on a case of the rooftop of a neoclassical building. The proposed approach detected the global reflection from 1.4 million points in 23.5 s, and the whole symmetry hierarchy of reflections in about ten minutes. The detected symmetry hierarchy was applied to a regularity-based rooftop modeling method. The contribution of this paper is twofold. First, this paper exposes the problem of ASD to many mathematical methods through an innovative problem formulation. Secondly, the proposed DFO approach is accurate, efficient, and capable of processing large-scale 3D urban point clouds for semantic CIMs/BIMs.
Fan Xue
added a research item
Over the past decade a considerable number of studies have focused on generating semantically rich as‐built building information models (BIMs). However, the prevailing methods rely on laborious manual segmentation or automatic but error‐prone segmentation. In addition, the methods failed to make good use of existing semantics sources. This article presents a novel segmentation‐free derivative‐free optimization (DFO) approach that translates the generation of as‐built BIMs from 2D images into an optimization problem of fitting BIM components regarding architectural and topological constraints. The semantics of the BIMs are subsequently enriched by linking the fitted components with existing semantics sources. The approach was prototyped in two experiments using an outdoor and an indoor case, respectively. The results showed that in the outdoor case 12 out of 13 BIM components were correctly generated within 1.5 hours, and in the indoor case all target BIM components were correctly generated with a root‐mean‐square deviation (RMSD) of 3.9 cm in about 2.5 hours. The main computational novelties of this study are: (1) to translate the automatic as‐built BIM generation from 2D images as an optimization problem; (2) to develop an effective and segmentation‐free approach that is fundamentally different from prevailing methods; and (3) to exploit online open BIM component information for semantic enrichment, which, to a certain extent, alleviates the dilemma between information inadequacy and information overload in BIM development.
Fan Xue
added a research item
Many studies have been conducted to create building information models (BIMs) or city information models (CIMs) as the digital infrastructure to support various smart city programs. However, automatic generation of such models for high-density (HD) urban areas remains a challenge owing to (a) complex topographic conditions and noisy data irrelevant to the buildings, and (b) exponentially growing computational complexity when the task is reconstructing hundreds of buildings at an urban scale. This paper develops a method - multi-Source recTification of gEometric Primitives (mSTEP) - for automatic reconstruction of BIMs in HD urban areas. By retrieving building base, height, and footprint geodata from topographic maps, level of detail 1 (LoD1) BIMs representing buildings with flat roof configuration were first constructed. Geometric primitives were then detected from LiDAR point clouds and rectified using architectural knowledge about building geometries (e.g. a rooftop object would normally be in parallel with the outer edge of the roof). Finally, the rectified primitives were used to refine the LoD1 BIMs to LoD2, which show detailed geometric features of roofs and rooftop objects. A total of 1361 buildings located in a four square kilometer area of Hong Kong Island were selected as the subjects for this study. The evaluation results show that mSTEP is an efficient BIM reconstruction method that can significantly improve the level of automation and decrease the computation time. mSTEP is also well applicable to point clouds of various densities. The research is thus of profound significance; other cities and districts around the world can easily adopt mSTEP to reconstruct their own BIMs/CIMs to support their smart city programs.
Fan Xue
added an update
A paper titled Automatic building information model reconstruction in high-density urban areas: Augmenting multi-source data with architectural knowledge is published in Automation in Construction. Interested readers can obtain a free copy in 50 days via the following link:
This is an extended study based on a conference paper
We express our gratitude to
 
Fan Xue
added an update
A conference paper on this topic was accepted by CIB W78 2018 conference (http://mypages.iit.edu/~cibw78/index.html). The paper talks about the theoretical stance and preliminary experiments on the automated architectural symmetry recognition.
For you guys who are interested, we can gather at Chicago this October to have a close discussion.
 
Private Profile
added 2 research items
Fan Xue
added a research item
There has been a world-wide interest in using 3D building models to support urban planning, design, and management. For high-density urban areas with complex topographic conditions, how to fast develop such models with adequate accuracy remains a challenge, particularly when a single data source is utilized. This study seeks to develop such quality models by integrating multi-source data including the topographic map and LiDAR data. A total of 1,361 high-rise buildings located in Hong Kong are chosen as the subjects. The topographic map is used to develop a level of detail 1 (LoD1) model of each building. The LiDAR data is then used to fine-tune building shapes, in particular roofs, so that more accurate models with LoD2 can be derived. It is found that the integration of topographic map and LiDAR data can improve the completeness and accuracy of building models. The speed and computations to operate the integration are acceptable. Future studies are recommended to achieve LoD3 building models by utilizing other data sources (e.g., semantic information of individual buildings, and points of interest).
Fan Xue
added an update
The first attempt of this project is fusing symmetry information from multi-source data, in a city-scale reconstruction of LoD2 (regarding CityGML) models from airborne LiDAR data. Enclosed figures explain our preliminary trials in modeling our HKU main campus.
 
Fan Xue
added a project goal
This proposed research project aims to recognize architectural symmetry from 3D point clouds based on known architectural styles and building technologies. Examples of architectural styles are neoclassical architecture and modernist architecture, which have been reflected in real-life physical architecture and influenced by architecture preferences and construction standards and other factors. Examples of building technologies include popular materials and building codes at the moment of construction.
The key scientific question is “How can the symmetry be efficiently recognized with constraints of architectural styles and knowledge by computer program and then be effectively integrated into as-built model generation methods?”