Chenbo Zhao’s research while affiliated with The University of Tokyo and other places

Deep learning-based building extraction methods have widespread applications in diverse fields. However, the evaluation of large-scale extraction results remains challenging, due to traditional evaluation metrics rely on manually created ground-truth samples and the lack of comprehensive reference-building data for developing countries. To address these problems, we proposed a two-level framework for evaluating large-scale footprint extraction. First, we utilised global open-source population and land use data as the proxy data, to assess grid-level completeness for the areas with insufficient reference data. Second, we introduced an improved two-way area-overlapping method to match the extracted footprints with the reference buildings, thereby enabling a comprehensive evaluation of the study region. Tested in Hyogo Prefecture and Numazu City, Japan, the results demonstrated a 2.6-% improvement in grid classification accuracy and an increase of 0.53 in the completeness correlation, compared with the results obtained using a single proxy indicator. Moreover, the optimised matching method achieved an outstanding semantic matching accuracy of 99%, with high efficiency and robustness in multi-scale matching. Therefore, the proposed approach can effectively evaluate large-scale footprint extraction results and interpret their semantic relationship with actual buildings, applicable globally regardless of the availability of reference building datasets. ARTICLE HISTORY

Developing a model to evaluate urban streetscapes based on subjective perceptions is important for quantitative understanding. However, previous studies have only considered limited types of subjective perceptions, neglecting the relationships between them. Further, accurately measuring subjective perception with low computational costs for large-scale urban regions at high spatial resolutions has been difficult. We present a deep-learning-based multilabel classification model that can measure 22 subjective perceptions scores from street-view images. This model uses the results of a web questionnaire survey encompassing 22 subjective perceptions, with 8.8 million responses. Our model demonstrates high accuracy (0.80–0.91) in measuring subjective perception scores from street-view images and achieves low computational cost by training on 22 subjective perception relationships. The 22 subjective perceptions were analyzed using PCA and k-means analysis. By categorizing the 22 subjective perceptions into a two-dimensional space visualized and grouped into distinct groups—positive, negative, calm, and lively—we unearthed vital insights into the intricate nuances of human perception. In addition, the study used semantic segmentation to extract landscape elements from street-view images and applied ℓ1-regularized sparse modeling to identify the landscape elements structurally correlating with each subjective perception class. The analysis revealed that only seven out of nineteen landscape elements significantly correlated with subjective impressions, and these effects varied by class. Notably, sky coverage positively influences positive subjective perceptions, such as attractiveness and calmness, but negatively affects lively impressions. The proposed model can be used to map the overall image of a city and identify landscape design issues in community development design.

Remote sensing image semantic segmentation of land use, benefitted from the development of deep learning and consequently made considerable progress in terms of inferencing accuracy and speed. However, the effective training of semantic segmentation models for remote sensing imagery necessitates extensively detailed pixel-level annotations, and gathering such data is both time-intensive and laborious. Thus, this study implemented low-rank adaptation on a stable diffusion algorithm to learn the distribution of the pixel-level annotations in case of the LoveDA dataset. Consequently, the annotation-image pairs were used to train the remote sensing image generator based on stable diffusion guided by ControlNet. We proposed a stable diffusion based approach, which can generate image-annotation pairs from scratch. The generated annotation and image pairs achieved a high accuracy of 0.520 mean intersection-over-union on LoveDA dataset, which is close to the original data training result of 0.539 mIoU. Furthermore, the mixed training using generated and original data achieved 0.542 mIoU, thereby demonstrating the data augmentation function of our approach. This study provided a solution for the high-cost pixel-level annotation issue, and thus, exhibited the potential of artificial intelligence generated content.

Land price is an important economic factor in producing meaningful references for regional planners by assisting them in urban planning, economic decision-making, and land resource allocation. However, related studies in land price analysis were mainly focused on the factors of site area and plot ratio, analysis of the potential impact of streetscape factors and human subjective perception on the land price has been lacking, regardless of the impact on supply and demand relationship. Therefore, this study developed a new approach for estimating and analyzing land prices through deep learning that considered the streetscape and human subjective perception factors. In the estimation part, we developed a fine-grained end-to-end deep learning model, the input is street view images, and the output is land prices. In the analysis part, we extracted the semantic segmentation results and human subjective perception scores from the images and combined them with the results of land price estimation. We then introduced a combination of quantitative analysis using the gradient-weighted class activation mapping and L1-based sparse linear regression to model the relationship between the streetscape and the human subjective perception quantitatively. The gradient-weighted class activation mapping was used to determine which categories of pixels deep learning relied on to output the results of land price estimation quantitatively. Combined with segmentation results, we implemented L1-based sparse linear regression and quantitatively determined the importance of the streetscape factors for land prices. Overall, our deep learning model achieved 77.99 % accuracy by only using street views in estimating the land price, and we illustrated the impacts of the streetscape and perception on the land price by showing that perception scores are more important than streetscape factors such as road and mountain in the streetscape. Comfortable, lived-in feel in perception have the most important impact on land price estimation.

People flow trend estimation is crucial to traffic and urban safety planning and management. However, owing to privacy concerns, the collection of individual location data for people flow statistical analysis is difficult; thus, an alternative approach is urgently needed. Furthermore, the trend in people flow is reflected in streetscape factors, yet the relationship between them remains unclear in the existing literature. To address this, we propose an end-to-end deep-learning approach that combines street view images and human subjective score of each street view. For a more detailed people flow study, estimation and analysis were implemented using different time and movement patterns. Consequently, we achieved a 78% accuracy on the test set. We also implemented the gradient-weighted class activation mapping deep learning visualization and L1 based statistical methods and proposed a quantitative analysis approach to understand the land scape elements and subjective feeling of street view and to identify the effective elements for the people flow estimation based on a gradient impact method. In summary, this study provides a novel end-to-end people flow trend estimation approach and sheds light on the relationship between streetscape, human subjective feeling, and people flow trend, thereby making an important contribution to the evaluation of existing urban development.

The built year and structure of individual buildings are crucial factors for estimating and assessing potential earthquake and tsunami damage. Recent advances in sensing and analysis technologies allow the acquisition of high-resolution street view images (SVIs) that present new possibilities for research and development. In this study, we developed a model to estimate the built year and structure of a building using omnidirectional SVIs captured using an onboard camera. We used geographic information system (GIS) building data and SVIs to generate an annotated built year and structure dataset by developing a method to automatically combine the GIS data with images of individual buildings cropped through object detection. Furthermore, we trained a deep learning model to classify the built year and structure of buildings using the annotated image dataset based on a deep convolutional neural network (DCNN) and a vision transformer (ViT). The results showed that SVI accurately predicts the built year and structure of individual buildings using ViT (overall accuracies for structure = 0.94 [3 classes] and 0.96 [2 classes] and for age = 0.68 [6 classes] and 0.90 [3 classes]). Compared with DCNN-based networks, the proposed Swin transformer based on ViT architectures effectively improves prediction accuracy. The results indicate that multiple high-resolution images can be obtained for individual buildings using SVI, and the proposed method is an effective approach for classifying structures and determining building age. The automatic, accurate, and large-scale mapping of the built year and structure of individual buildings can help develop specific disaster prevention measures.

The detection of road facilities or roadside structures is essential for high-definition (HD) maps and intelligent transportation systems (ITSs). With the rapid development of deep-learning algorithms in recent years, deep-learning-based object detection techniques have provided more accurate and efficient performance, and have become an essential tool for HD map reconstruction and advanced driver-assistance systems (ADASs). Therefore, the performance evaluation and comparison of the latest deep-learning algorithms in this field is indispensable. However, most existing works in this area limit their focus to the detection of individual targets, such as vehicles or pedestrians and traffic signs, from driving view images. In this study, we present a systematic comparison of three recent algorithms for large-scale multi-class road facility detection, namely Mask R-CNN, YOLOx, and YOLOv7, on the Mapillary dataset. The experimental results are evaluated according to the recall, precision, mean F1-score and computational consumption. YOLOv7 outperforms the other two networks in road facility detection, with a precision and recall of 87.57% and 72.60%, respectively. Furthermore, we test the model performance on our custom dataset obtained from the Japanese road environment. The results demonstrate that models trained on the Mapillary dataset exhibit sufficient generalization ability. The comparison presented in this study aids in understanding the strengths and limitations of the latest networks in multiclass object detection on large-scale street-level datasets.