Article

# Deep learning-based road damage detection and classification for multiple countries

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

## No full-text available

... The background information for CRDDC is provided as follows. A lack of financial resources makes many local governments unable to conduct sufficient road condition inspections on time [6]. Some municipalities automate road damage detection by using high-performance sensors. ...
... The development of a method that applies to more than one country leads to the possibility of designing a stand-alone system for road damage detection worldwide. Considering the requirement, Arya et al. [6] augmented the Japanese dataset with road damage images from India and the Czech Republic. The authors proposed the data, Road Damage Dataset-2020 (RDD2020 [16], [17]), which comprises 26620 images, almost thrice the volume of the 2018 dataset [18] utilized for RDDC 2018. ...
... The authors proposed the data, Road Damage Dataset-2020 (RDD2020 [16], [17]), which comprises 26620 images, almost thrice the volume of the 2018 dataset [18] utilized for RDDC 2018. Further, the authors [6] conducted a comprehensive analysis of models trained using different combinations of the data. ...
Preprint
Full-text available
This paper summarizes the Crowdsensing-based Road Damage Detection Challenge (CRDDC), a Big Data Cup organized as a part of the IEEE International Conference on Big Data'2022. The Big Data Cup challenges involve a released dataset and a well-defined problem with clear evaluation metrics. The challenges run on a data competition platform that maintains a real-time online evaluation system for the participants. In the presented case, the data constitute 47,420 road images collected from India, Japan, the Czech Republic, Norway, the United States, and China to propose methods for automatically detecting road damages in these countries. More than 60 teams from 19 countries registered for this competition. The submitted solutions were evaluated using five leaderboards based on performance for unseen test images from the aforementioned six countries. This paper encapsulates the top 11 solutions proposed by these teams. The best-performing model utilizes ensemble learning based on YOLO and Faster-RCNN series models to yield an F1 score of 76% for test data combined from all 6 countries. The paper concludes with a comparison of current and past challenges and provides direction for the future.
... From 2018 to 2020, the Road Damage Detection Challenge (RDDC) develops a new horizon of cost-effective road condition monitoring and damage detection. This challenge receives wide attention from researchers all over the world, and several novel methods along the pipeline of 2D object detection [12], [19] are proposed for improving automatic pavement distress identification performance [1], [20], [21], [22], [23]. In this paper, we would also investigate efficient yet effective algorithm for low-cost road damage detection. ...
... Previous works motivate several pavement distress datasets, i.e., CrackTree200 [25] and GAPs [26], each of which contains hundreds of top-view images with fine-grained pixel annotations. The road damage dataset (RDD2020) [1] is publicly available with 26620 front-view samples collected from different countries (Japan, India and Czech) using smartphone, with bounding-box annotation for various distress types. Similarly, the damage dataset recently released by Jiangsu Pro., ...
... Extensive cross-domain damage detection experiments are conducted on RDD2020 and CN2021, and results demonstrate the generality and adaptability of DA-RDD, which outperform the baseline and other counterparts over a substantial margin. Additionally, we build up a large-scale road damage dataset (based on Road Damage Dataset 2020 (RDD2020) [1]) namely RDD2021, comprising 100k front-view synthetic images with boundingbox annotations on four damage categories across three different countries. Empirical studies verify its effectiveness and advancement, with substantial performance gains on crack identification. ...
Preprint
Full-text available
Recent advances on road damage detection relies on a large amount of labeled data, whilst collecting pavement image is labor-intensive and time-consuming. Unsupervised Domain Adaptation (UDA) provides a promising solution to adapt a source domain to the target domain, however, cross-domain crack detection is still an open problem. In this paper, we propose domain adaptive road damage detection termed as DA-RDD, by incorporating image-level with instance-level feature alignment for domain-invariant representation learning in an adversarial manner. Specifically, importance weighting is introduced to evaluate the intermediate samples for image-level alignment between domains, and we aggregate RoI-wise feature with multi-scale contextual information to recover the crack details for progressive domain alignment at instance level. Additionally, a large-scale road damage dataset (based on Road Damage Dataset 2020 (RDD2020)) named as RDD2021 is constructed with $100k$ synthetic labeled distress images. Extensive experimental results on damage detection across different countries demonstrate the universality and superiority of DA-RDD, and empirical studies on RDD2021 further claim its effectiveness and advancement. To our best knowledge, it is the first time to investigate domain adaptative pavement crack detection, and we expect the contributions in this work would facilitate the development of generalized road damage detection in the future.
... Their model resulted in low recall of this category which was attributed to less number of training data. The dataset RDD2018 was extended to include images from multiple countries in RDD2020 [23]. However, the extended dataset does not include road rutting. ...
... The road damage datasets made available by the works of [6], [18], [23], [25], [26], etc. mainly contains instances of cracks, potholes, and their variants. In contrast, the road damage dataset proposed in [27] contains 263 images of road rutting. ...
... Further, the technique of image augmentation using albumentations [45] was explored with the objective to improve the detection accuracy of YOLOv4 model by increasing the size of the training dataset. Horizontal flip, random brightness contrast, Gaussian noise, RGB shift, sharpen, etc. were used in accordance with techniques mentioned in [23]. However, vertical flip was not used as images with vertically flipped road will not appear in the original dataset while taking videos or images using dashboard cameras on vehicles. ...
Preprint
Full-text available
Road rutting is a severe road distress that can cause premature failure of road incurring early and costly maintenance costs. Research on road damage detection using image processing techniques and deep learning are being actively conducted in the past few years. However, these researches are mostly focused on detection of cracks, potholes, and their variants. Very few research has been done on the detection of road rutting. This paper proposes a novel road rutting dataset comprising of 949 images and provides both object level and pixel level annotations. Object detection models and semantic segmentation models were deployed to detect road rutting on the proposed dataset, and quantitative and qualitative analysis of model predictions were done to evaluate model performance and identify challenges faced in the detection of road rutting using the proposed method. Object detection model YOLOX-s achieves mAP@IoU=0.5 of 61.6% and semantic segmentation model PSPNet (Resnet-50) achieves IoU of 54.69 and accuracy of 72.67, thus providing a benchmark accuracy for similar work in future. The proposed road rutting dataset and the results of our research study will help accelerate the research on detection of road rutting using deep learning.
... Road damage detection (RDD) is an important task in the field of traffic infrastructure and involves locating and classifying road damage [1][2][3][4]. It can identify roads that need maintenance to reduce potential safety hazards. ...
... Most of the existing methods use convolutional networks to learn classification features and have achieved significant performance improvement on public foreign datasets [3,6]. As shown in Figure 1, multiple types of damage can appear in the same image. ...
... This fine-tuning can enhance the network's perception of different damage and thus reduce the impact of some types of damage with similar appearance on classification. We evaluate the proposed baseline on the popular RDD2020 dataset [3] and the self-collected CNRDD dataset. It was found that the CNRDD dataset is more challenging than the RDD2020 dataset, and cracks are easier to distinguish than other damage. ...
Article
Full-text available
Automated detection of road damage (ADRD) is a challenging topic in road maintenance. It focuses on automatically detecting road damage and assessing severity by deep learning. Because of the sparse distribution of characteristic pixels, it is more challenging than object detection. Although some public datasets provide a database for the development of ADRD, their amounts of data and the standard of classification cannot meet network training and feature learning. With the aim of solving this problem, this work publishes a new road damage dataset named CNRDD, which is labeled according to the latest evaluation standard for highway technical conditions in China (JTG5210-2018). The dataset is collected by professional onboard cameras and is manually labeled in eight categories with three different degrees (mild, moderate and severe), which can effectively help promote research of automated detection of road damage. At the same time, a novel baseline with attention fusion and normalization is proposed to evaluate and analyze the published dataset. It explicitly leverages edge detection cues to guide attention for salient regions and suppresses the weights of non-salient features by attention normalization, which can alleviate the interference of sparse pixel distribution on damage detection. Experimental results demonstrate that the proposed baseline significantly outperforms most existing methods on the existing RDD2020 dataset and the newly released CNRDD dataset. Further, the CNRDD dataset is proved more robust, as its high damage density and professional classification are more conducive to promote the development of ADRD.
... Despite road networks' critical functions as catalysts for economic development, many governments still rely on either relatively inefficient and inaccurate human visual inspections or relatively expensive and difficult to scale laser-and high definition camera-based systems to carry out surface-level quality evaluations of asphalt roadways in order to identify potentially hazardous pavement distresses such as potholes and cracks that may cause road accidents and endanger motorists [4]. For instance as the majority of U.S. Department of Transportation (DOT) state agencies employ government workers or third party contractors to provide annual or biennial estimates of state highway pavement deterioration measuring the prevalence and severity of cracking, patching, faulting and joint deterioration per roadmile for different sections of state highways in order to meet federal reporting requirements mandated under the 2012 Moving Ahead for Progress in the 21st Century Act (MAP-21), inspectors will typically complete these estimates through on-the-ground or windshield visual surveys of road distresses in a way that therefore subjects these calculations to a large element of human error [4]- [6]. ...
... Despite road networks' critical functions as catalysts for economic development, many governments still rely on either relatively inefficient and inaccurate human visual inspections or relatively expensive and difficult to scale laser-and high definition camera-based systems to carry out surface-level quality evaluations of asphalt roadways in order to identify potentially hazardous pavement distresses such as potholes and cracks that may cause road accidents and endanger motorists [4]. For instance as the majority of U.S. Department of Transportation (DOT) state agencies employ government workers or third party contractors to provide annual or biennial estimates of state highway pavement deterioration measuring the prevalence and severity of cracking, patching, faulting and joint deterioration per roadmile for different sections of state highways in order to meet federal reporting requirements mandated under the 2012 Moving Ahead for Progress in the 21st Century Act (MAP-21), inspectors will typically complete these estimates through on-the-ground or windshield visual surveys of road distresses in a way that therefore subjects these calculations to a large element of human error [4]- [6]. Other methods, such as driving specialized vehicles equipped with various sensors such as laser scanners [7], ground penetrating radar ("GPR") antennas [8] and high definition cameras [6] [9] such as shown in Figure 1 along sections of roadway in order to collect high-definition images and 3D reconstructions of the road pavement, are similarly impractical given the significant capital and labor costs required in operating these technologies [4]. ...
... For instance as the majority of U.S. Department of Transportation (DOT) state agencies employ government workers or third party contractors to provide annual or biennial estimates of state highway pavement deterioration measuring the prevalence and severity of cracking, patching, faulting and joint deterioration per roadmile for different sections of state highways in order to meet federal reporting requirements mandated under the 2012 Moving Ahead for Progress in the 21st Century Act (MAP-21), inspectors will typically complete these estimates through on-the-ground or windshield visual surveys of road distresses in a way that therefore subjects these calculations to a large element of human error [4]- [6]. Other methods, such as driving specialized vehicles equipped with various sensors such as laser scanners [7], ground penetrating radar ("GPR") antennas [8] and high definition cameras [6] [9] such as shown in Figure 1 along sections of roadway in order to collect high-definition images and 3D reconstructions of the road pavement, are similarly impractical given the significant capital and labor costs required in operating these technologies [4]. Given these limitations of existing methods, the relatively more cost-effective alternative of detecting and cataloging road distresses using computer vision algorithms trained on low-cost smartphone-captured pavement images has recently emerged as a subject of interest in academia as public releases of various annotated image datasets of road pavement distresses such as the 2020 GRDC dataset have encouraged further research on the subject. ...
Preprint
Full-text available
... The problem is that such methods can only obtain low-level image information and cannot extract highlevel semantic information, which has a significant impact on the robustness of the algorithms. With the increase of parallel computing power and the development of deep learning, data-driven methods are widely used in PMS in various countries [11,12]. Convolutional neural network (CNN) [13] is one of the well-known data-driven methods that can automatically learn high-level information from large amounts of data through a multilayered artificial neural network (ANN), which can be used to classify images or detect objects. ...
... One question is how much data we need at a minimum to train a model to provide largely satisfactory results. For image classification tasks, Arya et al. [12] argued that at least 5000 labeled images per category are needed. For object detection, Maeda et al. [17] suggested that at least 1000 images per category are needed, while according to Shahinfar et al. [58], the rate of improvement in model performance starts to level off when the number of images is greater than 150-500. ...
Article
Full-text available
Thanks to the development of deep learning, the use of data-driven methods to detect pavement distresses has become an active research field. This research makes four contributions to address the problem of efficiently detecting cracks and sealed cracks in asphalt pavements. First, a dataset of pavement cracks and sealed cracks is created, which consists of 10,400 images obtained by a vehicle equipped with a highway condition monitor, with 202,840 labeled distress instances included in these pavement images. Second, we develop a dense and redundant crack annotation method based on the characteristics of the crack images. Compared with traditional annotation, the method we propose generates more object instances, and the localization is more accurate. Next, to achieve efficient crack detection, a semi-automatic crack annotation method is proposed, which reduces the working time by 80% compared with fully manual annotation. Finally, comparative experiments are conducted on our dataset using 13 currently prevailing object detection algorithms. The results show that dense and redundant annotation is effective; moreover, cracks and sealed cracks can be efficiently and accurately detected using the YOLOv5 series model and YOLOv5s is the most balanced model with an F1-score of 86.79% and an inference time of 14.8ms. The pavement crack and sealed crack dataset created in this study is publicly available.
... Deeksha Arya et al. [4] introduced a deep learning based approach for road damage detection and classification. This framework was utilized for detecting road damages in multiple countries. ...
... Figure 1 represents the proposed block diagram of the present work. [4] The deep learning network for the road damage detection ...
Article
Full-text available
... Later on, a more thorough classification was implemented, in which the damages are classified into eight different categories [13], as shown in Table 1. Afterward, a road damage detection for multiple countries was developed [4], which expanded the dataset used in [13], which contained only images from Japan, with images from India and the Czech Republic. The proposed method categorized the damages into four main types: longitudinal/parallel cracks, transverse/perpendicular cracks, alligator/complex cracks, and potholes. ...
... The proposed method categorized the damages into four main types: longitudinal/parallel cracks, transverse/perpendicular cracks, alligator/complex cracks, and potholes. The proposed dataset in [4] was made publicly available [5] and formed the basis for the organization of the Global Road Damage Detection Challenge (GRDDC), which led to many innovative solutions that are summarized in [3]. In this work, an image-based system for road infrastructure monitoring, based on YOLOv5 detection model is presented, which classifies the defects into three main damage types. ...
Chapter
Road infrastructure is positively associated with a country’s socio-economic growth and therefore road maintenance is of great importance for every country. One of the critical maintenance steps is road damage detection, which typically requires large amounts of time and high costs. In this work, the YOLOv5 two-stage detector is leveraged, in order to create an image-based solution for road defect detection and classification. The damages are classified into three main categories: cracks, potholes, and blurred markings. The YOLOv5 can achieve a relatively high detection accuracy with a score of Intersection over Union (IoU) up to 88.89% and classification accuracy with an F1 score up to 80.72%. The precision and recall scores are 84.26% and 78.38%, respectively.
... The expanded dataset helped achieve better accuracy for road damage detection and classification models. In 2020, [5,6] attempted to apply the models trained using RDD2019 to detect road damage outside Japan. Experiments were performed using the data from India and the Czech Republic. ...
... However, in RDD2020, the involvement of multiple countries required considering multiple road damage standards. Since the criteria for assessing Road Marking deteriorations such as Crosswalk or White Line Blur vary significantly across different countries, these categories were excluded from the RDD2020 dataset [1,5]. Subsequently, the following four damage categories -Longitudinal Cracks (D00), Transverse Cracks (D10), Alligator Cracks (D20), and Potholes (D40), were included in the RDD2020 dataset. ...
Preprint
Full-text available
The data article describes the Road Damage Dataset, RDD2022, which comprises 47,420 road images from six countries, Japan, India, the Czech Republic, Norway, the United States, and China. The images have been annotated with more than 55,000 instances of road damage. Four types of road damage, namely longitudinal cracks, transverse cracks, alligator cracks, and potholes, are captured in the dataset. The annotated dataset is envisioned for developing deep learning-based methods to detect and classify road damage automatically. The dataset has been released as a part of the Crowd sensing-based Road Damage Detection Challenge (CRDDC2022). The challenge CRDDC2022 invites researchers from across the globe to propose solutions for automatic road damage detection in multiple countries. The municipalities and road agencies may utilize the RDD2022 dataset, and the models trained using RDD2022 for low-cost automatic monitoring of road conditions. Further, computer vision and machine learning researchers may use the dataset to benchmark the performance of different algorithms for other image-based applications of the same type (classification, object detection, etc.).
... Concerning the interoperability of this kind of system, D. Arya et al. presented extensive work to study the usability of a single-source model in other countries and proposed models capable of detecting and classifying road damages in more than one country [22]. They conclude that the performance of a model is significantly degraded when applied to road images from another country and recommend the mixed-modelling strategy. ...
... The work presented in this paper follows the recommendations of [22], however, it is focused exclusively on the blurred line defect (D44). Thus, on one hand, we used the publicly available Road Damage Dataset RDD2019 [23] and truncated it to keep only the images containing D44 damage type. ...
Article
The future of autonomous driving is slowly approaching, but there are still many steps to take before it can become a reality. It is crucial to pay attention to road infrastructure, because without it, intelligent vehicles will not be able to operate reliably, and it will never be possible to dispense of driver’s control. This paper presents the work carried out for the detection of road markings damage using computer vision techniques. This is a complex task for which there are currently not many papers and large image sets in the literature. This study uses images from the public Road Damage Detection dataset for the D44 defect and also provides 971 new labelled images for Spanish roads. For this purpose, three detectors based on deep learning architectures (Faster RCNN, SDD and EfficientDet) have been used and single-source and mixed-source models have been studied to find the model that best fits the target images. Finally, F1-score values reaching 0.929 and 0.934 have been obtained for Japanese and Spanish images respectively which improve the state-of-the-art results by 25%. It can be concluded that the results of this study are promising, although the collection of many more images will be necessary for the scientific community to continue advancing in the future in this field of research.
... For now, several studies about the use of machine learning are very useful in developing the automotive industry [1], [2], and this fact shows that the development of the automotive industry does not always depend on the field of mechanical engineering i.e. fuel and engine performance [3][4][5], but also on the field of machine learning of computer vision. Through computer vision, a machine vision is created to detect the way surface object, which is the crucial piece of information for automated machine technologies users [6]. Determination of road surface type is crucial, especially to develop security aspects for transportation users and minimize congestion and accidents [7]. ...
... In final, the KNN classifier produces better accuracy around 78% than the Naïve Bayes classifier around 72%. This result is following previous research regarding road surface conditions, which were discussed with a different perspective [6], [23], where they stated that the KNN classifier is ISSN 2580-0817 Vol. 6, No. 1, July 2022, pp. 40-47 more recommended for detecting slippery road surface conditions during the rainy season because it is able to produce a better level of accuracy than the Naïve Bayes classifier. ...
Article
Full-text available
The mechanized ability to specify the way surface type is a piece of key enlightenment for autonomous transportation machine navigation like wheelchairs and smart cars. In the present work, the extracted features from the object are getting based on structure and surface evidence using Gray Level Co-occurrence Matrix (GLCM). Furthermore, K-Nearest Neighbor (K-NN) Classifier was built to classify the road surface image into three classes, asphalt, gravel, and pavement. A comparison of KNN and Naïve Bayes (NB) was used in present study. We have constructed a road image dataset of 450 samples from real-world road images in the asphalt, gravel, and pavement. Experiment result that the classification accuracy using the K-NN classifier is 78%, which is better as compared to Naïve Bayes classifier which has a classification accuracy of 72%. The paving class has the smallest accuracy in both classifier methods. The two classifiers have nearly the same computing time, 3.459 seconds for the KNN Classifier and 3.464 seconds for the Naive Bayes Classifier.
... The proposed work aims at utilizing AI for automating the monitoring of road conditions. Our article (Arya et al. 2021c) provides an extensive survey of the related research work. Considering that some municipalities in the world have already proposed AI-driven solutions, the proposed work addresses the requirement of municipalities/countries that are still struggling to find an operational solution. ...
... Our article (Arya et al. 2021c) addresses the research questions mentioned above. It presents an analysis assessing the applicability of Japanese data and models to develop models for India and the Czech Republic. ...
Article
The doctoral work summarized here is an application of Artificial Intelligence (AI) for social good. The successful implementation would contribute towards low-cost, faster monitoring of road conditions across different nations, resulting in safer roads for everyone. Additionally, the study provides recommendations for re-using the road image data and the Deep Learning models released by any country for detecting road damage in other countries.
... The index has been normalized to have the same scale with the other indices using Eq. (5). ...
... Few numbers of parameters applied in this model which can be easily coded in an ANN. In addition, the above three types of cracking which have been selected herein can be evaluated through deep learning models [5,20,31] leading to significant reduction in data processing and analysis time and cost. ...
Article
Pavement management systems play a major role in preservation of a road network. The core of such systems is pavement condition evaluation. In order to evaluate pavement condition, a pavement condition index is required. To date, several pavement condition indices have been developed; however, they have not been comprehensive, cost-effective, and practical for automated data collection. The objective of this study is to develop a novel pavement condition index expressing comprehensive representation of pavement condition considering structural adequacy, pavement roughness, road safety, and surface distress using a machine learning model. The outcome shows approximately 84% reduction in pavement distress analysis efforts. Moreover, the model with more than 80% accuracy and precision is highly correlated with the Pavement Condition Index (PCI). Thus, the proposed index not only provides similar results to the PCI, but it is also much more cost-effective, practical, and time-saver than the PCI.
... The presented model in [34] can detect potholes in real time, The authors collected data from multiple countries and applied different deep learning algorithms to predict the potholes automatically. To detect the road damage, they implemented a single model which is used to detect potholes in India, Japan, and, and the Czech Republic. ...
... Different metrics were used to validate the performance of sequential CNN and YOLO models in accordance with [34,36]. ...
Article
Full-text available
Detecting potholes is one of the important tasks for determining proper strategies in ITS (Intelligent Transportation System) service and road management system. Several efforts have been made for developing a technology which can automatically detect and recognize potholes. The main contribution of this paper lies in collecting the pothole data in different Indian traffic conditions and detecting of the same using a vision-based method by defining the performance of deep learning methods like sequential convolutional neural network (CNN), and anchor-based You only Look Once3 (YOLOV3) and analyzing the models in terms of resources and performance of detection. The experiments were conducted on both models and a conclusion was drawn to bring out the benefits of the model with 98% accuracy using CNN and 83% precision using Yolov3.
... Using a laser or color camera mounted at the back of the vehicles, a top view of the road, as used by [43][44][45][46][47], focuses on crack detections and potholes. A wide-angle view of the road, using a camera mounted on the front of the dashboard or top of the car, as used by [32,48,49], is used for detecting types of cracks, potholes, and types of surfaces and surface ratings. ...
Article
Full-text available
Road pavement condition assessment is essential for maintenance, asset management, and budgeting for pavement infrastructure. Countries allocate a substantial annual budget to maintain and improve local, regional, and national highways. Pavement condition is assessed by measuring several pavement characteristics such as roughness, surface skid resistance, pavement strength, deflection, and visual surface distresses. Visual inspection identifies and quantifies surface distresses, and the condition is assessed using standard rating scales. This paper critically analyzes the research trends in the academic literature, professional practices and current commercial solutions for surface condition ratings by civil authorities. We observe that various surface condition rating systems exist, and each uses its own defined subset of pavement characteristics to evaluate pavement conditions. It is noted that automated visual sensing systems using intelligent algorithms can help reduce the cost and time required for assessing the condition of pavement infrastructure, especially for local and regional road networks. However, environmental factors, pavement types, and image collection devices are significant in this domain and lead to challenging variations. Commercial solutions for automatic pavement assessment with certain limitations exist. The topic is also a focus of academic research. More recently, academic research has pivoted toward deep learning, given that image data is now available in some form. However, research to automate pavement distress assessment often focuses on the regional pavement condition assessment standard that a country or state follows. We observe that the criteria a region adopts to make the evaluation depends on factors such as pavement construction type, type of road network in the area, flow and traffic, environmental conditions, and region’s economic situation. We summarized a list of publicly available datasets for distress detection and pavement condition assessment. We listed approaches focusing on crack segmentation and methods concentrating on distress detection and identification using object detection and classification. We segregated the recent academic literature in terms of the camera’s view and the dataset used, the year and country in which the work was published, the F1 score, and the architecture type. It is observed that the literature tends to focus more on distress identification (“presence/absence” detection) but less on distress quantification, which is essential for developing approaches for automated pavement rating.
... Precision is the percentage of correctly predicted images out of the total number of predicted images, whereas recall is the percentage of correctly predicted images out of the total number of images in the actual class. Ideally, for good classifiers, both precision and recall values should be 1, but they counter each other, and increasing one usually reduces the other (Arya et al., 2020). The macro and weighted average were calculated for the precision, recall, and f1-score. ...
Article
Weed identification is fundamental toward developing a deep learning-based weed control system. Deep learning algorithms assist to build a weed detection model by using weed and crop images. The dynamic environmental conditions such as ambient lighting, moving cameras, or varying image backgrounds could affect the performance of deep learning algorithms. There are limited studies on how the different image backgrounds would impact the deep learning algorithms for weed identification. The objective of this research was to test deep learning weed identification model performance in images with potting mix (non-uniform) and black pebbled (uniform) backgrounds interchangeably. The weed and crop images were acquired by four canon digital cameras in the greenhouse with both uniform and non-uniform background conditions. A Convolutional Neural Network (CNN), Visual Group Geometry (VGG16), and Residual Network (ResNet50) deep learning architectures were used to build weed classification models. The model built from uniform background images was tested on images with a non-uniform background, as well as model built from non-uniform background images was tested on images with uniform background. Results showed that the VGG16 and ResNet50 models built from non-uniform background images were evaluated on the uniform background, achieving models' performance with an average f1-score of 82.75% and 75%, respectively. Conversely, the VGG16 and ResNet50 models built from uniform background images were evaluated on the non-uniform background images, achieving models' performance with an average f1-score of 77.5% and 68.4% respectively. Both the VGG16 and ResNet50 models' performances were improved with average f1-score values between 92% and 99% when both uniform and non-uniform background images were used to build the model. It appears that the model performances are reduced when they are tested with images that have different object background than the ones used for building the model.
... As shown in Equation (4), the calculation of the precision denotes the percentage of the true positives amongst the predicted result consisting of true positives and false positives. The recall represents the percentage of true positives amongst the predicted result composed of true positives and false positives, as shown in Equation (5). Based on the precision and recall, the F1-score can be calculated as shown in Equation (6): ...
Article
Full-text available
Pavement cracks can result in the degradation of pavement performance. Due to the lack of timely inspection and reparation for the pavement cracks, with the development of cracks, the safety and service life of the pavement can be decreased. To curb the development of pavement cracks, detecting these cracks accurately plays an important role. In this paper, an automatic pavement crack detection method is proposed. For achieving real-time inspection, the YOLOV5 was selected as the base model. Due to the small size of the pavement cracks, the accuracy of most of the pavement crack deep learning-based methods cannot reach a high degree. To further improve the accuracy of those kind of methods, attention modules were employed. Based on the self-building datasets collected in Linyi city, the performance among various crack detection models was evaluated. The results showed that adding attention modules can effectively enhance the ability of crack detection. The precision of YOLOV5-CoordAtt reaches 95.27%. It was higher than other conventional and deep learning methods. According to the pictures of the results, the proposed methods can detect accurately under various situations.
... The curation process is shown in Figure 1. The curated dataset includes 506 images, carefully selected from the public dataset RDD2020 [6] which consists of 26,620 images of 3 countries (i.e., Japan, Czech, and India) taken by smartphone, with 850 road cracks and potholes. We selected images of acceptable quality and rejected the ones of bad quality. ...
Chapter
Full-text available
Most crack image datasets are developed for crack segmentation or detection. They cannot be used to train a deep learning model to detect and segment cracks simultaneously. Most of existing datasets do not include a very accurate annotation. Besides, some crack images cannot be used to train deep learning models because of their inferior quality. In this paper, we propose a promising curated crack image dataset that allows the development of crack segmentation, detection, and classification on the same set of images simultaneously. There is no dataset for road crack that involves detection and segmentation tasks to the best of our knowledge. The current version of the curated database consists of 506 images derived from the RDD2020 dataset taken from multi-countries (Japan, Czech, and India). We use the curated dataset to build different deep learning-based crack detection and segmentation methods. Our experiments demonstrate that the proposed dataset yields promising results for crack detection and segmentation.
... This dataset is dedicated to road damage detection, unlike RDD-2018 and RDD-2019, it only covers 4 damage classes, i.e. potholes, alligator cracks, longitudinal cracks, and transverse cracks. Some extra damage classes are included in images collected in Japan for data consistency, more details can be found in [90]. For each damage in the image from the training set, the damage class and its corresponding bounding box coordinates are labeled. ...
Article
Deep learning breakthrough stimulates new research trends in civil infrastructure inspection, whereas the lack of quality-guaranteed, human-annotated, free-of-charge, and publicly available defect datasets with sufficient amounts of data hinders the progress of deep learning in defect inspection. To boost research in deep learning-based visual defect inspection, this paper first reviews and summarizes 40 publicly available defect datasets, covering common defects in various types of buildings and infrastructures. The taxonomy of the datasets is proposed based on specific deep learning objectives (classification, segmentation, and detection). Clarifications are also made for each dataset regarding its corresponding data volume, data resolution, data source, defect categories covered, infrastructure types focused, material types targeted, algorithms adopted for validation, annotation levels, context levels, and publication license for future utilization. Consequently, the summarized defect datasets offer around 13.38M labeled images, cover more than 5 defect types, 5 infrastructure types, 5 material types, and 3 levels of image context. Given that the crack is a common interest in civil engineering, this paper further combines existing datasets with self-labeled crack images to establish a benchmark dataset providing more than 15,000 and 11,000 labeled images for crack classification and segmentation, respectively. Based on the established crack dataset, experiments are conducted for classification, segmentation, and the subsequent non-maximum suppression-based detection tasks. The proposed multi-branch self-attention module and multi-stage-fused attentional pyramid network have been successfully adapted into the state-of-the-art (SOTA) classification network-Swin Transformer and segmentation networks including DeepLab V3+, DenseNet, and Full Resolution ResNet. The resulting classification network achieves 88.0% accuracy, and the adapted segmentation models reach 77.8%,77.6%,76.9% mIoU (mean Intersection over Union), respectively. Moreover, a comprehensive comparison between 11 SOTA classification algorithms and 12 SOTA segmentation algorithms has been conducted. The algorithms proposed in this work are shown to achieve satisfactory performance with an acceptable efficiency on modern graphic processing units. Detailed suggestions are provided for constructing high-quality datasets and inspection algorithms. Finally, this paper remarks on the quantity, diversity, difficulty, and scalability of the reviewed defect datasets, feasibility on robotic platforms, superiority of proposed algorithms, and criticality of algorithm comparison results, formulating a solid baseline for future defect inspection research.
... The first one is the views of the pavement images. Most of these datasets captured the images using smartphones with wide views, such as the most successful one RDD-2020 [2]. However, many real-world projects of pavement inspection require a top-down view because pavement maintenance needs information on damage morphology, such as the dimensions of cracks and portholes. ...
Preprint
Pavement damage segmentation has benefited enormously from deep learning. % and large-scale datasets. However, few current public datasets limit the potential exploration of deep learning in the application of pavement damage segmentation. To address this problem, this study has proposed Pavementscapes, a large-scale dataset to develop and evaluate methods for pavement damage segmentation. Pavementscapes is comprised of 4,000 images with a resolution of $1024 \times 2048$, which have been recorded in the real-world pavement inspection projects with 15 different pavements. A total of 8,680 damage instances are manually labeled with six damage classes at the pixel level. The statistical study gives a thorough investigation and analysis of the proposed dataset. The numeral experiments propose the top-performing deep neural networks capable of segmenting pavement damages, which provides the baselines of the open challenge for pavement inspection. The experiment results also indicate the existing problems for damage segmentation using deep learning, and this study provides potential solutions.
... De maneira geral, muitos municípios e autoridades procuram implementar avaliações automatizadas dos danos nas pavimentações. No entanto, geralmente se deparam com a limitações de tecnologia, de know-how e de recursos financeiros para adquirir equipamentos deúltima geração para coleta e análise de dados em tais contextos [Arya et al. 2021a Desta feita, para colaborar na resolução do problema da inspeção automática da qualidade de pavimentações asfálticas, o objetivo deste trabalho consiste em avaliar o desempenho de Redes Neurais Convolucionais Regionais (R-CNNs, do inglês Regional Convolutional Neural Networks) da família YOLO (acrônimo para You Only Look Once) na tarefa de Visão Computacional de detecção (localização e classificação) de quatro tipos distintos de falhas. Nesta solução, considerou-se a aquisição de experiência a partir de 18.667 exemplos oriundos da base de dados RDD2020 [Arya et al. 2021b]. ...
Conference Paper
Neste artigo foi abordado o problema da inspeção automática de danos em pavimentações como um problema de detecção em Visão Computacional com vistas a colaborar para o desenvolvimento de soluções que apoiem cidades inteligentes na melhoria da qualidade e da segurança no trânsito. Neste sentido considerouse um cenário experimental com dados realísticos oriundos de três países diferentes e quatro configurações para as redes YOLO. Ao comparar os resultados obtidos com a literatura, verificam-se ganhos significativos no tempo de previsão ainda que utilizando um menor número de parâmetros, produzindo um mAP igual a 0,53. Também avaliamos a solução proposta em um estudo de caso com imagens do Brasil, o qual ressaltou diversos desafios práticos a serem levados em conta na ocasião da proposição de modelos automáticos de detecção para o problema em consideração.
... From earthquakes, the road damage caused by the seismic waves traveling through the earth is much greater than that of normal road degradation, and requires a new methodology for detecting road damage. Most of the current road damage detection solutions aim to detect the smallest of cracks [2]- [4], [8], [9], whereas in our case, such small of cracks would minimally impact the stability of the selfdriving vehicle. We consider these minor-sized cracks as generally within millimeters or centimeters wide, whereas the cracks considered for self-navigation in earthquake-struck zones are major-sized and are generally within tens of centimeters wide or larger. ...
Article
Full-text available
The role of unmanned vehicles for searching and localizing the victims in disaster impacted areas such as earthquake-struck zones is getting more important. Self-navigation on an earthquake zone has a unique challenge of detecting irregularly shaped obstacles such as road cracks, debris on the streets, and water puddles. In this paper, we characterize a number of state-of-the-art Fully Convolutional Network (FCN) models on mobile embedded platforms for self-navigation at these sites containing extremely irregular obstacles. We evaluate the models in terms of accuracy, performance, and energy efficiency. We present a few optimizations for our designed vision system. Lastly, we discuss the trade-offs of these models for a couple of mobile platforms that can each perform self-navigation. To enable vehicles to safely navigate earthquake-struck zones, we compile a new annotated image database of various earthquake impacted regions that is different than traditional road damage databases. We train our database with a number of state-of-the-art semantic segmentation models in order to identify obstacles unique to earthquake-struck zones. Based on the statistics and tradeoffs, an optimal FCN model is selected and applied to the mobile vehicular platforms. To our best knowledge, this is the first study that identifies unique challenges and discusses the accuracy, performance, and energy impact of edge-based self-navigation mobile vehicles for earthquake-struck zones. Our proposed database and trained models are publicly available.
... The major categories of image processing techniques used for automatic road crack recognition [3] include change-detection-based methods, threshold intensitybased methods, edge-detection-based algorithms [4,5], area-based methods that involve the algorithms making use of morphological operations, percolation and segmentation, etc. Further, several other methods have been developed based on machine learning techniques, including deep neural networks [6][7][8][9], support vector machines [10], k-means clustering, etc. Some authors [11] utilize a combination of two or more approaches. ...
Chapter
Road cracks have a stimulating effect on the short-term and long-term life of pavements. For efficient maintenance of pavements, early detection of these cracks is imperative. The conventional methods for road condition assessments involve manual surveys that fail to meet the present-day requirements. As a result, there arises a need to use image-based approaches that can automatically detect the pavement conditions from the images of the roads. The current manuscript presents one such approach utilizing Gray-Level Co-occurrence Matrix (GLCM) and the associated features. In the proposed approach, the GLCM features are used to develop a road crack detection system based on machine learning techniques, capable of detecting the cracks, localizing them in the image, and providing valuable inputs related to the severity of the cracks present on a road section. The applicability of the proposed approach to separate cracks and non-cracks is assessed using images containing different types of cracks, captured using disparate mechanisms and from various locations (India, Japan, and China). The outcomes show promising results, proving the efficacy of the presented approach.
... Finally, it was proven that the proposed method could be used to classify road anomalies into eight categories with high precision. Singh et al. summarized some road anomaly detection and classification methods from the 2018 IEEE Big Data Cup international conference, showing that the use of smartphone cameras to obtain images through the deep learning algorithm can quickly detect and classify different types of road damage [21]. Detection methods based on smartphones have rapidly become popular research directions in the field of road detection, because of their superiority and feasibility. ...
Article
Full-text available
Road surface condition detection is an important application for many intelligent transportation systems (ITSs). A manhole cover depression is one of the common factors affecting road conditions. Smartphones are equipped with different sensors, which can be used to collect image data and inertial data. A new large-scale manhole cover detection dataset is developed by using smartphones to collect road image data, and a hierarchical classification method based on the convolutional neural network is proposed in this paper. The proposed method first coarsely classifies the images into nonrainy and rainy types and then performs manhole cover detections based on the coarse classification results. As a result, the proposed method achieves an accuracy of approximately 86.3% for road manhole cover detection. Based on the observation that different degrees of manhole cover subsidence produce different degrees of inertial sensor data, this paper used a machine learning method, which can automatically classify the detected manhole covers into different degrees of subsidence, namely good, average, and poor. The average recalls, average precisions, and average F1-measures achieve approximately 87.3%, 86.9%, and 87.2% accuracy, respectively. The results show that the proposed approach can effectively detect manhole covers in different weather and road conditions, which can effectively reduce the cost of road manhole cover data collection and detection, providing a new method for road manhole cover detection.
... Actually, a part of RDD2020 was utilized for Global Road Detection Challenge 2020. Road images were captured using a smartphone running a publicly available image-capturing application developed by Sekimoto Lab [64]. ...
Article
Full-text available
This work introduces the need to develop competitive, low-cost and applicable technologies to real roads to detect the asphalt condition by means of Machine Learning (ML) algorithms. Specifically, the most recent studies are described according to the data collection methods: images, ground penetrating radar (GPR), laser and optic fiber. The main models that are presented for such state-of-the-art studies are Support Vector Machine, Random Forest, Naïve Bayes, Artificial neural networks or Convolutional Neural Networks. For these analyses, the methodology, type of problem, data source, computational resources, discussion and future research are highlighted. Open data sources, programming frameworks, model comparisons and data collection technologies are illustrated to allow the research community to initiate future investigation. There is indeed research on ML-based pavement evaluation but there is not a widely used applicability by pavement management entities yet, so it is mandatory to work on the refinement of models and data collection methods.
... From earthquakes, the road damage caused by the seismic waves traveling through the earth is much greater than that of normal road degradation, and requires a new methodology for detecting road damage. Most of the current road damage detection solutions aim to detect the smallest of cracks [1]- [3], [7], [8], whereas in our case, such small of cracks would minimally impact the stability of the self-driving vehicle. We consider these minor-sized cracks as generally within millimeters or centimeters wide, whereas the cracks considered for selfnavigation in earthquake-struck zones are major-sized and are generally within tens of centimeters wide or larger. ...
Preprint
Full-text available
The role of unmanned vehicles for searching and localizing the victims in disaster impacted areas such as earthquake-struck zones is getting more important. Self-navigation on an earthquake zone has a unique challenge of detecting irregularly shaped obstacles such as road cracks, debris on the streets, and water puddles. In this paper, we characterize a number of state-of-the-art FCN models on mobile embedded platforms for self-navigation at these sites containing extremely irregular obstacles. We evaluate the models in terms of accuracy, performance, and energy efficiency. We present a few optimizations for our designed vision system. Lastly, we discuss the trade-offs of these models for a couple of mobile platforms that can each perform self-navigation. To enable vehicles to safely navigate earthquake-struck zones, we compiled a new annotated image database of various earthquake impacted regions that is different than traditional road damage databases. We train our database with a number of state-of-the-art semantic segmentation models in order to identify obstacles unique to earthquake-struck zones. Based on the statistics and tradeoffs, an optimal CNN model is selected for the mobile vehicular platforms, which we apply to both low-power and extremely low-power configurations of our design. To our best knowledge, this is the first study that identifies unique challenges and discusses the accuracy, performance, and energy impact of edge-based self-navigation mobile vehicles for earthquake-struck zones. Our proposed database and trained models are publicly available.
... Many studies have been conducted on assisting engineers with the aim of constructing techniques to perform the maintenance inspection efficiently and accurately [6][7][8][9]. Especially, in recent years, computer vision approaches have been widely applied, construction of techniques for the automatic detection and classification of distresses [10,11] has been studied. ...
Article
Full-text available
This paper presents reliable estimation of deterioration levels via late fusion using multi-view distress images for practical inspection. The proposed method simultaneously solves the following two problems that are necessary to support the practical inspection. Since maintenance of infrastructures requires a high level of safety and reliability, this paper proposes a neural network that can generate an attention map from distress images and text data acquired during the inspection. Thus, deterioration level estimation with high interpretability can be realized. In addition, since multi-view distress images are taken for single distress during the actual inspection, it is necessary to estimate the final result from these images. Therefore, the proposed method integrates estimation results obtained from the multi-view images via the late fusion and can derive an appropriate result considering all the images. To the best of our knowledge, no method has been proposed to solve these problems simultaneously, and this achievement is the biggest contribution of this paper. In this paper, we confirm the effectiveness of the proposed method by conducting experiments using data acquired during the actual inspection.
Article
Currently, forest road monitoring reached a critical stage and need requires low-cost or cost-effective monitoring. Today, smartphones have been used on public roads to identify road deterioration due to benefits such as usability, cost, ease of access, and expected accuracy. The use of smartphones in forest road development by the proposed system is a distributed information system that converts data from enterprise mode to field mode by harvesting and assessing forest road conditions and image processing technologies. The technology proposed in this research allows different information YOLOv4-v5 with improvements to this version including mosaic data augmentation and automatic learning of enclosing frames. In this research, we applied a new hybrid YOLOv4-v5 to the dataset’s general applicability. We assessed the forest road dataset to run an experiment, smartphone images by various aspects of the smartphone images (SI) dataset which is specialized for detecting forest road deterioration. To enhance YOLO’s ability to detect damaged scenes by proposing a new technique that takes information into frames. We expanded the scope of the model by applying it to a new orientation estimation task. The main disadvantage is the provision of qualitative model information on forest road activity and the indication of potential deterioration.
Article
Full-text available
Urban and residential roads play an integral role in the infrastructure system of a city. Although they take up a large proportion of the national road network, maintenance plans for urban roads are beset by many problems. These include difficulty in collecting enormous volumes of data, implementing analyses, and interpreting results because of complicated frameworks. Thus, this study aims to introduce an effective and reliable method of formulating a maintenance plan using integrated criteria of spatial autocorrelation analysis and roadside conditions. The results demonstrate that defective pavements are clustered in certain areas, for example, mountainous and forested areas, which indicate environmental effects. Using a mixed index as a criterion for prioritization, approximately 55% of roadside residents (represented by the total residential housing floor area) and 90% of commercial and medical facilities surrounding critical sections gained benefit from maintenance activities in the second year. Importantly, the proposed method presents the advantages of simplifying implications and quantitative outcomes that could support local agents in not only implementing but also making decisions and interpreting such decisions for the community.
Article
Full-text available
Pavement condition prediction plays a vital role in pavement maintenance. Many prediction models and analyses have been conducted based on long-term pavement condition data. However, the condition evaluation for road sections can hardly support daily routine maintenance. This paper uses high-frequency pavement distress data to explore the relationship between distress initiation, weather, and geometric factors. Firstly, a framework is designed to extract the initial time of pavement distress. Weather and geometric data are integrated to establish a pavement distress initiation dataset. Then, the time-lag cross-correlation analysis methods were utilized to explore the relationship between distress initiation and environmental factors. In addition, the logistic regression model is used to establish the distress initiation prediction model. Finally, Akaike information criterion (AIC), Bayesian information criterions (BIC), and areas under receiver operating characteristic curves (AUC) of logistic regression models with or without time-lag variables are compared as performance measurements. The results show that pavement distress initiation is susceptible to weather factors and location relationships. Daily total precipitation, minimum temperature, and daily average temperature have a time delay effect on the initiation of the pavement distress. Distress initiation is negatively correlated with the distance from the nearby intersection and positively correlated with adjacent distresses. The weather factors, considering the time-lag effect, can improve the model performance of the distress initiation prediction model and provide support for emergency management after severe weather.
Article
Deep learning methods have attained promising performance on road defect detection from on-board cameras. However, they oftentimes rely heavily on well-annotated datasets with sufficient samples, limiting the practical applications when only few labeled samples are available. To fill this gap, this paper proposes a framework based on Faster Region-Convolutional Neural Network (Faster R-CNN) for road defect detection with scarce and cross-domain data. First, a defect weighting branch is developed to enable Faster R-CNN to quickly learn to detect road defects with few annotated data, then a data augmentation method is proposed to enlarge the abundance of annotated data and alleviate the cross-domain issue. Experimental results demonstrate that the proposed framework has attained better performance compared to a state-of-the-art few-shot detector, in terms of an improved mean average precision of 1.83% when only limited samples (i.e., 30 images per category) are provided for training. In the future, the proposed framework could also be extended to other detection tasks with limited data (e.g., construction vehicle detection), allowing humans to reduce their efforts and time required for arduous data collection and annotation.
Article
Pavement damage detection is essential for subsequent road maintenance decisions. However, recent detection networks have low accuracy and fail to detect most diseases on the road, which means that testing is very inefficient. Therefore, this study uses the unmanned aerial vehicle (UAV) road damage database and describes a multi-level attention mechanism called Multi-level Attention Block (MLAB) to strengthen the utilization of essential features by the You Only Look Once version 3 (YOLO v3). Adding MLAB between the backbone and feature fusion parts effectively increases the mAP value of the proposed network to 68.75%, while the accuracy of the original network is only 61.09%. The network is able to detect longitudinal cracks, transverse cracks, repairs, and potholes with high accuracy, and significantly improves the accuracy of alligator cracks and oblique cracks. The findings of this study will accelerate the application of non-destructive automatic road damage detection.
Chapter
Article
Full-text available
Road damage detection is an important task to ensure road safety and realize the timely repair of road damage. The previous manual detection methods are low in efficiency and high in cost. To solve this problem, an improved YOLOv5 road damage detection algorithm, MN-YOLOv5, was proposed. We optimized the YOLOv5s model and chose a new backbone feature extraction network MobileNetV3 to replace the basic network of YOLOv5, which greatly reduced the number of parameters and GFLOPs of the model, and reduced the size of the model. At the same time, the coordinate attention lightweight attention module is introduced to help the network locate the target more accurately and improve the target detection accuracy. The KMeans clustering algorithm is used to filter the prior frame to make it more suitable for the dataset and to improve the detection accuracy. To improve the generalization ability of the model, a label smoothing algorithm is introduced. In addition, the structure reparameterization method is used to accelerate model reasoning. The experimental results show that the improved YOLOv5 model proposed in this paper can effectively identify pavement cracks. Compared with the original model, the mAP increased by 2.5%, the F1 score increased by 2.6%, and the model volume was smaller than that of YOLOv5. 1.62 times, the parameter was reduced by 1.66 times, and the GFLOPs were reduced by 1.69 times. This method can provide a reference for the automatic detection method of pavement cracks.
Article
Rapid inspection of urban road cracks is vital to maintain traffic smoothness and ensure traffic safety. A rapid pavement crack inspection method using low-altitude aerial images captured by the unmanned aerial system (UAS) and deep-learning aided 3D reconstruction method, learning-based object segmentation algorithm is proposed to measure road cracks automatically. The contributions include: (1) An efficient 3D reconstruction method for low-altitude aerial images captured by UAS is proposed, which applies an instance segmentation network to segment road targets from raw images with complex backgrounds first and then perform structure from motion (SFM) to reconstruct a large-scale road orthophoto from a large number of aerial images. (2) To detect cracks from the reconstructed large-size road orthophoto, the sliding window algorithm and U-Net model optimized with transformer structure are used to automatically identify and segment the cracks from the orthophoto at a pixel level. Then the connected domain feature analysis method is used to measure the road crack length. The proposed method is applied to detect road cracks in a 1.5km2 area of a city. The results show that the proposed method can effectively and accurately detect cracks and measure the length of cracks in the 6.2km-long road, which proves the practicality of the proposed method.
Article
Full-text available
Road cycling is a cycling discipline in which riders ride on public roads. Traffic calming measures are made to make public roads safer for everyday usage for all its users. However, these measures are not always yielding a safer cycling racecourse. In this paper we present a methodology that inspects the safety of roads tailored to road bicycle racing. The automated approach uses computer vision and geospatial analysis to give an indicative racecourse safety score based on collected, calculated and processed multimodal data. The current version of our workflow uses OpenStreetMap (OSM), turn detection and stage type / bunch sprint classification for the geospatial analysis and uses road segmentation and an extensible object detector that is currently trained to detect road cracks and imperfections for visual analysis. These features are used to create a mechanism that penalizes dangerous elements on the route based on the remaining distance and the generated penalties with its relative importance factors. This results in a comprehensive safety score along with a detailed breakdown of the most concerning passages on the course which can be used by race organizers and officials to help them in the iterative process to create an engaging, yet safe course for the riders.
Article
Automatic pavement distress detection is essential to monitoring and maintaining pavement condition. Currently, many deep learning-based methods have been utilized in pavement distress detection. However, distress segmentation remains as a challenge under complex pavement conditions. In this paper, a novel deep neural network architecture, W-segnet, based on multi-scale feature fusions, is proposed for pixel-wise distress segmentation. The proposed W-segnet concatenates distress location information with distress classification features in two symmetric encoder-decoder structures. Three major types of distresses: crack, pothole, and patch are segmented and the results were discussed. Experimental results show that the proposed W-segnet is robust in various scenarios, achieving a mean pixel accuracy (MPA) of 87.52% and a mean intersection over union (MIoU) of 75.88%. The results demonstrate that W-segnet outperforms other state-of-the-art semantic segmentation models of U-net, SegNet, and PSPNet. Comparison of cost of model training and inference indicates that W-segnet has the largest number of parameters, which needs a slightly longer training time while it does not increase the inference cost. Four public datasets were used to test the generalization ability of the proposed model and the results demonstrate that the W-segnet possesses well segmentation performance.
Article
Full-text available
Damage detection via drones is fundamental in infrastructure health assessment. However, object scale variation due to drones' swift movement and sparse scenes make damage detection challenging. This paper describes a multi-task framework, EnsembleDetNet, for damage detection and multi-label scene classification by leveraging object detectors and classifiers based on ensemble learning which induces diversity and strength-correlation. Further, a novel attention module that significantly improves EnsembleDetNet by about 5% is proposed via explicit ensembling of parallel and sequential channel and spatial attention maps. Extensive experiments with a public dataset and an onsite verification utilizing a micro drone indicate that EsembleDetNet outperforms state-of-the-art detectors and classifiers under variant evaluation metrics. EnsembleDetNet has the potential to become a new paradigm in infrastructure health assessment.
Chapter
Road quality significantly influences safety and comfort while driving. Especially for most kinds of two-wheelers, road damage is a real threat, where vehicle components and enjoyment are heavily impacted by road quality. This can be avoided by planning a route considering the surface quality. We propose a new publicly available and manually annotated dataset collected from Google Street View photos. This dataset is devoted to a road quality classification task considering six levels of damage. We evaluated some preprocessing methods such as shadow removal, CLAHE, and data augmentation. We adapted several pre-trained networks to classify road quality. The best performance was reached by MobileNet using augmented dataset (75.55%).
Article
Full-text available
Article
We discuss a range of problems relating to road pavement defects detection and modern approaches to their solution. The presented comparison of publicly available datasets allows one to make a conclusion that the problem of segmentation of road pavement defects in driver wide-view road images is difficult and poorly investigated. To solve this problem, we have developed algorithms for generating a synthetic dataset for cracks and potholes distress based on computer graphics methods and deep convolutional generative adversarial networks. A comparison of the accuracy of road distress segmentation was performed by training a fully convolutional neural network U-Net on real and combined datasets. © 2021, Institution of Russian Academy of Sciences. All rights reserved.
Article
Decorating speed bumps is a common and effective measure to compulsively reduce vehicular velocity in dangerous regions, especially at tunnels, slopes, and parking lots. Despite a decade of deployment, the bump information is still sporadic in map-based applications, e.g., driving alert ahead of bumps. One major obstacle is the lack of an up-to-date bump database, thus service providers have to hire dedicated personnel to gather and periodically calibrate such data, which is effort-intensive and timing-consuming to large-scale coverage. In this paper, we propose Personal Bump Seeker (PBS), a novel mobile crowdsensing application to automatically identify and update speed bumps in urban cities. Specially, we formulate bump detection as a regression model, thus improve the robustness of inertial patterns on temporal domain. We also explore a leader-follower mechanism that automatically extracts the bump signal for training and inference in individual smartphones, regardless of different smartphones, vehicles, and driving habits during crowdsensing. Finally, we implement a prototype and conduct extensive experiments on large-scale real-world traffic datasets collected by the DiDi platform, and the results demonstrate our effectiveness in producing a city-level bump database.
Article
Full-text available
This data article provides details for the RDD2020 dataset comprising 26336 road images from India, Japan, and the Czech Republic with more than 31000 instances of road damage. The dataset captures four types of road damage: longitudinal cracks, transverse cracks, alligator cracks, and potholes; and is intended for developing deep learning-based methods to detect and classify road damage automatically. The images in RDD2020 were captured using vehicle-mounted smartphones, making it useful for municipalities and road agencies to develop methods for low-cost monitoring of road pavement surface conditions. Further, the machine learning researchers can use the datasets for benchmarking the performance of different algorithms for solving other problems of the same type (classification, object detection, etc.). RDD2020 is freely available at [1]. The latest updates and the corresponding articles related to the dataset can be accessed at [2].
Article
Full-text available
Proper maintenance of roads is an extremely complex task and also an important issue all over the world. One of the most critical road monitoring and maintenance activities is the detection of road anomalies such as potholes. Identification of potholes is necessary to avoid road accidents, prevent damage of vehicles, enhance travelling comforts, etc. Although maintenance of roads is considered to be a serious issue by the authorities over the years, lack of proper detection and mapping of road potholes makes the issue more severe. To overcome this problem, an end-to-end system called PotSpot is built for real-time detection, monitoring, and spatial mapping of potholes across the city A Convolutional Neural Network (CNN) model is proposed and evaluated on real-world dataset for pothole detection. Additionally, real-time pothole-marked maps are generated with the help of Google Maps API (Application Programming Interface). To provide an end-to-end service through this system, both the pothole detection and pothole mapping are integrated through an android application. The proposed model is also compared with six baselines namely Artificial Neural Network (ANN), Support Vector Machine (SVM), K-Nearest Neighbors (KNN), and three pre-trained CNN models InceptionV3, VGG19 and VGG16 in terms of performance metrics to verify its effectiveness. The proposed model achieves better accuracy (≈ 97.6 %) as compared to the above-mentioned baseline methods. It is also observed that the Area Under the Curve (AUC) value for the proposed pothole detection model (AUC= 0.97) is higher than the baseline methods.
Conference Paper
Full-text available
Information about road damages are of great interest for federal road authorities and their infrastructure management as well as the automated driving task and thus safety and comfort of vehicle occupants. Therefore, the investigation of the automatic detection of different types of road damages by images from a front-facing camera in the vehicle is of utter importance. Here we show a novel deep Learning approach utilizes the pre-trained Faster Region Based Convolutional Neural Networks (R-CNN). The data basis of our work is provided by the 'IEEE BigData Cup Challenge' and its dataset 'RDD-2020' with a large number of labelled images from Japan, India and the Czech Republic. In the first step, we classify the destination of the image followed by expert networks for each region. Between the explanation of our applied Deep Learning methodology, some remaining sources of errors are discussed and further, partly failed approaches during our development period are presented, which could be of interest for future work. Our results are convincing and we are able to achieve an F1 score of 0.487 across all regions for longitudinal and lateral cracks, alligator cracks and potholes.
Conference Paper
Full-text available
Deep learning-based technology is a good key to unlock the object detection tasks in our real world. By using deep neural networks, we could break a problem that is dangerous and very time-consuming but has to be done every day like detecting the road state. This paper describes the solution using YOLO to detect the various types of road damage in the IEEE BigData Cup Challenge 2020. Our YOLOv5x based-solution is lightweight and fast, even it has good accuracy. We achieved an F1 score of 0.58 using our ensemble model with TTA, and it could be an adequate candidate for detecting real road damage in real-time.
Article
Full-text available
A variety of civil engineering applications require the identification of cracks in roads and buildings. In such cases, it is frequently helpful for the precise location of cracks to be identified as labelled parts within an image to facilitate precision repair for example. CrackIT is known as a crack detection algorithm that allows a user to choose between a block-based or a pixel-based approach. The block-based approach is noise-tolerant but is not accurate in edge localization while the pixel-based approach gives accurate edge localisation but is not noise-tolerant. We propose a new approach that combines both techniques and retains the advantages of each. The new method is evaluated on three standard crack image datasets. The method was compared with the CrackIT method and three deep learning methods namely, HED, RCF and the FPHB. The new approach outperformed the existing arts and reduced the discretisation errors significantly while still being noise-tolerant.
Conference Paper
Full-text available
Pavement condition evaluation is essential to time the preventative or rehabilitative actions and control distress propagation. Failing to conduct timely evaluations can lead to severe structural and financial loss of the infrastructure and complete reconstructions. Automated computer-aided surveying measures can provide a database of road damage patterns and their locations. This database can be utilized for timely road repairs to gain the minimum cost of maintenance and the asphalt's maximum durability. This paper introduces a deep learning-based surveying scheme to analyze the image-based distress data in real-time. A database consisting of a diverse population of crack distress types such as longitudinal, transverse, and alligator cracks, photographed using mobile-device is used. Then, a family of efficient and scalable models that are tuned for pavement crack detection is trained. Proposed models, resulted in F1-scores, ranging from 52% to 56%, and average inference time from 178-10 images per second. Finally, the performance of the object detectors are examined, and error analysis is reported against various images. The source code is available at https://github.com/mahdi65/roadDamageDetection2020.
Article
Full-text available
Machine learning can produce promising results when sufficient training data are available; however, infrastructure inspections typically do not provide sufficient training data for road damage. Given the differences in the environment, the type of road damage and the degree of its progress can vary from structure to structure. The use of generative models, such as a generative adversarial network (GAN) or a variational autoencoder, makes it possible to generate a pseudoimage that cannot be distinguished from a real one. Combining a progressive growing GAN along with Poisson blending artificially generates road damage images that can be used as new training data to improve the accuracy of road damage detection. The addition of a synthesized road damage image to the training data improves the F ‐measure by 5% and 2% when the number of original images is small and relatively large, respectively. All of the results and the new Road Damage Dataset 2019 are publicly available (https://github.com/sekilab/RoadDamageDetector).
Article
Full-text available
Automatic crack detection on pavement surfaces is an important research field in the scope of developing an intelligent transportation infrastructure system. In this paper, a cost effective solution for road crack inspection by mounting the commercial grade sport camera, GoPro, on the rear of the moving vehicle is introduced. Also, a novel method called ConnCrack combining conditional Wasserstein generative adversarial network and connectivity maps is proposed for road crack detection. In this method, a 121-layer densely connected neural network with deconvolution layers for multi-level feature fusion is used as generator, and a 5-layer fully convolutional network is used as discriminator. To overcome the scattered output issue related to deconvolution layers, connectivity maps are introduced to represent the crack information within the proposed ConnCrack. The proposed method is tested on a publicly available dataset as well our collected data. The results show that the proposed method achieves state-of-the-art performance compared with other existing methods in terms of precision, recall and F1 score.
Article
Full-text available
Governments are faced with countless challenges to maintain conditions of road networks. This is due to financial and physical resource deficiencies of road authorities. Therefore, low-cost automated systems are sought after to alleviate these issues and deliver adequate road conditions for citizens. There have been several attempts at creating such systems and integrating them within Pavement management systems. This paper utilizes replicable deep learning techniques to carry out hotspot analyses on urban road networks highlighting important pavement distress types and associated severities. Following this, analyses were performed illustrating how the hotspot analysis can be carried out to continuously monitor the structural health of the pavement network. The methodology is applied to a road network in Sicily, Italy where there are numerous roads in need of rehabilitation and repair. Damage detection models were created which accurately highlight the location and a severity assessment. Harmonized distress categories, based on industry standards, are utilized to create practical workflows. This creates a pipeline for future applications of automated pavement distress classification and a platform for an integrated approach towards optimizing urban pavement management systems.
Preprint
Full-text available
Automated pavement distresses detection using road images remains a challenging topic in the computer vision research community. Recent developments in deep learning has led to considerable research activity directed towards improving the efficacy of automated pavement distress identification and rating. Deep learning models require a large ground truth data set, which is often not readily available in the case of pavements. In this study, a labeled dataset approach is introduced as a first step towards a more robust, easy-to-deploy pavement condition assessment system. The technique is termed herein as the Pavement Image Dataset (PID) method. The dataset consists of images captured from two camera views of an identical pavement segment, i.e., a wide-view and a top-down view. The wide-view images were used to classify the distresses and to train the deep learning frameworks, while the top-down view images allowed calculation of distress density, which will be used in future studies aimed at automated pavement rating. For the wide view group dataset, 7,237 images were manually annotated and distresses classified into nine categories. Images were extracted using the Google Application Programming Interface (API), selecting street-view images using a python-based code developed for this project. The new dataset was evaluated using two mainstream deep learning frameworks: You Only Look Once (YOLO v2) and Faster Region Convolution Neural Network (Faster R-CNN). Accuracy scores using the F1 index were found to be 0.84 for YOLOv2 and 0.65 for the Faster R-CNN model runs; both quite acceptable considering the convenience of utilizing Google maps images.
Article
Full-text available
Pavement crack detection is a critical task for insuring road safety. Manual crack detection is extremely time-consuming. Therefore, an automatic road crack detection method is required to boost this progress. However, it remains a challenging task due to the intensity inhomogeneity of cracks and complexity of the background, e.g., the low contrast with surrounding pavements and possible shadows with a similar intensity. Inspired by recent advances of deep learning in computer vision, we propose a novel network architecture, named feature pyramid and hierarchical boosting network (FPHBN), for pavement crack detection. The proposed network integrates context information to low-level features for crack detection in a feature pyramid way, and it balances the contributions of both easy and hard samples to loss by nested sample reweighting in a hierarchical way during training. In addition, we propose a novel measurement for crack detection named average intersection over union (AIU). To demonstrate the superiority and generalizability of the proposed method, we evaluate it on five crack datasets and compare it with the state-of-the-art crack detection, edge detection, and semantic segmentation methods. The extensive experiments show that the proposed method outperforms these methods in terms of accuracy and generalizability. Code and data can be found in https://github.com/fyangneil/pavement-crack-detection.
Conference Paper
Full-text available
Article
Full-text available
SDNET2018 is an annotated image dataset for training, validation, and benchmarking of artificial intelligence based crack detection algorithms for concrete. SDNET2018 contains over 56,000 images of cracked and non-cracked concrete bridge decks, walls, and pavements. The dataset includes cracks as narrow as 0.06 mm and as wide as 25 mm. The dataset also includes images with a variety of obstructions, including shadows, surface roughness, scaling, edges, holes, and background debris. SDNET2018 will be useful for the continued development of concrete crack detection algorithms based on deep convolutional neural networks (DCNNs), which are a subject of continued research in the field of structural health monitoring. The authors present benchmark results for crack detection using SDNET2018 and a crack detection algorithm based on the AlexNet DCNN architecture. SDNET2018 is freely available at https://doi.org/10.15142/T3TD19.
Article
Full-text available
Edge detection is a fundamental problem in computer vision. Recently, convolutional neural networks (CNNs) have pushed forward this field significantly. Existing methods which adopt specific layers of deep CNNs may fail to capture complex data structures caused by variations of scales and aspect ratios. In this paper, we propose an accurate edge detector using richer convolutional features (RCF). RCF encapsulates all convolutional features into more discriminative representation, which makes good usage of rich feature hierarchies, and is amenable to training via backpropagation. RCF fully exploits multiscale and multilevel information of objects to perform the image-to-image prediction holistically. Using VGG16 network, we achieve state-of-the-art performance on several available datasets. When evaluating on the well-known BSDS500 benchmark, we achieve ODS F-measure of 0.811 while retaining a fast speed (8 FPS). Besides, our fast version of RCF achieves ODS F-measure of 0.806 with 30 FPS. We also demonstrate the versatility of the proposed method by applying RCF edges for classical image segmentation.
Article
Full-text available
Over the past few years, several countries, including Spain, have been experiencing a period of economic recession. As a result, these governments have reduced their budgets for transport infrastructures (both construction and maintenance operations). The main objective of this study is to analyze whether these budget reductions have an effect on increased accident rates and to perform an assessment of their real economic benefit. Thus, we analyze whether significant changes over recent years are perceptible in the road safety indexes in Spain, in terms of risk, accident fatality, and accident severity. The relation between lower budgets and higher road safety indices is analyzed through linear regression techniques. The results show a strong relation between the Risk Index and the maintenance budget, measured as an average of the last years. In addition, a final economic assessment demonstrates that this reduction in investment had no real economic benefits, especially as the costs of the accidents exceeded the savings in the conservation plans.
Article
Full-text available
Crack detection is a crucial task in periodic pavement survey. This study establishes and compares the performance of two intelligent approaches for automatic recognition of pavement cracks. The first model relies on edge detection approaches of the Sobel and Canny algorithms. Since the implementation of the two edge detectors require the setting of threshold values, Differential Flower Pollination, as a metaheuristic, is employed to fine-tune the model parameters. The second model is constructed by the implementation of the Convolution Neural Network (CNN) – a deep learning algorithm. CNN has the advantage of performing the feature extraction and the prediction of crack/non-crack condition in an integrated and fully automated manner. Experimental results show that the model based on CNN achieves a good prediction performance of Classification Accuracy Rate (CAR) = 92.08%. This performance is significantly better than the method based on the edge detection algorithms (CAR = 79.99%). Accordingly, the proposed CNN based crack detection model is a promising alternative to support transportation agencies in the task of periodic pavement inspection.
Article
Full-text available
Research on damage detection of road surfaces using image processing techniques has been actively conducted. This study makes three contributions to address road damage detection issues. First, to the best of our knowledge, for the first time, a large‐scale road damage data set is prepared, comprising 9,053 road damage images captured using a smartphone installed on a car, with 15,435 instances of road surface damage included in these road images. Next, we used state‐of‐the‐art object detection methods using convolutional neural networks to train the damage detection model with our data set, and compared the accuracy and runtime speed on both, using a GPU server and a smartphone. Finally, we demonstrate that the type of damage can be classified into eight types with high accuracy by applying the proposed object detection method. The road damage data set, our experimental results, and the developed smartphone application used in this study are publicly available (https://github.com/sekilab/RoadDamageDetector/).
Conference Paper
Full-text available
Article
Automated pavement distress detection based on 2D images is facing various challenges. To efficiently complete the crack and pothole segmentation in a practical environment, an automated pixel-level pavement distress detection framework integrating stereo vision and deep learning is developed in this study. Based on the multi-view stereo imaging system, multi-feature pavement image datasets containing color images, depth images and color-depth overlapped images are established, providing a new perspective for deep learning. To alleviate computational burden, a modified U-net deep learning architecture introducing depthwise separable convolution is proposed for crack and pothole segmentation. These methods are tested in asphalt roads with different circumstances. The results show that the 3D pavement image achieves millimeter-level accuracy. The enhanced 3D crack segmentation model outperforms other models in terms of segmentation accuracy and inference speed. After obtaining the high-resolution pothole segmentation map, the automated pothole volume measurement is realized with high accuracy.
Conference Paper
Fast and accurate road damage detection is essential for the automatization of road inspection. This paper describes our solution submitted to the Global Road Damage Detection Challenge of the 2020 IEEE International Conference on Big Data, for typical road damage detection in digital images based on deep learning. The recently proposed YOLOv4 is chosen as the baseline network, while the effects of data augmentation, transfer learning, Optimized Anchors, and their combination are evaluated. We propose a novel road damage data generation method based on a generative adversarial network, which can generate multi-class samples with a single model. The evaluation results demonstrate the effectiveness of different tricks and their combinations on the road damage detection task, which provides a reference for practical application. The code of our solution is available at https://github.com/ZhangXG001/RoadDamgeDetection.git.
Conference Paper
This paper summarizes the Global Road Damage Detection Challenge (GRDDC), a Big Data Cup organized as a part of the IEEE International Conference on Big Data’2020. The Big Data Cup challenges involve a released dataset and a well-defined problem with clear evaluation metrics. The challenges run on a data competition platform that maintains a leaderboard for the participants. In the presented case, the data constitute 26336 road images collected from India, Japan, and the Czech Republic to propose methods for automatically detecting road damages in these countries. In total, 121 teams from several countries registered for this competition. The submitted solutions were evaluated using two datasets test1 and test2, comprising 2,631 and 2,664 images. This paper encapsulates the top 12 solutions proposed by these teams. The best performing model utilizes YOLO-based ensemble learning to yield an F1 score of 0.67 on test1 and 0.66 on test2. The paper concludes with a review of the facets that worked well for the presented challenge and those that could be improved in future challenges.
Article
Routine visual inspection is essential to maintain adequate safety and serviceability of civil infrastructures. Computer vision and machine learning based software techniques are becoming recognized methods that can potentially help the inspectors analyze the physical and functional condition of infrastructures from images and/or videos of the region of interest. More recently, deep learning approaches have been shown robust in identifying damages; yet these methods require precisely labeled large amount of training data for high accuracy complementary to visual assessment of inspectors. Especially in image segmentation operations, in which damages are subtracted from the image background for further analysis, there is a strong need to localize the damaged region prior to segmentation operation. However, available segmentation methods mostly focus on the latter step (i.e., delineation), and mis-localization of damaged regions causes accuracy drops. Inspired by the superiority of human cognitive system, where recognizing objects is simpler and more efficient than machine learning algorithms, which are superior to human in local tasks, this paper describes a novel method to dramatically improve the accuracy of the damage quantification (detection + segmentation) using an attention-guided technique. In the proposed method, a fast object detection model, Single Shot Detector (SSD) trained on VGG-16 base classifier architecture, performs a real-time crack and spall detection while working interactively with the human inspector to ensure recognition of the region of interest is well-performed. Upon the inspector's verification, happening in real-time, the detected damage region is used for damage segmentation for further analysis. This initial region of interest selection drastically lowers the computational cost, required amount of training data and reduces number of outliers. For optimal performance, a modified version of SegNet architecture was used for damage segmentation. Based on various performance criteria, the proposed attention-guided infrastructure damage analysis technique provides 30% more precision with a very minor sacrifice in computational speed compared to analysis without using attention guide.
Article
The condition of the road surface should be inspected to increase the service life of the road and to ensure safety and comfort. This study aims to automatically detect and measure road distress from unmanned aerial vehicle (UAV)-based images. The proposed methodology consists of three steps. First, images acquired from the UAV are used to generate the three-dimensional point cloud. Then, the road surface is extracted from the 3D point cloud. Finally, the developed algorithm is used to automatically detect and measure road distress. The accuracy assessment is conducted by comparing the analyses from point cloud data and measurements obtained from the traditional inspection method. The root mean square error values range from 2.09–6.72 cm. Finally, the outcomes of the proposed methodology are compared with those of commercial GIS software. Both produce statistically similar results for detecting road surface distress.