Article

Artificial Intelligence Assisted Infrastructure Assessment using Mixed Reality Systems

Authors:
  • Connected Wise
To read the full-text of this research, you can request a copy directly from the authors.

Abstract and Figures

Conventional methods for visual assessment of civil infrastructures have certain limitations, such as subjectivity of the collected data, long inspection time, and high cost of labor. Although some new technologies (i.e., robotic techniques) that are currently in practice can collect objective, quantified data, the inspector’s own expertise is still critical in many instances because these technologies are not designed to work interactively with a human inspector. This study aims to create a smart, human-centered method that offers significant contributions to infrastructure inspection, maintenance, management practice, and safety for the bridge owners. By developing a smart mixed reality (MR) framework, which can be integrated into a wearable holographic headset device, a bridge inspector, for example, can automatically analyze a certain defect such as a crack that he or she sees on an element, and display its dimension information in real-time along with the condition state. Such systems can potentially decrease the time and cost of infrastructure inspections by accelerating essential tasks of the inspector such as defect measurement, condition assessment, and data processing to management systems. The human-centered artificial intelligence (AI) will help the inspector collect more quantified and objective data while incorporating the inspector’s professional judgment. This study explains in detail the described system and related methodologies of implementing attention guided semisupervised deep learning into MR technology, which interacts with the human inspector during assessment. Thereby, the inspector and the AI will collaborate/communicate for improved visual inspection.
Content may be subject to copyright.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... In the application of CV-SHM, no matter LL or GL, the implementation of projective transform for camera calibration is always necessary. For CV-SHM-LL application, Karaaslan et al. 10 implemented CV to estimate the camera pose of a headset and determine the length/width/area of detected cracks on structures. Then they assessed the structural condition as ''Good, Fair, Poor or Severe'' according to AASHTO codes. ...
... The sizes of the detected regions can be different. Karaaslan et al. 10 retrained the VGG-16 network weights in Single Shot MultiBox Detector (SSD) architecture to recognize crack regions with a bounding box and applied the SegNet model to segment the cracks inside the bounding box. ...
... Also, they applied the Class Active Map (CAM) with heat map extracted from the retrained neural network to visualize the possible spalling areas in the images. Karaaslan et al. 10 first retrained the CNN architecture (SSD) for object detection purposes by using transfer learning to detect spalling in an image, and then proposed an attention guided segmentation network (SegNet) to segment spalling from concrete columns and walls. The guided segmentation with human aided operation does not need full image search and can improve the accuracy. ...
Article
Full-text available
Structural health monitoring at local and global levels using computer vision technologies has gained much attention in the structural health monitoring community in research and practice. Due to the computer vision technology application advantages such as non-contact, long distance, rapid, low cost and labor, and low interference to the daily operation of structures, it is promising to consider computer vision–structural health monitoring as a complement to the conventional structural health monitoring. This article presents a general overview of the concepts, approaches, and real-life practice of computer vision–structural health monitoring along with some relevant literature that is rapidly accumulating. The computer vision–structural health monitoring covered in this article at local level includes applications such as crack, spalling, delamination, rust, and loose bolt detection. At the global level, applications include displacement measurement, structural behavior analysis, vibration serviceability, modal identification, model updating, damage detection, cable force monitoring, load factor estimation, and structural identification using input–output information. The current research studies and applications of computer vision–structural health monitoring mainly focus on the implementation and integration of two-dimensional computer vision techniques to solve structural health monitoring problems and the projective geometry methods implemented are utilized to convert the three-dimensional problems into two-dimensional problems. This review mainly puts emphasis on two-dimensional computer vision–structural health monitoring applications. Subsequently, a brief review of representative developments of three-dimensional computer vision in the area of civil engineering is presented along with the challenges and opportunities of two-dimensional and three-dimensional computer vision–structural health monitoring. Finally, the article presents a forward look to the future of computer vision–structural health monitoring.
... Among these technologies, vision-based approaches are gathering increasing attention in the field of SHM (Dong and Catbas 2019;Ye et al. 2016a) due to the advantages such as non-contact, long distance, low cost, time saving and ease of use. Generally, the studies and practices of vision-based monitoring are divided into two aspects: 1) inspection and condition assessment at local level such as crack, spalling (Karaaslan et al. 2018) and delamination detection and 2) structural monitoring at global level such as vibration and deflection monitoring (Dong et al. , 2018b(Dong et al. , 2019bXu and Brownjohn 2018;Ye et al. 2013aYe et al. , 2015a, cable force monitoring (Feng et al. 2017;Ye et al. 2016c), modal analysis (Chen et al. 2018a;Hoskere et al. 2019;Yang et al. 2017), load estimation , load rating (Catbas et al. 2012) and load capacity estimation (Lee et al. 2006) etc. With vision-based inspection at local level, the condition assessment is carried out when damages already appear and are visible and large enough. ...
Preprint
Currently most of the vision-based structural identification research focus either on structural input (vehicle location) estimation or on structural output (structural displacement and strain responses) estimation. The structural condition assessment at global level just with the vision-based structural output cannot give a normalized response irrespective of the type and/or load configurations of the vehicles. Combining the vision-based structural input and the structural output from non-contact sensors overcomes the disadvantage given above, while reducing cost, time, labor force including cable wiring work. In conventional traffic monitoring, sometimes traffic closure is essential for bridge structures, which may cause other severe problems such as traffic jams and accidents. In this study, a completely non-contact structural identification system is proposed, and the system mainly targets the identification of bridge unit influence line (UIL) under operational traffic. Both the structural input (vehicle location information) and output (displacement responses) are obtained by only using cameras and computer vision techniques. Multiple cameras are synchronized by audio signal pattern recognition. The proposed system is verified with a laboratory experiment on a scaled bridge model under a small moving truck load and a field application on a footbridge on campus under a moving golf cart load. The UILs are successfully identified in both bridge cases. The pedestrian loads are also estimated with the extracted UIL and the predicted weights of pedestrians are observed to be in acceptable ranges.
... The smart contract grants permission to access considering the access methods and requirements. In [9], the authors have used AI in civil infrastructures to assess the visual quality of the constructions made. The authors proposed a smart mixed reality framework, by integrating it with a wearable device to detect the cracks. ...
Article
Full-text available
The emerging technologies trending up in information and communication technology are tuning the enterprises for betterment. The existing infrastructure and operations (I&O) are supporting enterprises with their services and functionalities, considering the diverse requirements of the end-users. However, they are not free of the challenges and issues to address as the technology has advanced. This paper explains the impact of artificial intelligence (AI) in the enterprises using software-defined networking (SDN) in I&O. The fusion of artificial intelligence with software-defined networking in infrastructure and operations enables to automate the process based on experience and provides opportunities to the management to make quick decisions. But this fusion has many challenges to be addressed. This research aimed to discuss the trends and challenges impacting infrastructure and operations, and the role of AI-enabled SDN in I&O and discusses the benefits it provides that influence the directional path. Furthermore, the challenges to be addressed in implementing the AI-enabled SDN in I&O shows future directions to explore.
... Among these technologies, vision-based approaches are gathering increasing attention in the field of SHM Catbas 2019, Ye et al. 2016a) due to the advantages such as non-contact, long distance, low cost, time saving and ease of use. Generally, the studies and practices of vision-based monitoring are divided into two aspects: 1) inspection and condition assessment at local level such as crack, spalling (Karaaslan et al. 2018) and delamination detection and 2) structural monitoring at global level such as vibration and deflection monitoring (Dong et al. , 2018b(Dong et al. , 2019bXu and Brownjohn 2018;Ye et al. 2013aYe et al. , 2015aYe et al. , 2016b), cable force monitoring (Feng et al. 2017;Ye et al. 2016c), modal analysis (Chen et al. 2018a, Hoskere et al. 2019, Yang et al. 2017, load estimation , load rating (Catbas et al. 2012) and load capacity estimation (Lee et al. 2006) etc. With vision-based inspection at local level, the condition assessment is carried out when damages already appear and are visible and large enough. ...
Article
Currently most of the vision-based structural identification research focus either on structural input (vehicle location) estimation or on structural output (structural displacement and strain responses) estimation. The structural condition assessment at global level just with the vision-based structural output cannot give a normalized response irrespective of the type and/or load configurations of the vehicles. Combining the vision-based structural input and the structural output from non-contact sensors overcomes the disadvantage given above, while reducing cost, time, labor force including cable wiring work. In conventional traffic monitoring, sometimes traffic closure is essential for bridge structures, which may cause other severe problems such as traffic jams and accidents. In this study, a completely non-contact structural identification system is proposed, and the system mainly targets the identification of bridge unit influence line (UIL) under operational traffic. Both the structural input (vehicle location information) and output (displacement responses) are obtained by only using cameras and computer vision techniques. Multiple cameras are synchronized by audio signal pattern recognition. The proposed system is verified with a laboratory experiment on a scaled bridge model under a small moving truck load and a field application on a footbridge on campus under a moving golf cart load. The UILs are successfully identified in both bridge cases. The pedestrian loads are also estimated with the extracted UIL and the predicted weights of pedestrians are observed to be in acceptable ranges.
... In a related work, a framework for MR enabled in-133 spection system using AR/MR devices was recently 134 presented in [41]. The authors demonstrate that is utilized to segment the damage region on an im-163 age and refine the results interactively through user 164 feedback. ...
Article
Full-text available
In this study, a new visual inspection method that can interactively detect and quantify structural defects using an Extended Reality (XR) device (headset) is proposed. The XR device, which is at the core of this method, supports an interactive environment using a holographic overlay of graphical information on the spatial environment and physical objects being inspected. By leveraging this capability, a novel XR-supported inspection pipeline, called eXtended Reality-based Inspection and Visualization (XRIV), is developed. Key tasks supported by this method include detecting visual damage from sensory data acquired by the XR device, estimating its size, and visualizing (overlaying) information on the spatial environment. The crucial step of real-time interactive segmentation—detection and pixel-wise damage boundary refinement—is achieved using a feature Back-propagating Refinement Scheme (f-BRS) algorithm. Then, a ray-casting algorithm is applied to back-project the 2D image pixel coordinates of the damage region to their 3D world coordinates for damage area quantification in real-world (physical) units. Finally, the area information is overlaid and anchored to the scene containing damage for visualization and documentation. The performance of XRIV is experimentally demonstrated by measuring surface structural damage of an in-service concrete bridge with less than 10% errors for two different test cases, and image processing latency of 2–3 s (or 0.5 s per seed point) from f-BRS. The proposed XRIV pipeline underscores the advantages of real-time interaction between expert users and the XR device through immersive visualization so that a human–machine collaborative workflow can be established to obtain better inspection outcomes in terms of accuracy and robustness.
... Stankov U et al. emphasize the establishment of a new relationship between brands and audiences through advertising that conveys product messages and is liked by the advertising audience [9]. e contextual advertising mentioned in the book involves outdoor media forms, citing a large number of outdoor advertising cases, using the spatial characteristics of outdoor media coupled with unique advertising creativity to make advertising a public space art to enhance the interactive experience with consumers, which plays an important role in enhancing brand image [10]. In the study of interactive advertising, if interactive advertising appears as a single form of advertising model, it often faces the problems of the small audience, inability to spread publicity, and low-value conversion due to the limitation of the base; then, in response to these problems, it is precisely ordinary advertising that can fill them. ...
Article
Full-text available
Along with the development and promotion of Internet technology, new media are increasingly diversified, enriching and changing our lives. This paper focuses on outdoor interactive advertising as a communication method, analyzing its creative features and applications. Unlike traditional advertising, where audiences receive passively, new media interactive advertising establishes a more direct communication method for consumers and products, creating different sensory stimulation and emotional experience environments according to the characteristics of products and conducting a series of interactions with consumers’ sensory systems such as vision, hearing, smell, taste, and touch. The advertising method has also changed from passive reception to active participation. Finally, combining the characteristics of accurate delivery and the diversity of presentation forms of interactive advertising under the big data platform, the theme conception and form design of interactive advertising is proposed. The interactive design principles of highlighting the simplicity of the theme are explored. The interactive design principle of highlighting the simplicity of the theme is proposed, and the combination of color, text, and graphics is explored by the aesthetic visual design of the audience and the premise of focusing on the audience’s feelings. Starting from the visual effect of outdoor interactive advertising, the change in the way advertising is conveyed is interpreting to us the arrival of a new communication concept, the arrival of the Internet-centered and technology-centered data integration era. The research in this paper aims to provide useful support for the diversified development of outdoor interactive advertising in the new media environment by exploring how to skillfully realize the creative expression of outdoor interactive advertising in the new media environment.
... Hololens is a wearable device that can be used to inspect bridges in the field and office. Hololens applications have been developed for onsite bridge inspection (Karaaslan et al., 2019;Moreu et al., 2017). Users can automatically detect some types of defects such as cracking and spalling with dimension information in real-time by Hololens. ...
Article
Full-text available
Purpose The purpose of this study is to develop a building information modelling (BIM)-based mixed reality (MR) application to enhance and facilitate the process of managing bridge inspection and maintenance works remotely from office. It aims to address the ineffective decision-making process on maintenance tasks from the conventional method which relies on documents and 2D drawings on visual inspection. This study targets two key issues: creating a BIM-based model for bridge inspection and maintenance; and developing this model in a MR platform based on Microsoft Hololens. Design/methodology/approach Literature review is conducted to determine the limitation of MR technology in the construction industry and identify the gaps of integration of BIM and MR for bridge inspection works. A new framework for a greater adoption of integrated BIM and Hololens is proposed. It consists of a bridge information model for inspection and a newly-developed Hololens application named “HoloBridge”. This application contains the functional modules that allow users to check and update the progress of inspection and maintenance. The application has been implemented for an existing bridge in South Korea as the case study. Findings The results from pilot implementation show that the inspection information management can be enhanced because the inspection database can be systematically captured, stored and managed through BIM-based models. The inspection information in MR environment has been improved in interpretation, visualization and visual interpretation of 3D models because of intuitively interactive in real-time simulation. Originality/value The proposed framework through “HoloBridge” application explores the potential of integrating BIM and MR technology by using Hololens. It provides new possibilities for remote inspection of bridge conditions.
... This use case often serves as a precursor to the following use cases -product quality control -predictive maintenance. Examples include various applications developed to inspect the structural integrity of bridges or other concrete structures (Karaaslan et al. 2018;Spencer et al. 2019) as well as Bühler's equipment monitoring system (Bühler 2021). • Product quality control: Automated assessment of the quality of created products in real time. ...
Chapter
With a growing number of connected devices producing exponentially more data, the value of artificial intelligence and machine learning (AI/ML) is increasing rapidly for businesses. We outline the value add of AI/ML for decision-making in firms and present use cases and tools to generate data-driven value. We discuss various implementation challenges and solution approaches. Successfully executing on AI/ML applications hinges on preparing the company, managing the portfolio of projects, using interdisciplinary teams, establishing strong technical foundations, and, importantly, generating trust in AI/ML throughout the system lifecycle.
... These models have shown breakthrough performance in especially image classification tasks (e.g., 2012 ImageNet Competition) [39]. CNN-based image analysis of infrastructure damage has been vastly studied in the past [40,41]. As shown in Figure 9, the CNN models are typically composed of convolution, activation, and pooling layers. ...
Article
Full-text available
Developing a bridge management strategy at the network level with efficient use of capital is very important for optimal infrastructure remediation. This paper introduces a novel decision support system that considers many aspects of bridge management and successfully implements the investigated methodology in a web-based platform. The proposed decision support system uses advanced prediction models, decision trees, and incremental machine learning algorithms to generate an optimal decision strategy. The system aims to achieve adaptive and flexible decision making while entailing powerful utilization of nondestructive evaluation (NDE) methods. The NDE data integration and visualization allow automatic retrieval of inspection results and overlaying the defects on a 3D bridge model. Furthermore, a deep learning-based damage growth prediction model estimates the future condition of the bridge elements and utilizes this information in the decision-making process. The decision ranking takes into account a wide range of factors including structural safety, serviceability, rehabilitation cost, life cycle cost, and societal and political factors to generate optimal maintenance strategies with multiple decision alternatives. This study aims to bring a complementary solution to currently in-use systems with the utilization of advanced machine-learning models and NDE data integration while still equipped with main bridge management functions of bridge management systems and capable of transferring data to other systems.
... However, there are examples for crack detection in concretes using the abovementioned strategy. Karaaslan et al. [28] applied the single shot multibox detector (SSD) algorithm to detect the crack regions on the surface of concrete structures and then applied a semantic segmentation algorithm to segment the cracks at pixel level. ...
Article
Full-text available
Fatigue cracks are critical types of damage in steel structures due to repeated loads and distortion effects. Fatigue crack growth may lead to further structural failure and even induce collapse. Efficient and timely fatigue crack detection and segmentation can support condition assessment, asset maintenance, and management of existing structures and prevent the early permit post and improve life cycles. In current research and engineering practices, visual inspection is the most widely implemented approach for fatigue crack inspection. However, the inspection accuracy of this method highly relies on the subjective judgment of the inspectors. Furthermore, it needs large amounts of cost, time, and labor force. Non-destructive testing methods can provide accurate detection results, but the cost is very high. To overcome the limitations of current fatigue crack detection methods, this study presents a pixel-level fatigue crack segmentation framework for large-scale images with complicated backgrounds taken from steel structures by using an encoder-decoder network, which is modified from the U-net structure. To effectively train and test the images with large resolutions such as 4928 × 3264 pixels or larger, the large images were cropped into small images for training and testing. The final segmentation results of the original images are obtained by assembling the segment results in the small images. Additionally, image post-processing including opening and closing operations were implemented to reduce the noises in the segmentation maps. The proposed method achieved an acceptable accuracy of automatic fatigue crack segmentation in terms of average intersection over union (mIOU). A comparative study with an FCN model that implements ResNet34 as backbone indicates that the proposed method using U-net could give better fatigue crack segmentation performance with fewer training epochs and simpler model structure. Furthermore, this study also provides helpful considerations and recommendations for researchers and practitioners in civil infrastructure engineering to apply image-based fatigue crack detection.
... Many possible ways of using AR for infrastructure management have also been proposed in the past. For example, the inspector can mark the points to be inspected during the inspection in the AR model and then use it during the inspection in the real world, reducing the time spent on the inspection site [44]; an AR interface allows the inspector to capture the images of damaged members can be used to generate damage profile after image processing [45]; annotation (area markings, crack markings, etc.,) made on the images can be transferred to the AR interface for further analysis [46]; and an AR interface with semi-supervised machine learning algorithms can support on-site inspectors identify the severity of cracks [47]. ...
Article
Full-text available
Bridges are indispensable links of transportation infrastructure systems, and inspections play a critical role in maintaining bridge components in the state of good repair. Through a survey of bridge inspectors, the authors revealed that visual inspection techniques are the prominent inspection method but result in inaccuracy and ambiguity due to high variances among inspection results; modern inspections using drones and robots could improve efficiency but pose new challenges and do not reduce subjectivity. As a result, a novel, building information modeling-and augmented reality-based supportive inspection system (BASIS) that objectively captures bridge defects is proposed and validated. On-site inspectors can access the bridge model containing historical defect information (defect type, length/width/depth, and location) and overlay relevant content on the actual infrastructure through BASIS for inspection data collection with more accuracy and less ambiguity. A proof-of-concept prototype of the BASIS for bridges was developed as an android application and verified by bridge inspectors for effectiveness on a small pedestrian bridge. It was found that BASIS was able to collect accurate inspection data irrespective of the level of experience of the user, thusly minimizing the data subjectivity caused by differences among inspectors' judgment and/or human errors. This research explores the utilization of emerging tools to collect bridge condition information in a more comprehensive and objective manner. Collected information can be further integrated it into a digital model that reflects the bridge's most accurate and up-to-date condition, heading toward a digital twin of the physical infrastructure. The proposed system may also be adapted for other types of infrastructure (e.g., dams, levees, and railroads) that also require routine inspections.
... Hololens is a wearable device that can be used to inspect bridges in the field and office. Previous researches have focused on developing the Hololens application for onsite bridge inspection (Karaaslan et al., 2019;Moreu et al., 2017). By using Hololens, users can automatically detect some types of defects such as crack, spalling with dimension information in real-time. ...
Conference Paper
Full-text available
Bridge inspection, which collects data for assessment and decision-making processes, plays an important role in the maintenance job of bridge structures. However, all of the inspection jobs, including general inspection, principal inspection and special inspection, generate large and unstructured data resulting in ineffective maintenance. Besides, traditional inspection generates only 2D-based and less visionary information. For a more reliable assessment of structures, it is necessary to improve the current inspection technology and process. In recent years, mixed reality (MR) technology has been proved to be effective in improving interaction, communication and collaboration among stakeholders for evaluating the database. MR and holographic technology blend 3D models with physical assets and support users to engage in the models and interact with the project data more intuitively in the real-time simulation. This paper presents an application of MR-based system called HoloBridge to enhance and facilitate bridge inspection and maintenance. The application consists of modules of inspection, evaluation, and damage mapping. The HoloBridge application is being deployed to Microsoft Hololens for tracking and assessing conditions of bridges. The application has been developed by building information modelling (BIM)-based system linked and integrated into a cross-platform game engine to evaluate bridge damage information. The application is piloted with a highway bridge in South Korea and has shown the benefits not only in the digitalized inspection processes, but also in systematic managing bridge performance.
... The smart contract grants permission to access considering the access methods and requirements. In [9], the authors have used AI in civil infrastructures to assess the visual quality of the constructions made. The authors proposed a smart mixed reality framework, by integrating it with a wearable device to detect the cracks. ...
Article
Full-text available
Abstract: The emerging technologies trending up in information and communication technology are tuning the enterprises for betterment. The existing infrastructure and operations (I&O) are supporting enterprises with their services and functionalities, considering the diverse requirements of the end-users. However, they are not free of the challenges and issues to address as the technology has advanced. This paper explains the impact of artificial intelligence (AI) in the enterprises using software-defined networking (SDN) in I&O. The fusion of artificial intelligence with software-defined networking in infrastructure and operations enables to automate the process based on experience and provides opportunities to the management to make quick decisions. But this fusion has many challenges to be addressed. This research aimed to discuss the trends and challenges impacting infrastructure and operations, and the role of AI-enabled SDN in I&O and discusses the benefits it provides that influence the directional path. Furthermore, the challenges to be addressed in implementing the AI-enabled SDN in I&O shows future directions to explore.
Chapter
Full-text available
Vehicle-to-infrastructure (V2I) technologies are known as the next generation of intelligent transportation systems (ITS), enabling data transmission between vehicles and road infrastructure. V2I communication models offer a variety of safety, mobility, and environmental applications in smart cities. Another innovative concept within the ITS is the human-to-infrastructure (H2I) communication model. H2I deals with the vulnerability of the users to damages and deteriorations of civil infrastructure. While many studies deal with V2I, limited research has been conducted in the area of interaction between humans and transportation infrastructure systems. This study presents a comprehensive review of the current studies in the arena of V2I and proposes the vital basics for establishing a robust H2I model for the next generation of the smart, civil infrastructure systems.
Article
Bridges are critical infrastructure, and effective operation and maintenance (O&M) is essential for ensuring the good condition of bridges. Owing to the increasing complexity of modern bridges and the availability of information technologies (e.g. sensors, laser scanners, and ultrasonic radar) for the collection of massive data, bridge O&M and decision-making gradually shift toward a data-driven manner. However, both the bridge industry and the academia still do not have a common understanding of the latest progress, challenges, and trends of data-driven bridge O&M decision-making. Thus, through a critical review of 485 articles, this paper investigates current data-driven bridge O&M decision-making in detail, including mainstream data types, issues related to data management, and typical application areas using these data. Challenges to implement data-driven bridge O&M decision-making are identified, such as lack of standard data needs, lack of data integration, and lack of standard procedures. Future research opportunities to address the challenges are proposed. This paper can help bridge O&M teams by identifying suitable data and applications to make informed decisions that align well with their needs meanwhile serve as a basis for future research efforts in this area.
Article
Information technologies have widely been used in the construction and maintenance of civil infrastructure. The advantages of information technologies provided a broader range of methods for infrastructure and enhanced its level of maintenance. However, a systematic summary of the research development of information technologies used in civil infrastructure is limited. This study aims to supplement this field by providing an objective, systematic summary of relevant literature in mainstream journals employing bibliometric retrieval and quantitative analysis from 2010 to 2020. The following results are obtained: (1) This study discusses the application of advanced information technologies in different phases and provides a critical analysis of the application of these existing information technologies, which includes wireless sensor networks (WSN), fiber optic sensing (FOS), building information modelling (BIM), radio frequency identification (RFID) and other advanced information technologies. (2) The digital twins can be used as tools for the planning and management of next-generation smart infrastructure, making the future of civil infrastructure smarter and more sustainable.
Article
Routine visual inspection is essential to maintain adequate safety and serviceability of civil infrastructures. Computer vision and machine learning based software techniques are becoming recognized methods that can potentially help the inspectors analyze the physical and functional condition of infrastructures from images and/or videos of the region of interest. More recently, deep learning approaches have been shown robust in identifying damages; yet these methods require precisely labeled large amount of training data for high accuracy complementary to visual assessment of inspectors. Especially in image segmentation operations, in which damages are subtracted from the image background for further analysis, there is a strong need to localize the damaged region prior to segmentation operation. However, available segmentation methods mostly focus on the latter step (i.e., delineation), and mis-localization of damaged regions causes accuracy drops. Inspired by the superiority of human cognitive system, where recognizing objects is simpler and more efficient than machine learning algorithms, which are superior to human in local tasks, this paper describes a novel method to dramatically improve the accuracy of the damage quantification (detection + segmentation) using an attention-guided technique. In the proposed method, a fast object detection model, Single Shot Detector (SSD) trained on VGG-16 base classifier architecture, performs a real-time crack and spall detection while working interactively with the human inspector to ensure recognition of the region of interest is well-performed. Upon the inspector's verification, happening in real-time, the detected damage region is used for damage segmentation for further analysis. This initial region of interest selection drastically lowers the computational cost, required amount of training data and reduces number of outliers. For optimal performance, a modified version of SegNet architecture was used for damage segmentation. Based on various performance criteria, the proposed attention-guided infrastructure damage analysis technique provides 30% more precision with a very minor sacrifice in computational speed compared to analysis without using attention guide.
Conference Paper
Full-text available
This paper details the novel combination of a population-based swarm intelligence algorithm, structure mechanics, geotechnical variability, and a specific construction staging methodology, to propose an autonomous construction system which can erect a bridge over arbitrary crossing geometry while satisfying project constraints and conventional design code requirements (the Proposal). The fields of metaheuristic applications in structures, unmanned aerial systems (UAS), and additive manufacturing are each independent innovations in the transportation industry. The Proposal brings these fields together, with a vision that independent drone robots with very basic programming could construct a bridge with no user interruption and without performing a conventional structural design. A simulation as proof-of-concept is compared to a modern bridge design; the result is a modest increase in materials with an anticipated dramatic savings in labor. The paper also studies the effects of geotechnical variability for insight into preparation for possible future construction methodologies and algorithms. The Proposal has promise where socio-economic issues may be a factor, such as construction of infrastructure in remote locations or developing nations, and emergency repair or replacement of bridges. With recent U.S. policy supporting a permanent lunar presence and expanding Moon to Mars exploration through the Artemis program, the Proposal may also assist infrastructure development in advance of human missions. This paper available at: http://www.trb.org/Publications/Blurbs/182304.aspx
Conference Paper
Full-text available
The nonlinear behavior of shallow bridge and building foundations under large-amplitude loading is an important aspect of performance-based earthquake engineering (PBEE). Soil yielding beneath foundations can be an effective energy dissipation mechanism; however, this yielding may lead to excessive permanent deformations. The research and engineering community have established decades of model testing and analytical modeling to understand foundation nonlinearity and uplift. Physical model testing provides insight into the mechanisms at work in rocking, shallow foundations, while analytical modeling and simulation are validated against these experiment results such that modeling recommendations can be applied in practice. This body of work lead to the current design procedures in industry, such that the benefits and consequences of foundations allowed to rock under seismic events can be understood and implemented in practice generally for retrofit of existing structures. While there are various established analytical modeling approaches, this paper focuses specifically on both the elastic and nonlinear Winkler analytical modeling approaches used to better understand, mimic or approximate observations from experiment, current codified design procedures, and includes speculation on the future of research and design in this field. download link: http://www.trb.org/Publications/Blurbs/182304.aspx
Chapter
Full-text available
Smart infrastructures aim more efficient and accurate methods of routine inspection for long-term monitoring of the infrastructure to make smarter decision on maintenance and rehabilitation. Although some recent technologies (i.e., robotic techniques) that are currently in practice can collect objective, quantified data, the inspector’s own expertise is still critical in many instances. Yet, these technologies are designed to replace human expertise, or are ineffective in terms of saving time and labor. This chapter investigates a new methodology for structural inspections with the help of mixed reality technology and real-time machine learning to accelerate certain tasks of the inspector such as detection, measurement, and assessment of defects, and easy accessibility to defect locations. A functional, real-time machine learning system that can be ideally deployed in mixed reality devices and headsets which can be used by inspectors during their routine concrete infrastructure inspection is introduced. The deep learning models to be employed in the AI system can localize a concrete defect in real time and further analyze it by performing pixel wise segmentation while running on a mobile device architecture. First, a sufficiently large database of concrete defect images is gathered from various sources including publicly available crack and spalling datasets, real-world images taken during bridge inspections, and the public images from the internet search results. For defect localization, various state-of-the-art deep learning model architectures are investigated based on their memory allocation, inference speed, and flexibility to deploy different deep learning platforms. YoloV5s model was found to be the optimal model architecture for concrete defect localization to be deployed in the mixed reality system. For defect quantification, several segmentation architectures with three different classification backbones are trained on the collected image dataset with segmentation labels. Based on the model evaluation results, the PSPNet with EfficientNet-b0 backbone is found to be the best performing model in terms of inference speed and accuracy. The selected models for defect localization and quantification are deployed to the mixed reality platform and image tracking libraries are configured in the platform environment, and accurate distance estimation is accomplished using a calibration process. Lastly, a methodology for condition assessment of concrete defects using the mixed reality system is discussed. The proposed methodology can locate and track the defects using the mixed reality platform, which can eventually be transferred to cloud data and potentially used for remote assessments or updating a digital twins or BIMs.
Chapter
The current study aims to understand the reasons for bridge failures. It considers the recent damages and collapses of in-service real bridge structures and discusses mitigation methods based on monitoring technologies. Transportation systems serve a crucial function in the strategies for mitigating bridge damages and failures. After investigating recent damages and failures in real bridges, structural bridge failures are classified in this study according to their safety and operational function. For each function, global and local level failures are also defined with two groups: major/long-span and highway bridges. In the light of this classification, structural monitoring methods are identified according to global and local failures. Along with standard Structural Health Monitoring (SHM) systems with comprehensive sensor networks, at vision-based SHM system and developments in this area are shown to provide some opportunities for mitigation of bridge failures resulting from service loads rather than natural hazard-induced failures.
Article
Full-text available
Convolutional neural networks (CNNs) have shown remarkable results over the last several years for a wide range of computer vision tasks. A new architecture recently introduced by Sabour et al., referred to as a capsule networks with dynamic routing, has shown great initial results for digit recognition and small image classification. The success of capsule networks lies in their ability to preserve more information about the input by replacing max-pooling layers with convolutional strides and dynamic routing, allowing for preservation of part-whole relationships in the data. This preservation of the input is demonstrated by reconstructing the input from the output capsule vectors. Our work expands the use of capsule networks to the task of object segmentation for the first time in the literature. We extend the idea of convolutional capsules with locally-connected routing and propose the concept of deconvolutional capsules. Further, we extend the masked reconstruction to reconstruct the positive input class. The proposed convolutional-deconvolutional capsule network, called SegCaps, shows strong results for the task of object segmentation with substantial decrease in parameter space. As an example application, we applied the proposed SegCaps to segment pathological lungs from low dose CT scans and compared its accuracy and efficiency with other U-Net-based architectures. SegCaps is able to handle large image sizes (512 x 512) as opposed to baseline capsules (typically less than 32 x 32). The proposed SegCaps reduced the number of parameters of U-Net architecture by 95.4% while still providing a better segmentation accuracy.
Article
Full-text available
Research on damage detection of road surfaces using image processing techniques has been actively conducted, achieving considerably high detection accuracies. Many studies only focus on the detection of the presence or absence of damage. However, in a real-world scenario, when the road managers from a governing body need to repair such damage, they need to clearly understand the type of damage in order to take effective action. In addition, in many of these previous studies, the researchers acquire their own data using different methods. Hence, there is no uniform road damage dataset available openly, leading to the absence of a benchmark for road damage detection. This study makes three contributions to address these issues. First, to the best of our knowledge, for the first time, a large-scale road damage dataset is prepared. This dataset is composed of 9,053 road damage images captured with a smartphone installed on a car, with 15,435 instances of road surface damage included in these road images. In order to generate this dataset, we cooperated with 7 municipalities in Japan and acquired road images for more than 40 hours. These images were captured in a wide variety of weather and illuminance conditions. In each image, we annotated the bounding box representing the location and type of damage. Next, we used a state-of-the-art object detection method using convolutional neural networks to train the damage detection model with our dataset, and compared the accuracy and runtime speed on both, using a GPU server and a smartphone. Finally, we demonstrate that the type of damage can be classified into eight types with high accuracy by applying the proposed object detection method. The road damage dataset, our experimental results, and the developed smartphone application used in this study are publicly available (https://github.com/sekilab/RoadDamageDetector/).
Thesis
Full-text available
Deterioration of road infrastructure arises from aging and various other factors. Consequently, inspection and maintenance have been a serious worldwide problem. In the United States, degradation of concrete bridge decks is a widespread problem among several bridge components. In order to prevent the impending degradation of bridges, periodic inspection and proper maintenance are indispensable. However, the transportation system faces unprecedented challenges because the number of aging bridges is increasing under limited resources, both in terms of budget and personnel. Therefore, innovative technologies and processes that enable bridge owners to inspect and evaluate bridge conditions more effectively and efficiently with less human and monetary resources are desired. Traditionally, qualified engineers and inspectors implemented hammer sounding and/or chain drag, and visual inspection for concrete bridge deck evaluations, but these methods require substantial field labor, experience, and lane closures for bridge deck inspections. Under these circumstances, Non-Destructive Evaluation (NDE) techniques such as computer vision-based crack detection, impact echo (IE), ground-penetrating radar (GPR) and infrared thermography (IRT) have been developed to inspect and monitor aging and deteriorating structures rapidly and effectively. However, no single method can detect all kinds of defects in concrete structures as well as the traditional inspection combination of visual and sounding inspections; hence, there is still no international standard NDE methods for concrete bridges, although significant progress has been made up to the present. This research presents the potential to reduce a burden of bridge inspections, especially for bridge decks, in place of traditional chain drag and hammer sounding methods by IRT with the combination of computer vision-based technology. However, there were still several challenges and uncertainties in using IRT for bridge inspections. This study revealed those challenges and uncertainties, and explored those solutions, proper methods and ideal conditions for applying IRT in order to enhance the usability, reliability and accuracy of IRT for concrete bridge inspections. Throughout the study, detailed investigations of IRT are presented. Firstly, three different types of infrared (IR) cameras were compared under active IRT conditions in the laboratory to examine the effect of photography angle on IRT along with the specifications of cameras. The results showed that when IR images are taken from a certain angle, each camera shows different temperature readings. However, since each IR camera can capture temperature differences between sound and delaminated areas, they have a potential to detect delaminated areas under a given condition in spite of camera specifications even when they are utilized from a certain angle. Furthermore, a more objective data analysis method than just comparing IR images was explored to assess IR data. Secondly, coupled structural mechanics and heat transfer models of concrete blocks with artificial delaminations used for a field test were developed and analyzed to explore sensitive parameters for effective utilization of IRT. After these finite element (FE) models were validated, critical parameters and factors of delamination detectability such as the size of delamination (area, thickness and volume), ambient temperature and sun loading condition (different season), and the depth of delamination from the surface were explored. This study presents that the area of delamination is much more influential in the detectability of IRT than thickness and volume. It is also found that there is no significant difference depending on the season when IRT is employed. Then, FE model simulations were used to obtain the temperature differences between sound and delaminated areas in order to process IR data. By using this method, delaminated areas of concrete slabs could be detected more objectively than by judging the color contrast of IR images. However, it was also found that the boundary condition affects the accuracy of this method, and the effect varies depending on the data collection time. Even though there are some limitations, integrated use of FE model simulation with IRT showed that the combination can be reduce other pre-tests on bridges, reduce the need to have access to the bridge and also can help automate the IRT data analysis process for concrete bridge deck inspections. After that, the favorable time windows for concrete bridge deck inspections by IRT were explored through field experiment and FE model simulations. Based on the numerical simulations and experimental IRT results, higher temperature differences in the day were observed from both results around noontime and nighttime, although IRT is affected by sun loading during the daytime heating cycle resulting in possible misdetections. Furthermore, the numerical simulations show that the maximum effect occurs at night during the nighttime cooling cycle, and the temperature difference decreases gradually from that time to a few hours after sunrise of the next day. Thus, it can be concluded that the nighttime application of IRT is the most suitable time window for bridge decks. Furthermore, three IR cameras with different specifications were compared to explore several factors affecting the utilization of IRT in regards to subsurface damage detection in concrete structures, specifically when the IRT is utilized for high-speed bridge deck inspections at normal driving speeds under field laboratory conditions. The results show that IRT can detect up to 2.54 cm delamination from the concrete surface at any time period. This study revealed two important factors of camera specifications for high-speed inspection by IRT as shorter integration time and higher pixel resolution. Finally, a real bridge was scanned by three different types of IR cameras and the results were compared with other NDE technologies that were implemented by other researchers on the same bridge. When compared at fully documented locations with 8 concrete cores, a high-end IR camera with cooled detector distinguished sound and delaminated areas accurately. Furthermore, indicated location and shape of delaminations by three IR cameras were compared to other NDE methods from past research, and the result revealed that the cooled camera showed almost identical shapes to other NDE methods including chain drag. It should be noted that the data were collected at normal driving speed without any lane closures, making it a more practical and faster method than other NDE technologies. It was also presented that the factor most likely to affect high-speed application is integration time of IR camera as well as the conclusion of the field laboratory test. The notable contribution of this study for the improvement of IRT is that this study revealed the preferable conditions for IRT, specifically for high-speed scanning of concrete bridge decks. This study shows that IRT implementation under normal driving speeds has high potential to evaluate concrete bridge decks accurately without any lane closures much more quickly than other NDE methods, if a cooled camera equipped with higher pixel resolution is used during nighttime. Despite some limitations of IRT, the data collection speed is a great advantage for periodic bridge inspections compared to other NDE methods. Moreover, there is a high possibility to reduce inspection time, labor and budget drastically if high-speed bridge deck scanning by the combination of IRT and computer vision-based technology becomes a standard bridge deck inspection method. Therefore, the author recommends combined application of the high-speed scanning combination and other NDE methods to optimize bridge deck inspections.
Article
Full-text available
New technology that intelligently combines the physical and digital worlds looks set to revolutionise the way civil engineers monitor infrastructure, both during and after construction. Ioannis Brilakis of the University of Cambridge reports.
Article
Full-text available
This study is to develop a detector that automatically detects cracks from the photographs of concrete structures, using convolution neural network which is a kind of deep learning. Firstly, photographs of concrete were collected for the learning data. Secondly, pictures of cracked part, chalk letter part, joint part, surface part and others part were produced from these photographs for the dataset. Thirdly, classifier to classify into these 5 class from pictures was created using the dataset and convolution neural network. Finally, the automatic detector was produced using this classifier.
Article
Full-text available
Deep learning is a subfield of machine learning, which aims to learn a hierarchy of features from input data. Nowadays, researchers have intensively investigated deep learning algorithms for solving challenging problems in many areas such as image classification, speech recognition, signal processing, and natural language processing. In this study, we not only review typical deep learning algorithms in computer vision and signal processing but also provide detailed information on how to apply deep learning to specific areas such as road crack detection, fault diagnosis, and human activity detection. Besides, this study also discusses the challenges of designing and training deep neural networks.
Conference Paper
Full-text available
We present a method for detecting objects in images using a single deep neural network. Our approach, named SSD, discretizes the output space of bounding boxes into a set of default boxes over different aspect ratios and scales per feature map location. At prediction time, the network generates scores for the presence of each object category in each default box and produces adjustments to the box to better match the object shape. Additionally, the network combines predictions from multiple feature maps with different resolutions to naturally handle objects of various sizes. SSD is simple relative to methods that require object proposals because it completely eliminates proposal generation and subsequent pixel or feature resampling stages and encapsulates all computation in a single network. This makes SSD easy to train and straightforward to integrate into systems that require a detection component. Experimental results on the PASCAL VOC, COCO, and ILSVRC datasets confirm that SSD has competitive accuracy to methods that utilize an additional object proposal step and is much faster, while providing a unified framework for both training and inference. For \(300 \times 300\) input, SSD achieves 74.3 % mAP on VOC2007 test at 59 FPS on a Nvidia Titan X and for \(512 \times 512\) input, SSD achieves 76.9 % mAP, outperforming a comparable state of the art Faster R-CNN model. Compared to other single stage methods, SSD has much better accuracy even with a smaller input image size. Code is available at https:// github. com/ weiliu89/ caffe/ tree/ ssd.
Article
Full-text available
There is a need for rapid and objective assessment of concrete bridge decks for maintenance decision making. Infrared Thermography (IRT) has great potential to identify deck delaminations more objectively than routine visual inspections or chain drag tests. In addition, it is possible to collect reliable data rapidly with appropriate IRT cameras attached to vehicles and the data are analyzed effectively. This research compares three infrared cameras with different specifications at different times and speeds for data collection, and explores several factors affecting the utilization of IRT in regards to subsurface damage detection in concrete structures, specifically when the IRT is utilized for high-speed bridge deck inspection at normal driving speeds. These results show that IRT can detect up to 2.54 cm delamination from the concrete surface at any time period. It is observed that nighttime would be the most suitable time frame with less false detections and interferences from the sunlight and less adverse effect due to direct sunlight, making more " noise " for the IRT results. This study also revealed two important factors of camera specifications for high-speed inspection by IRT as shorter integration time and higher pixel resolution.
Article
Full-text available
We present a novel and practical deep fully convolutional neural network architecture for semantic pixel-wise segmentation termed SegNet. This core trainable segmentation engine consists of an encoder network, a corresponding decoder network followed by a pixel-wise classification layer. The architecture of the encoder network is topologically identical to the 13 convolutional layers in the VGG16 network . The role of the decoder network is to map the low resolution encoder feature maps to full input resolution feature maps for pixel-wise classification. The novelty of SegNet lies is in the manner in which the decoder upsamples its lower resolution input feature map(s). Specifically, the decoder uses pooling indices computed in the max-pooling step of the corresponding encoder to perform non-linear upsampling. This eliminates the need for learning to upsample. The upsampled maps are sparse and are then convolved with trainable filters to produce dense feature maps. We compare our proposed architecture with the fully convolutional network (FCN) architecture and its variants. This comparison reveals the memory versus accuracy trade-off involved in achieving good segmentation performance. The design of SegNet was primarily motivated by road scene understanding applications. Hence, it is efficient both in terms of memory and computational time during inference. It is also significantly smaller in the number of trainable parameters than competing architectures and can be trained end-to-end using stochastic gradient descent. We also benchmark the performance of SegNet on Pascal VOC12 salient object segmentation and the recent SUN RGB-D indoor scene understanding challenge. We show that SegNet provides competitive performance although it is significantly smaller than other architectures. We also provide a Caffe implementation of SegNet and a webdemo at http://mi.eng.cam.ac.uk/projects/segnet/
Article
Full-text available
Template warping is a popular technique in visionbased 3D motion tracking and 3D pose estimation due to its flexibility of being applicable to monocular video sequences. However, the method suffers from two major limitations that hamper its successful use in practice. First, it requires the camera to be calibrated prior to applying the method. Second, it may fail to provide good results if the inter-frame displacements are too large. To overcome the first problem, we propose to estimate the unknown focal length of the camera from several initial frames by an iterative optimization process. To alleviate the second problem, we propose a tracking method based on combining complementary information provided by dense optical flow and tracked SIFT features. While optical flow is good for small displacements and provides accurate local information, tracked SIFT features are better in handling larger displacements or global transformations. To combine these two complementary information, we introduce a "forgetting factor" to bootstrap the 3D pose estimates provided by SIFT features, and refine the final results using optical flow. Experiments are performed on three public databases, i.e. the BIWI head pose dataset, the BU dataset, and the McGill faces datasets. Results illustrate that the proposed solution provides more accurate results than baseline methods that rely solely on either template warping or SIFT features. In addition, the approach can be applied in a larger variety of scenarios, due to circumventing the need for camera calibration, providing thus a more flexible solution to the problem than existing methods.
Article
Full-text available
Detection of cracks on bridge decks is a vital task for maintaining the structural health and reliability of concrete bridges. Robotic imaging can be used to obtain bridge surface image sets for automated on-site analysis. We present a novel automated crack detection algorithm, the STRUM (spatially tuned robust multifeature) classifier, and demonstrate results on real bridge data using a state-of-the-art robotic bridge scanning system. By using machine learning classification, we eliminate the need for manually tuning threshold parameters. The algorithm uses robust curve fitting to spatially localize potential crack regions even in the presence of noise. Multiple visual features that are spatially tuned to these regions are computed. Feature computation includes examining the scale-space of the local feature in order to represent the information and the unknown salient scale of the crack. The classification results are obtained with real bridge data from hundreds of crack regions over two bridges. This comprehensive analysis shows a peak STRUM classifier performance of 95% compared with 69% accuracy from a more typical image-based approach. In order to create a composite global view of a large bridge span, an image sequence from the robot is aligned computationally to create a continuous mosaic. A crack density map for the bridge mosaic provides a computational description as well as a global view of the spatial patterns of bridge deck cracking. The bridges surveyed for data collection and testing include Long-Term Bridge Performance program's (LTBP) pilot project bridges at Haymarket, VA, USA, and Sacramento, CA, USA.
Article
Full-text available
To ensure the safety and the serviceability of civil infrastructure it is essential to visually inspect and assess its physical and functional condition. This review paper presents the current state of practice of assessing the visual condition of vertical and horizontal civil infrastructure; in particular of reinforced concrete bridges, precast concrete tunnels, underground concrete pipes, and asphalt pavements. Since the rate of creation and deployment of computer vision methods for civil engineering applications has been exponentially increasing, the main part of the paper presents a comprehensive synthesis of the state of the art in computer vision based defect detection and condition assessment related to concrete and asphalt civil infrastructure. Finally, the current achievements and limitations of existing methods as well as open research challenges are outlined to assist both the civil engineering and the computer science research community in setting an agenda for future research.
Article
Full-text available
Background Many context-aware techniques have been proposed to deliver cyber-information, such as project specifications or drawings, to on-site users by intelligently interpreting their environment. However, these techniques primarily rely on RF-based location tracking technologies (e.g., GPS or WLAN), which typically do not provide sufficient precision in congested construction sites or require additional hardware and custom mobile devices. Method This paper presents a new vision-based mobile augmented reality system that allows field personnel to query and access 3D cyber-information on-site by using photographs taken from standard mobile devices. The system does not require any location tracking modules, external hardware attachments, and/or optical fiducial markers for localizing a user’s position. Rather, the user’s location and orientation are purely derived by comparing images from the user’s mobile device to a 3D point cloud model generated from a set of pre-collected site photographs. Results The experimental results show that 1) the underlying 3D reconstruction module of the system generates complete 3D point cloud models of target scene, and is up to 35 times faster than other state-of-the-art Structure-from-Motion (SfM) algorithms, 2) the localization time takes at most few seconds in actual construction site. Conclusion The localization speed and empirical accuracy of the system provides the ability to use the system on real-world construction sites. Using an actual construction case study, the perceived benefits and limitations of the proposed method for on-site context-aware applications are discussed in detail.
Article
Full-text available
There is consensus on the importance of objectively and reliably assessing the condition and load capacity of aged bridges. Although each bridge may be considered as a unique structure, the behavior of many bridge types may be governed by only a few mechanisms and related parameters, especially if a population is constructed from standard designs. By identifying these parameters, and their variation within the population, it is possible to extend findings such as load rating obtained from a statistical sample to the entire population. Bridge type-specific strategies for load rating and condition assessment in conjunction with statistical sampling may therefore offer significant advantages for inspecting and load rating bridges sharing common materials, similar geometry and detailing, and the same critical behavior mechanisms. In this paper, the writers present their recent work on load rating of the reinforced concrete T-beam bridge population in Pennsylvania to objectively re-qualify them based on field-calibrated finite element models.
Article
Full-text available
Mixed Reality (MR) visual displays, a particular subset of Virtual Reality (VR) related technologies, involve the merging of real and virtual worlds somewhere along the 'virtuality continuum' which connects completely real environments to completely virtual ones. Augmented Reality (AR), probably the best known of these, refers to all cases in which the display of an otherwise real environment is augmented by means of virtual (computer graphic) objects. The converse case on the virtuality continuum is therefore Augmented Virtuality (AV). Six classes of hybrid MR display environments are identified. However quite different groupings are possible and this demonstrates the need for an efficient taxonomy, or classification framework, according to which essential differences can be identified. An approximately three-dimensional taxonomy is proposed comprising the following dimensions: extent of world knowledge, reproduction fidelity, and extent of presence metaphor.
Article
Full-text available
In this paper, we obtain sufficient conditions for the uniform convergence of trigonometric series with monotone (in the extended sense) coefficients.
Technical Report
In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively.
Chapter
The chapter covers topics relevant for the design of haptic interfaces and their use in virtual reality applications. It provides knowledge required for understanding complex force feedback approaches and introduces general issues that must be considered for designing efficient and safe haptic interfaces. Human haptics, mathematical models of virtual environment, collision detection, force rendering and control of haptic devices are the main theoretical topics covered in this chapter, which concludes with a summary of different haptic display technologies.
Article
Augmented reality (AR) allows to seamlessly insert virtual objects in an image sequence. In order to accomplish this goal, it is important that synthetic elements are rendered and aligned in the scene in an accurate and visually acceptable way. The solution of this problem can be related to a pose estimation or, equivalently, a camera localization process. This paper aims at presenting a brief but almost self-contented introduction to the most important approaches dedicated to vision-based camera localization along with a survey of several extension proposed in the recent years. For most of the presented approaches, we also provide links to code of short examples. This should allow readers to easily bridge the gap between theoretical aspects and practical implementations.
Article
In Civil Infrastructure System (CIS) applications, the requirement of blending synthetic and physical objects distinguishes Augmented Reality (AR) from other visualization technologies in three aspects: (1) it reinforces the connections between people and objects, and promotes engineers’ appreciation about their working context; (2) it allows engineers to perform field tasks with the awareness of both the physical and synthetic environment; and (3) it offsets the significant cost of 3D Model Engineering by including the real world background. This paper reviews critical problems in AR and investigates technical approaches to address the fundamental challenges that prevent the technology from being usefully deployed in CIS applications, such as the alignment of virtual objects with the real environment continuously across time and space; blending of virtual entities with their real background faithfully to create a sustained illusion of co-existence; and the integration of these methods to a scalable and extensible computing AR framework that is openly accessible to the teaching and research community. The research findings have been evaluated in several challenging CIS applications where the potential of having a significant economic and social impact is high. Examples of validation test beds implemented include an AR visual excavator-utility collision avoidance system that enables workers to “see” buried utilities hidden under the ground surface, thus helping prevent accidental utility strikes; an AR post-disaster reconnaissance framework that enables building inspectors to rapidly evaluate and quantify structural damage sustained by buildings in seismic events such as earthquakes or blasts; and a tabletop collaborative AR visualization framework that allows multiple users to observe and interact with visual simulations of engineering processes.
Article
Cracking can invite sudden failures of concrete structures. The objective of this research is to develop an integrated model based on digital image processing in developing the numerical representation of defects. The integration model consists of crack quantification, change detection, neural networks, and 3D visualization models to visualize the defects in such a way that it mimics the on-site visual inspections. The crack quantification model evaluates crack lengths based on the perimeter of the skeleton of a crack which considers the tortuosity of the crack. The change detection model is based on the Fourier Transform of digital images eliminating the need for image registration as required in the traditional. Also, the integrated model as proposed here for crack length and change detection is supported by neural networks to predict crack depth and 3D visualization of crack patterns considering crack density as a key attribute.
Article
This paper discusses the feasibility of using augmented reality AR to evaluate earthquake-induced building damage. In the proposed approach, previously stored building information is superimposed onto a real structure in AR. Structural damage can then be quantified by measuring and interpreting key differences between the real and augmented views of the facility. Proof-of-concept experi- ments were performed in conjunction with large-scale cyclic shear wall tests. In these, CAD images of the walls were superimposed onto the wall specimens. Then, as the wall specimens were deformed under applied loading, the horizontal drifts between the walls and the augmented images were computed using two different techniques and compared with actual wall drifts. The obtained results highlight the potential of using AR for rapid damage detection and indicate that the accuracy of structural displacements measured using AR is a direct function of the accuracy with which augmented images can be registered with the real world. The limitations of the technology, considerations for field implementation, and the potential for other related applications of AR are also discussed.
Article
Current inspection standards require an inspector to travel to a target structure site and visually assess the structure's condition. This approach is labor-intensive, yet highly qualitative. A less time-consuming and inexpensive alternative to current monitoring methods is to use a robotic system that could inspect structures more frequently, and perform autonomous damage detection. In this paper, a vision-based crack detection methodology is introduced. The proposed approach processes 2D digital images (image processing) by considering the geometry of the scene (computer vision). The crack segmentation parameters are adjusted automatically based on depth parameters. The depth perception is obtained using 3D scene reconstruction. This system extracts the whole crack from its background, where the regular edge-based approaches just segment the crack edges. This characteristic is appropriate for the development of a crack thickness quantification system. Experimental tests have been carried out to evaluate the performance of the proposed system.Highlights► A vision-based crack detection methodology is introduced and evaluated. ► The proposed approach utilizes depth perception to detect and segment cracks. ► The segmentation parameters are adjusted automatically based on depth parameters. ► The depth perception is obtained using 3D scene reconstruction.
Article
This paper describes research that investigated the application of the global positioning system and 3 degree-of-freedom 3-DOF angular tracking to address the registration problem during interactive visualization of construction graphics in outdoor augmented reality AR environments. The global position and the three-dimensional 3D orientation of a user's viewpoint are tracked, and this information is reconciled with the known global position and orientation of superimposed computer-aided design CAD objects. Based on this computation, the relative translation and axial rotations between the user's viewpoint and the CAD objects are continually calculated. The relative geometric transformations are then applied to the CAD objects inside a virtual viewing frustum that is coincided with the real world space that is in the user's view. The result is an augmented outdoor environment where superimposed graphical objects stay fixed to their real world locations as the user navigates. The algorithms are implemented in a software tool called UM-AR-GPS-ROVER that is capable of interactively placing static and dynamic 3D models at any location in outdoor augmented space. The concept and prototype are demonstrated with an example in which scheduled construction activities for the erection of a structural steel frame are graphically simulated in outdoor AR.
Conference Paper
Mixed reality systems seek to smoothly link the physical and data processing (digital) environments. Although mixed reality systems are becoming more prevalent, we still do not have a clear understanding of this interaction paradigm. Addressing this problem, this article introduces a new interaction model called Mixed Interaction model. It adopts a unified point of view on mixed reality systems by considering the interaction modalities and forms of multimodality that are involved for defining mixed environments. This article presents the model and its foundations. We then study its unifying and descriptive power by comparing it with existing classification schemes. We finally focus on the generative and evaluative power of the Mixed Interaction model by applying it to design and compare alternative interaction techniques in the context of RAZZLE, a mobile mixed reality game for which the goal of the mobile player is to collect digital jigsaw pieces localized in space.
Article
The paper derives a set of fundamental issues from a definition of collaboration, introduces two major approaches to human-computer collaboration, and surveys each approach, showing how it formulates and addresses the issues. It concludes by proposing some themes that should characterize a unified approach to human-computer collaboration.
Article
We live in a physical world whose properties we have come to know well through long familiarity. We sense an involvement with this physical world which gives us the ability to predict its properties well. For example, we can predict where objects will fall, how well-known shapes look from other angles, and how much force is required to push objects against friction. We lack corresponding familiarity with the forces on charged particles, forces in non-uniform fields, the effects of nonprojective geometric transformations, and high-inertia, low friction motion. A display connected to a digital computer gives us a chance to gain familiarity with concepts not realizable in the physical world. It is a looking glass into a mathematical wonderland. Computer displays today cover a variety of capabilities. Some have only the fundamental ability to plot dots. Displays being sold now generally have built in line-drawing capability. An ability to draw simple curves would be useful. Some available displays are able to plot very short line segments in arbitrary directions, to form characters or more complex curves. Each of these abilities has a history and a known utility.
The Leader in Mixed Reality Technology
  • Microsoft
Microsoft. The Leader in Mixed Reality Technology. Holo-Lens, 2016.
Long-Term Bridge Performance High Priority Bridge Performance Issues
  • M C Brown
  • J P Gomez
  • M L Hammer
  • J M Hooks
Brown, M. C., J. P. Gomez, M. L. Hammer, and J. M. Hooks. Long-Term Bridge Performance High Priority Bridge Performance Issues. McLean, Va., 2014.
Recent Advances in Augmented Reality
  • R Azuma
  • R Behringer
  • S Feiner
  • S Julier
  • B Macintyre
Azuma, R., R. Behringer, S. Feiner, S. Julier, and B. Macintyre. Recent Advances in Augmented Reality. IEEE Computer Graphics and Applications, Vol. 2011, 2001, pp. 1-27. https://doi.org/10.4061/2011/908468.
Hybrid Sensor-Camera Monitoring for Damage Detection: Case Study of a Real Bridge
  • R Zaurin
  • T Khuc
  • F N Catbas
  • F Asce
Zaurin, R., T. Khuc, F. N. Catbas, and F. Asce. Hybrid Sensor-Camera Monitoring for Damage Detection: Case Study of a Real Bridge. Journal of Bridge Engineering, Vol. 21, No. 6, 2015, pp. 1-27. https://doi.org/10.1061/(ASCE)BE.1943.
Guide Manual for Bridge Element Inspection
  • Aashto
Programming 3D Applications with HTML5 and WebGL: 3D Animation and Visualization for Web Pages
  • T Parisi
Parisi, T. Programming 3D Applications with HTML5 and WebGL: 3D Animation and Visualization for Web Pages. O'Reilly Media, 2014.
Manual for Bridge Element Inspection. Bridge Element Inspection Manual
  • Aashto
  • Guide
AASHTO. Guide Manual for Bridge Element Inspection. Bridge Element Inspection Manual, 2011, p. 172.
Federal Highway Administration. National Bridge Inspection Standards Regulations
Federal Highway Administration. National Bridge Inspection Standards Regulations. Federal Register, Vol. 69, No. 239, 2004, pp. 15-35.