Conference Paper

Coverage Path Planning with Adaptive Viewpoint Sampling to Construct 3D Models of Complex Structures for the Purpose of Inspection

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

In this paper, we introduce a coverage path planning algorithm with adaptive viewpoint sampling to construct accurate 3D models of complex large structures using Unmanned Aerial Vehicle (UAV). The developed algorithm, Adaptive Search Space Coverage Path Planner (ASSCPP), utilizes an existing 3D reference model of the complex structure and the onboard sensors’ noise models to generate paths that are evaluated based on the traveling distance and the quality of the model. The algorithm generates a set of viewpoints by performing adaptive sampling that directs the search towards areas with low accuracy and low coverage. The algorithm predicts the coverage percentage obtained by following the generated coverage path using the reference model. A set of experiments were conducted in real and simulated environments with structures of different complexities to test the validity of the proposed algorithm.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... We consider the problem of inspection planning where a robot needs to inspect a set of points of interest (POIs) in a given environment with its on-board sensor while optimizing plan cost. This problem has numerous applications such as product surface inspections for industrial quality control [1], structural inspections with unmanned aerial vehicles (UAVs) [2], [3], [4], [5], [6], ship-hull inspections [7], [8], [9], underwater inspections for scientific surveying [10], [11], [12], [13], and patient-anatomy inspections in medicalendoscopic procedures for disease diagnosis [14]. ...
... Then the trajectory-planning task is usually formulated using variants of the Traveling Salesman Problem (TSP) [22], [23], [24], [25]. To improve the quality of the solution, some use trajectory optimization [26], [27] while others resample viewpoints [3], [4], or adaptively sample viewpoints [6]. Unfortunately, this decomposition into two separate steps forgoes any guarantees on the quality of the solution. ...
Preprint
The inspection-planning problem calls for computing motions for a robot that allow it to inspect a set of points of interest (POIs) while considering plan quality (e.g., plan length). This problem has applications across many domains where robots can help with inspection, including infrastructure maintenance, construction, and surgery. Incremental Random Inspection-roadmap Search (IRIS) is an asymptotically-optimal inspection planner that was shown to compute higher-quality inspection plans orders of magnitudes faster than the prior state-of-the-art method. In this paper, we significantly accelerate the performance of IRIS to broaden its applicability to more challenging real-world applications. A key computational challenge that IRIS faces is effectively searching roadmaps for inspection plans -- a procedure that dominates its running time. In this work, we show how to incorporate lazy edge-evaluation techniques into \iris's search algorithm and how to reuse search efforts when a roadmap undergoes local changes. These enhancements, which do not compromise IRIS's asymptotic optimality, enable us to compute inspection plans much faster than the original IRIS. We apply IRIS with the enhancements to simulated bridge inspection and surgical inspection tasks and show that our new algorithm for some scenarios can compute similar-quality inspection plans 570x faster than prior work.
... Some applications require achieving complete coverage using the various CPP techniques such as agricultural surveying, structure painting, lawn mowing, surveillance, geospatial mapping, object reconstruction, and floor cleaning. Generally in CPP, either the model is reconstructed in real time utilizing the robot's sensing capabilities (non-model based) [23], [27], or a reference model is provided in advance for the structure or the environment of interest (modelbased) [10], [28]. Extensive reviews of the various CPP approaches in literature are presented in [2], [29] describing their functionalities and applications. ...
Article
Coverage Path Planning (CPP) is an essential capability for autonomous robots operating in various critical applications such as fire fighting, and inspection. Performing autonomous coverage using a single robot system consumes time and energy. In particular, 3D large structures might contain some complex and occluded areas that shall be scanned rapidly in certain application domains. In this paper, a new Hybrid Coverage Path Planning (HCPP) approach is proposed to explore and cover unknown 3D large structures using a decentralized multi-robot system. The HCPP approach combines a guided Next Best View (NBV) approach with a developed Long Short Term Memory (LSTM) waypoint prediction approach to decrease the CPP exploration time at each iteration and simultaneously achieve high coverage. The hybrid approach is the new ML paradigm which fosters intelligence by balancing between data efficiency and generality allowing the exchange of some CPP parts with a learned model. The HCPP uses a stateful LSTM network architecture which is trained based on collected paths that cover different 3D structures to predict the next viewpoint. This architecture captures the dynamic dependencies of adjacent viewpoints in the long term sequences like the coverage paths. The HCPP switches between these methods triggered by either the number of iterations or an entropy threshold. In the decentralized multi-robot system, the proposed HCPP is embedded in each robot where each one of them shares its global 3D map ensuring robustness. The results performed in a realistic Gazebo robotic simulator confirmed the advantage of the proposed HCPP approach by achieving high coverage on different 3D unknown structures in a shorter time compared to conventional NBV.
... A simulation software from Dronecode Foundation [74] helped account for some factors, such as UAV speed and wind effects. Almadhoun et al. [75] proposed an algorithm to evaluate the coverage and resample the VPIs after path planning focusing on the regions with low accuracy and no coverage. Table 1 lists a summary of related UAV path planning research along with a comparison of the previous methods and the proposed one including the following information: (1) The path planning method (i.e. ...
Article
Conventional visual bridge inspection methods are time consuming and may put the inspector in dangerous situations. Unmanned Aerial Vehicles (UAVs) have been recently introduced as an effective tool for bridge inspection. This paper proposes a 3D path planning method for using a UAV equipped with a Light Detection and Ranging (LiDAR) scanner for bridge inspection. The method integrates a Genetic Algorithm (GA) and A* algorithm to solve the Traveling Salesman Problem (TSP) considering the potential locations of bridge surface defects such as cracks. The objective is minimizing time of flight while achieving maximum visibility. The method provides the potential locations of surface defects to efficiently achieve perpendicular and overlapping views for sampling the viewpoints. Calculating the visibility with respect to the level of criticality leads to giving the priority to covering the areas with higher risk levels. The results reveal that considering overlapping views based on the level of criticality of the zones and perpendicular view for all viewpoints result in accurate and time-efficient data collection.
Conference Paper
Full-text available
We consider the problem of next-best view selection for volumetric reconstruction of an object by a mobile robot equipped with a camera. Based on a probabilistic volumetric map that is built in real time, the robot can quantify the expected information gain from a set of discrete candidate views. We propose and evaluate several formulations to quantify this information gain for the volumetric reconstruction task, including visibility likelihood and the likelihood of seeing new parts of the object. These metrics are combined with the cost of robot movement in utility functions. The next best view is selected by optimizing these functions, aiming to maximize the likelihood of discovering new parts of the object. We evaluate the functions with simulated and real world experiments within a modular software system that is adaptable to other robotic platforms and reconstruction problems. We release our implementation open source.
Chapter
Full-text available
In this chapter we present a modular Micro Aerial Vehicle (MAV) simulation framework, which enables a quick start to perform research on MAVs. After reading this chapter, the reader will have a ready to use MAV simulator, including control and state estimation. The sim-ulator was designed in a modular way, such that different controllers and state estimators can be used interchangeably, while incorporating new MAVs is reduced to a few steps. The provided controllers can be adapted to a custom vehicle by only changing a parameter file. Different controllers and state estimators can be compared with the provided evaluation framework. The simulation framework is a good starting point to tackle higher level tasks, such as collision avoidance, path planning, and vision based problems, like Simultaneous Localization and Mapping (SLAM), on MAVs. All components were designed to be analogous to its real world counterparts. This allows the usage of the same controllers and state estimators, including their parameters, in the simulation as on the real MAV.
Conference Paper
Full-text available
For large-scale and long-term simultaneous localization and mapping (SLAM), a robot has to deal with unknown initial positioning caused by either the kidnapped robot problem or multi-session mapping. This paper addresses these problems by tying the SLAM system with a global loop closure detection approach, which intrinsically handles these situations. However, online processing for global loop closure detection approaches is generally influenced by the size of the environment. The proposed graph-based SLAM system uses a memory management approach that only consider portions of the map to satisfy online processing requirements. The approach is tested and demonstrated using five indoor mapping sessions of a building using a robot equipped with a laser rangefinder and a Kinect.
Article
Full-text available
This paper presents a new algorithm for three-dimensional coverage path planning for autonomous structural inspection operations using aerial robots. The proposed approach is capable of computing short inspection paths via an alternating two-step optimization algorithm according to which at every iteration it attempts to find a new and improved set of viewpoints that together provide full coverage with decreased path cost. The algorithm supports the integration of multiple sensors with different fields of view, the limitations of which are respected. Both fixed-wing as well as rotorcraft aerial robot configurations are supported and their motion constraints are respected at all optimization steps, while the algorithm operates on both mesh- and occupancy map-based representations of the environment. To thoroughly evaluate this new path planning strategy, a set of large-scale simulation scenarios was considered, followed by multiple real-life experimental test-cases using both vehicle configurations.
Article
Full-text available
In this paper, we propose a novel and computationally efficient algorithm for simultaneous exploration and coverage with a vision-guided micro aerial vehicle (MAV) in unknown environments. This algorithm continually plans a path that allows the MAV to fulfil two objectives at the same time while avoiding obstacles: observe as much unexplored space as possible, and observe as much of the surface of the environment as possible given viewing angle and distance constraints. The former and latter objectives are known as the exploration and coverage problems respectively. Our algorithm is particularly useful for automated 3D reconstruction at the street level and in indoor environments where obstacles are omnipresent. By solving the exploration problem, we maximize the size of the reconstructed model. By solving the coverage problem, we maximize the completeness of the model. Our algorithm leverages the state lattice concept such that the planned path adheres to specified motion constraints. Furthermore, our algorithm is computationally efficient and able to run on-board the MAV in real-time. We assume that the MAV is equipped with a forward-looking depth-sensing camera in the form of either a stereo camera or RGB-D camera. We use simulation experiments to validate our algorithm. In addition, we show that our algorithm achieves a significantly higher level of coverage as compared to an exploration-only approach while still allowing the MAV to fully explore the environment.
Conference Paper
Full-text available
Searching with a sensor for objects and observing parts of a known environment efficiently is a fundamental problem in many real-world robotic applications such as household robots searching for objects, inspection robots searching for leaking pipelines, and rescue robots searching for survivors after a disaster. We consider the problem of identifying and planning efficient view point sequences for covering complex 3d environments. We compare empirically several variants of our algorithm that allow to trade-off schedule computation against execution time. Our results demonstrate that, despite the intractability of the overall problem, computing effective solutions for coverage search in real 3d environments is feasible.
Article
Full-text available
Inspection of ship hulls and marine structures using autonomous underwater vehicles has emerged as a unique and challenging application of robotics. The problem poses rich questions in physical design and operation, perception and navigation, and planning, driven by difficulties arising from the acoustic environment, poor water quality and the highly complex structures to be inspected. In this paper, we develop and apply algorithms for the central navigation and planning problems on ship hulls. These divide into two classes, suitable for the open, forward parts of a typical monohull, and for the complex areas around the shafting, propellers and rudders. On the open hull, we have integrated acoustic and visual mapping processes to achieve closed-loop control relative to features such as weld-lines and biofouling. In the complex area, we implemented new large-scale planning routines so as to achieve full imaging coverage of all the structures, at a high resolution. We demonstrate our approaches in recent operations on naval ships.
Conference Paper
Full-text available
Robust recognition of arbitrary object classes in natural visual scenes is an aspiring goal with numerous practical applications, for instance, in the area of autonomous robotics and autonomous vehicles. One obstacle on the way towards human-like recognition performance is the limitation of computational power, restricting the size of the training and testing dataset as well as the complexity of the object recognition system. In this work, we present a hierarchical, locally-connected neural network model that is well-suited for large-scale, high-performance object recognition. By using the NVIDIA CUDA framework, we create a massively parallel implementation of the model which is executed on a state-of-the-art graphics card. This implementation is up to 82 times faster than a single-core CPU version of the system. This significant gain in computational performance allows us to evaluate the model on a very large, realistic, and challenging set of natural images which we extracted from the LabelMe dataset. To compare our model to other approaches, we also evaluate the recognition performance using the well-known MNIST and NORB datasets, achieving a testing error rate of 0.76% and 2.87%, respectively.
Article
Full-text available
Consumer-grade range cameras such as the Kinect sensor have the potential to be used in mapping applications where accuracy requirements are less strict. To realize this potential insight into the geometric quality of the data acquired by the sensor is essential. In this paper we discuss the calibration of the Kinect sensor, and provide an analysis of the accuracy and resolution of its depth data. Based on a mathematical model of depth measurement from disparity a theoretical error analysis is presented, which provides an insight into the factors influencing the accuracy of the data. Experimental results show that the random error of depth measurement increases with increasing distance to the sensor, and ranges from a few millimeters up to about 4 cm at the maximum range of the sensor. The quality of the data is also found to be influenced by the low resolution of the depth measurements.
Article
Full-text available
Laser scanning range sensors are widely used for high-precision, high-density three-dimensional (3D) reconstruction and inspection of the surface of physical objects. The process typically involves planning a set of views, physically altering the relative object-sensor pose, taking scans, registering the acquired geometric data in a common coordinate frame of reference, and finally integrating range images into a nonredundant model. Efficiencies could be achieved by automating or semiautomating this process. While challenges remain, there are adequate solutions to semiautomate the scan-register-integrate tasks. On the other hand, view planning remains an open problem---that is, the task of finding a suitably small set of sensor poses and configurations for specified reconstruction or inspection goals. This paper surveys and compares view planning techniques for automated 3D object reconstruction and inspection by means of active, triangulation-based range sensors.
Conference Paper
In this paper, we introduce a coverage path planning algorithm for inspecting large structures optimized to generate highly accurate 3D models. Robotic inspection of structures such as aircrafts, bridges and buildings, is considered a critical task since missing any detail could affect the performance and integrity of the structures. Additionally, it is a time and resource intensive task that should be performed as efficiently and accurately as possible. The method we propose is a model based coverage path planning approach that generates an optimized path that passes through a set of admissible waypoints to cover a complex structure. The coverage path planning algorithm is developed with a heuristic reward function that exploits our knowledge of the structure mesh model, and the UAV's onboard sensors' models to generate optimal paths that maximizes coverage and accuracy, and minimizes distance travelled. Moreover, we accelerated critical components of the algorithm utilizing the Graphics Processing Unit (GPU) parallel architecture. A set of experiments were conducted in a simulated environment to test the validity of the proposed algorithm.
Technical Report
This paper presents a model-based view planning approach for automated object reconstruction or inspection using laser scanning range sensors. Quality objectives and performance measures are defined. Camera and positioning systems performance is modeled statistically. A theoretical framework is presented. The method is applicable to a broad class of objects with reasonable geometry and reflectance properties. Sampling of object surface and viewpoint space is characterized, including measurement noise and poses error effects. The technique is generalizable for common range camera and positioning system designs.
Conference Paper
We present a novel method for planning 3D coverage paths for inspection of complex structures on the ocean floor (such as seamounts or coral reefs) using an autonomous underwater vehicle (AUV). Our method initially uses an a priori map to plan a nominal coverage path that allows the AUV to pass its sensors over all points on the target structure. We then go beyond previous approaches in the literature by considering the vehicle's position uncertainty rather than relying on the unrealistic assumption of an idealized path execution. To this aim, we present a replanning algorithm based on stochastic trajectory optimization that reshapes the nominal path to cope with the actual target structure perceived in situ. The replanning algorithm runs onboard the AUV in realtime during the inspection mission, adapting the path according to the measurements provided by the vehicle's range sensing sonars. We demonstrate the efficacy of our method in experiments at sea using the GIRONA 500 AUV where we cover a concrete block of a breakwater structure in a harbor and an underwater boulder rising from 40 m up to 27 m depth. Moreover, we apply state-of-The-Art surface reconstruction techniques to the data acquired by the AUV and obtain 3D models of the inspected structures that show the benefits of our planning method for 3D mapping.
Conference Paper
Within this paper, a new fast algorithm that provides efficient solutions to the problem of inspection path planning for complex 3D structures is presented. The algorithm assumes a triangular mesh representation of the structure and employs an alternating two-step optimization paradigm to find good viewpoints that together provide full coverage and a connecting path that has low cost. In every iteration, the viewpoints are chosen such that the connection cost is reduced and, subsequently, the tour is optimized. Vehicle and sensor limitations are respected within both steps. Sample implementations are provided for rotorcraft and fixed-wing unmanned aerial systems. The resulting algorithm characteristics are evaluated using simulation studies as well as multiple real-world experimental test-cases with both vehicle types.
Article
Three-dimensional models provide a volumetric representation of space which is important for a variety of robotic applications including flying robots and robots that are equipped with manipulators. In this paper, we present an open-source framework to generate volumetric 3D environment models. Our mapping approach is based on octrees and uses probabilistic occupancy estimation. It explicitly represents not only occupied space, but also free and unknown areas. Furthermore, we propose an octree map compression method that keeps the 3D models compact. Our framework is available as an open-source C++ library and has already been successfully applied in several robotics projects. We present a series of experimental results carried out with real robots and on publicly available real-world datasets. The results demonstrate that our approach is able to update the representation efficiently and models the data consistently while keeping the memory requirement at a minimum.
Article
During the last decade, sampling-based path planning algorithms, such as Probabilistic RoadMaps (PRM) and Rapidly-exploring Random Trees (RRT), have been shown to work well in practice and possess theoretical guarantees such as probabilistic completeness. However, little effort has been devoted to the formal analysis of the quality of the solution returned by such algorithms, e.g., as a function of the number of samples. The purpose of this paper is to fill this gap, by rigorously analyzing the asymptotic behavior of the cost of the solution returned by stochastic sampling-based algorithms as the number of samples increases. A number of negative results are provided, characterizing existing algorithms, e.g., showing that, under mild technical conditions, the cost of the solution returned by broadly used sampling-based algorithms converges almost surely to a non-optimal value. The main contribution of the paper is the introduction of new algorithms, namely, PRM* and RRT*, which are provably asymptotically optimal, i.e., such that the cost of the returned solution converges almost surely to the optimum. Moreover, it is shown that the computational complexity of the new algorithms is within a constant factor of that of their probabilistically complete (but not asymptotically optimal) counterparts. The analysis in this paper hinges on novel connections between stochastic sampling-based path planning algorithms and the theory of random geometric graphs.
Article
This report describes an implementation of the Lin-Kernighan heuristic, one of the most successful methods for generating optimal or nearoptimal solutions for the symmetric traveling salesman problem. Computational tests show that the implementation is highly effective. It has found optimal solutions for all solved problem instances we have been able to obtain, including a 7397-city problem (the largest nontrivial problem instance solved to optimality today). Furthermore, the algorithm has improved the best known solutions for a series of large-scale problems with unknown optima, among these an 85900-city problem.
Article
This paper presents a model-based view planning approach for automated object reconstruction or inspection using laser scanning range sensors. Quality objectives and performance measures are defined. Camera and positioning systems performance is modeled statistically. A theoretical framework is presented. The method is applicable to a broad class of objects with reasonable geometry and reflectance properties. Sampling of object surface and viewpoint space is characterized, including measurement noise and poses error effects. The technique is generalizable for common range camera and positioning system designs. Le document présente une méthode de planification de vues basée sur un modèle pour la reconstitution et l'inspection automatique des objets à l'aide de capteurs de distance à balayage laser. Les objectifs de qualité et les mesures de rendement y sont décrits. Le rendement des systèmes de positionnement et de caméra est modélisé par méthode statistique. Un cadre théorique est présenté dans le document. La méthode peut être appliquée à une vaste gamme d'objets présentant une géométrie et des propriétés de réflectance raisonnables. Un échantillonnage de la surface de l'objet et de l'espace observé est caractérisé; les données comprennent entre autre la mesure du bruit et des effets des erreurs de pose. La technique peut s'appliquer de façon générale à la conception des systèmes courants de positionnement et de caméras télémétriques.
Robot Operating System (ROS): The Complete Reference
  • F Furrer
  • M Burri
  • M Achtelik
  • R Siegwart
F. Furrer, M. Burri, M. Achtelik, and R. Siegwart, Robot Operating System (ROS): The Complete Reference (Volume 1). Cham: Springer International Publishing, 2016, ch. RotorS-A Modular Gazebo MAV Simulator Framework, pp. 595-625.