Figure - available from: International Journal of Advanced Robotic Systems
This content is subject to copyright.
Source publication
People detection and tracking is an essential capability for mobile robots in order to achieve natural human–robot interaction. In this article, a human detection and tracking system is designed and validated for mobile robots using color data with depth information RGB-depth (RGB-D) cameras. The whole framework is composed of human detection, trac...
Similar publications
Point clouds are essential for storage and transmission of 3D content. As they can entail significant volumes of data, point cloud compression is crucial for practical usage. Recently, point cloud geometry compression approaches based on deep neural networks have been explored. In this paper, we evaluate the ability to predict perceptual quality of...
Citations
... Because the sensing range of RGB-D camera is limited, the RGB-D detection system is usually installed on indoor mobile robots [15,80] for sensing the surrounding environment. RGB-D cameras are also used in surveillance applications in indoor environment. ...
Object detection is an important problem and has a wide range of applications. In recent years, deep learning based object detection with conventional RGB cameras has made great progress. At the same time, people are more and more aware of the limitations of RGB cameras. The progress of algorithms alone can not fundamentally resolve the challenges of object detection. Unmanned vehicles or mobile robot platforms are often equipped with a variety of sensors in addition to RGB camera, each of which have its own characteristics, and can expand the sensing range of RGB camera from different dimensions. For example, infrared thermal imaging camera and multispectral camera broaden sensing range from spectral dimension, while LiDARs and depth cameras are able to broaden sensing range from the spatial dimension. This paper mainly summarizes the deep learning based object detection methods under the condition of multi-modal sensors, and surveys and categorizes the methods from the perspective of data fusion manner. The datasets of different modality are summarized, and the advantages and disadvantages with different combination of sensors are also discussed in this paper.
... While laser and sonic technologies have proven to be effective, modern studies have widely adopted cameras due to their flexibility and ability to simplify algorithm development using computer vision with minimal processing power. Common choices for visual input sensors include monocular and stereo cameras [4], [5], while depth cameras are frequently employed when fusing pixel and distance information is necessary for controlling the robot's movements [6], [7], [8], [9]. To ensure safe navigation, a companion robot must remain cognizant of the target's location within the environment while avoiding collisions with other objects. ...
... This eliminates the need for any further cutbacks. The proposed angular position estimation algorithm, referred to as APEA, comprises three kernels of varied sizes by default: (8,8), (4,4), and (2, 2). To clarify, the weightless kernels are not intended to learn specific features of an image at a low level. ...
... This eliminates the need for any further cutbacks. The proposed angular position estimation algorithm, referred to as APEA, comprises three kernels of varied sizes by default: (8,8), (4,4), and (2, 2). To clarify, the weightless kernels are not intended to learn specific features of an image at a low level. ...
The study proposed a dynamic path planning (DPP) method that combines instance image segmentation and elementary matrix calculations to enable a robot to identify the angular position of entities in its surroundings. The DPP method fuses visual and depth information for scene understanding and path estimation with reduced computation resources. This study designed, developed, and evaluated a deep-learning based companion robot prototype for indoor navigation and obstacle avoidance using an RGB-D camera as the sole input sensor. A simulated environment was employed to evaluate the robot’s path-planning ability using visual sensors. The DPP method enables the person-following robot to perform intelligent curve manipulation for safe path planning to avoid objects in the initial trajectory. The approach offers a unique and straightforward technique for scene understanding without the burden of extensive neural network configuration. Its modular architecture and flexibility make it a promising candidate for future development and refinement in this domain. Its effectiveness in collision prevention and path planning has potential implications for various applications, including medical robotics.
... In case that scenes can be captured three-dimensionally, such as using stereo cameras, RGB-D cameras, ToF cameras, or plenoptic cameras, clustering methods can be applied to the obtained point clouds to implement object detection. A new idea using mean shift clustering candidate segmentation for people detection based on the point clouds gathered by an RGB-D camera is introduced in [17]. The authors of [18] propose a fast clustering method on the basis of densitybased spatial clustering of applications with noise (DBSCAN) algorithm [19] to realize traffic detection for self-driving technology. ...
Multiple particle tracking-by-detection is a widely investigated issue in image processing. The paper presents approaches to detecting and tracking various refuse-derived fuel particles in a industrial environment using a plenoptic camera system, which is able to yield 2D gray value information and 3D point clouds with noticeable fluctuations. The presented approaches, including an innovative combined detection method and a post-processing framework for multiple particle tracking, aim at making the most of the acquired 2D and 3D information to deal with the fluctuations of the measuring system. The proposed novel detection method fuses the captured 2D gray value information and 3D point clouds, which is superior to applying single information. Subsequently, the particles are tracked by the linear Kalman filter and 2.5D global nearest neighbor (GNN) and joint probabilistic data association (JPDA) approach, respectively. As a result of several inaccurate detection results caused by the measuring system, the initial tracking results contain faulty and incomplete tracklets that entail a post-processing process. The developed post-processing approach based merely on particle motion similarity benefits a precise tracking performance by eliminating faulty tracklets, deleting outliers, connecting tracklets, and fusing trajectories. The proposed approaches are quantitatively assessed with manuelly labeled ground truth datasets to prove their availability and adequacy as well. The presented combined detection method provides the highest F
<sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub>
-score, and the proposed post-processing framework enhances the tracking performance significantly with regard to several recommended evaluation indices.
... Similarly, in [13], [14] people tracking was conducted in outdoor environments by multiple robots equipped with laser scanners and GPS sensors. In [15] people tracking (without pose estimation) was performed using an RGB-D camera on a single mobile robot. However, RGB-D cameras have a limited range. ...
In this work a multi-robot system is presented for
people detection and tracking in automated warehouses. Each
Automated Guided Vehicle (AGV) is equipped with multiple RGB
cameras that can track the workers’ current locations on the floor
thanks to a neural network that provides human pose estimation.
Based on the local perception of the environment each AGV
can exploit information about the tracked people for self-motion
planning or collision avoidance.
Additionally, data collected from each robot contributes to
a global people detection and tracking system. A warehouse
central management software fuses information received from
all AGVs into a map of the current locations of workers. The
estimated locations of workers are sent back to the AGVs to
prevent potential collision. The proposed method is based on
two-level hierarchy of Kalman filters. Experiments performed in
a real warehouse show the viability of the proposed approach.
... Fig 9shows human detection and tracking framework. The major procedures are listed as follows; a) Group detection and ceiling removal, b) Human detection using modified plan view map generation, c) Multiple person tracking[22]. ...
The emergence of autonomous machines is debated in the media. The progression of innovation in the field of
autonomous route for robots is one of the main patterns these days, where we have to experience tremendous
measure of eccentrics that is normal in military conditions during the defusing unexploded bombs or salvage
tasks. Autonomous robots work in various types of condition, and it is essential for them to move and avoid
obstacles around it. So as to explore the robot in an impact free way, way arranging calculations have been
introduced. A human recognition calculation running on an indoor versatile robot needs to address difficulties
including impediments because of jumbled situations, changing foundations because of the robot's movement,
and restricted on-board computational assets. Earlier human detection was mainly done on images taken by
visible light cameras. These methods take the detection process that humans use. We centre on continuous
human-location and following in tumultuous and dynamic conditions and furthermore featuring the hugeness of
fusing mechanical technology, correspondences and data examination in the field of land mine disclosure.
Keywords - Autonomous Robots, Balancing, Collision Avoidance, Human Detection, Landmine Detection
... In this work pedestrian detection is nailed as the background theory for the human crowds position detection. This kind of service robot has been explored by many publications [7,8,9]. In the pedestrian detection, the flow of people should be identified to find out the direction and number of persons that involved in a particular flow. ...
Identifying request or task for assistance in an elderly or retirement home is a challenge for service robot. It is an essential task for service robot to give their services along with human. Identifying human position in an environment, such as elderly home, is the most challenging task, as it involves static and dynamic object. During its operation, a service robot must avoid collision to objects, and also prepares path for its journey. This work makes use of RGB-D camera, such as Kinect, to identify the static and dynamic object via 3D maps. To manage its path finding, the prediction of people location and position is enabled via Long Term and Short Term Memory been implemented in Real-Time Appearance Based Mapping. The loop closure detection with real time constraints wrapping the collected information that uses RGB-D SLAM method. To identify the human position and flow, a technique of pedestrian detection is added to find group of people, which will gain more understanding on how human will interact to robot. We have demonstrated the work of this initial work by generating 3D point cloud for specific environment through an intensive mapping experiments.
... Liu et al. [34] proposed the new idea of spatial region of interest plan view maps for identifying human candidates after the ground plane was removed. A particle filter was adopted to track the motion models of multiple people. ...
Human detection and tracking is an essential component in several robotics applications, especially in indoor built environments where humans and robots are expected to coexist or collaborate. Due to their low cost and capability to capture both color and depth data, RGB-D cameras have shown significant promise in human detection and tracking for robotic applications. In this paper, a new human tracking method is proposed to detect and track a specific individual from a single RGB-D sensor using online learning classifiers with no ground plane assumption. Given a previous target human position, a candidate sampling method is designed to find potential positive samples while negative samples are obtained by a random sampling process. The kernelized Support Vector Machine (SVM) is employed as the online classifier to recognize the target human and updated using both the positive and negative examples. The experimental results on six RGB-D videos of a public dataset demonstrate that the proposed method achieves higher success rates compared to a 2D tracker and a 3D human detection method at a frame rate of 3.8 fps, and is capable of efficiently retrieving the target human following intermittent occlusion.
... Alone [28][29][30][31] or combined with the traditional methods [32], these novel deep learning methods are more and more widely used in people detection research. Based on the detection results, Kalman filter [33][34][35], particle filter [17,36], and probabilistic model [32] are often used to accomplish accurate and consecutive people tracking task. ...
Real-time astronaut visual tracking is the most important prerequisite for flying assistant robot to follow and assist the served astronaut in the space station. In this paper, an astronaut visual tracking algorithm which is based on deep learning and probabilistic model is proposed. Fine-tuned with feature extraction layers’ parameters being initialized by ready-made model, an improved SSD (Single Shot Multibox Detector) network was proposed for robust astronaut detection in color image. Associating the detection results with synchronized depth image measured by RGB-D camera, a probabilistic model is presented to ensure accurate and consecutive tracking of the certain served astronaut. The algorithm runs 10 fps at Jetson TX2, and it was extensively validated by several datasets which contain most instances of astronaut activities. The experimental results indicate that our proposed algorithm achieves not only robust tracking of the specified person with diverse postures or dressings but also effective occlusion detection for avoiding mistaken tracking.
... This approach faces the same problem of using the ground plane assumption. Liu et al. (2016) proposed the new idea of spatial region of interest plan view maps for identifying human candidates after the ground plane was removed. A particle filter was adopted to track the motion models of multiple people. ...
... However, it was not evaluated on public datasets, and the computational efficiency was not discussed. The methods that utilized HOG descriptors might fail when people squat down or are blocked by other objects (Liu et al. 2016). ...
Automation and robotics in construction (ARC) has the potential to assist in the performance of several mundane, repetitive, or dangerous construction tasks autonomously or under the supervision of human workers, and perform effective site and resource monitoring to stimulate productivity growth and facilitate safety management. When using ARC technologies, three-dimensional (3D) reconstruction is a primary requirement for perceiving and modeling the environment to generate 3D workplace models for various applications. Previous work in ARC has predominantly utilized 3D data captured from high-fidelity and expensive laser scanners for data collection and processing while paying little attention of 3D reconstruction and modeling using low-precision vision sensors, particularly for indoor ARC applications. This dissertation explores 3D reconstruction and modeling for ARC applications using low-precision vision sensors for both outdoor and indoor applications. First, to handle occlusion for cluttered environments, a joint point cloud completion and surface relation inference framework using red-green-blue and depth (RGB-D) sensors (e.g., Microsoft® Kinect) is proposed to obtain complete 3D models and the surface relations. Then, to explore the integration of prior domain knowledge, a user-guided dimensional analysis method using RGB-D sensors is designed to interactively obtain dimensional information for indoor building environments. In order to allow deployed ARC systems to be aware of or monitor humans in the environment, a real-time human tracking method using a single RGB-D sensor is designed to track specific individuals under various illumination conditions in work environments. Finally, this research also investigates the utilization of aerially collected video images for modeling ongoing excavations and automated geotechnical hazards detection and monitoring. The efficacy of the researched methods has been evaluated and validated through several experiments. Specifically, the joint point cloud completion and surface relation inference method is demonstrated to be able to recover all surface connectivity relations, double the point cloud size by adding points of which more than 87% are correct, and thus create high-quality complete 3D models of the work environment. The user-guided dimensional analysis method can provide legitimate user guidance for obtaining dimensions of interest. The average relative errors for the example scenes are less than 7% while the absolute errors less than 36mm. The designed human worker tracking method can successfully track a specific individual in real-time with high detection accuracy. The excavation slope stability monitoring framework allows convenient data collection and efficient data processing for real-time job site monitoring. The designed geotechnical hazard detection and mapping methods enable automated identification of landslides using only aerial video images collected using drones.
