Real-time 3D visual sensor for robust object recognition.
ABSTRACT This paper presents a novel 3D measurement system, which yields both depth and color information in real time, by calibrating a time-of-flight and two CCD cameras. The problem of occlusions is solved by the proposed fast occluded-pixel detection algorithm. Since the system uses two CCD cameras, missing color information of occluded pixels is covered by one another. We also propose a robust object recognition using the 3D visual sensor. Multiple cues, such as color, texture and 3D (depth) information, are integrated in order to recognize various types of objects under varying lighting conditions. We have implemented the system on our autonomous robot and made the robot do recognition tasks (object learning, detection, and recognition) in various environments. The results revealed that the proposed recognition system provides far better performance than the previous system that is based only on color and texture information.
- SourceAvailable from: Vineet Gandhi
[Show abstract] [Hide abstract]
- "Such a system can overcome the limitations of both the active-and passiverange (stereo) approaches, when considered separately, and provides accurate and fast 3D reconstruction of a scene at high resolution, e.g. 1200 × 1600 pixels at 30 frames/second, as in Fig. 1. hal-00725616, version 1 -27 Aug 2012 Author manuscript, published in "IEEE International Conference on Robotics and Automation (2012) 4742 -4749" DOI : 10.1109/ICRA.2012.6224771 A. Related Work The combination of a depth sensor with a color camera has been exploited in several robotic applications such as object recognition , , , person awareness, gesture recognition , simultaneous localization and mapping (SLAM) , , robotized plant-growth measurement , etc. These methods have to deal with the difficulty of noise in depth measurement and the inferior resolution of range data as compared to the color data. "
ABSTRACT: The combination of range sensors with color cameras can be very useful for robot navigation, semantic perception, manipulation, and telepresence. Several methods of combining range- and color-data have been investigated and successfully used in various robotic applications. Most of these systems suffer from the problems of noise in the range-data and resolution mismatch between the range sensor and the color cameras, since the resolution of current range sensors is much less than the resolution of color cameras. High-resolution depth maps can be obtained using stereo matching, but this often fails to construct accurate depth maps of weakly/repetitively textured scenes, or if the scene exhibits complex self-occlusions. Range sensors provide coarse depth information regardless of presence/absence of texture. The use of a calibrated system, composed of a time-of-flight (TOF) camera and of a stereoscopic camera pair, allows data fusion thus overcoming the weaknesses of both individual sensors. We propose a novel TOF-stereo fusion method based on an efficient seed-growing algorithm which uses the TOF data projected onto the stereo image pair as an initial set of correspondences. These initial “seeds” are then propagated based on a Bayesian model which combines an image similarity score with rough depth priors computed from the low-resolution range data. The overall result is a dense and accurate depth map at the resolution of the color cameras at hand. We show that the proposed algorithm outperforms 2D image-based stereo algorithms and that the results are of higher resolution than off-the-shelf color-range sensors, e.g., Kinect. Moreover, the algorithm potentially exhibits real-time performance on a single CPU.Proceedings - IEEE International Conference on Robotics and Automation 05/2012; DOI:10.1109/ICRA.2012.6224771
- "The idea of fusing several type of 3D perception sensors  is applied in several applications including face detection , scene analyses , simultaneous localization and mapping , architectural industry , etc. For these applications the relation between different recorded datasets represents a widely encountered problem. "
Conference Paper: Heterogeneous feature based correspondence estimation[Show abstract] [Hide abstract]
ABSTRACT: This paper gives an insight in the preliminary results of an ongoing work about heterogeneous point feature estimation acquired from different type of sensors including structured light camera, stereo camera and a custom 3D laser range finder. The main goal of the paper is to compare the performance of the different type of local descriptors for indoor office environment. Several type of 3D features were evaluated on different datasets including the output of an enhanced stereo image processing algorithm too. From the extracted features the correspondences were determined between two different recording positions for each type of sensor. These correspondences were filtered and the final benchmarking of the extracted feature correspondences were compared for the different data sets. Further on, there is proposed an open access dataset for public evaluation of the proposed algorithms.Multisensor Fusion and Integration for Intelligent Systems (MFI), 2012 IEEE Conference on; 01/2012
- [Show abstract] [Hide abstract]
ABSTRACT: We propose a method for learning novel objects from audio visual input. The proposed method is based on two techniques: out-of-vocabulary (OOV) word segmentation and foreground object detection in complex environments. A voice conversion technique is also involved in the proposed method so that the robot can pronounce the acquired OOV word intelligibly. We also implemented a robotic system that carries out interactive mobile manipulation tasks, which we call “extended mobile manipulation”, using the proposed method. In order to evaluate the robot as a whole, we conducted a task “Supermarket” adopted from the RoboCup@Home league as a standard task for real-world applications. The results reveal that our integrated system works well in real-world applications. KeywordsMobile manipulation–Object learning–Object recognition–Out-of-vocabulary–RoboCup@HomeJournal of Intelligent and Robotic Systems 04/2011; 66(1):187-204. DOI:10.1007/s10846-011-9605-1 · 0.81 Impact Factor