No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Targetless Lidar-camera registration using patch-wise mutual information
... Due to temperature changes, hardware degradation, and vibrations, the extrinsic calibration between the lidar and the camera can change over time. To address this issue, we presented a method for targetless lidar-camera registration that combines pre-training and optimization with neural networkbased mutual information estimation and Lie-group techniques (Hermann et al., 2022). ...
Autonomous surface vessels are a promising building block of the future's transport sector and are investigated by research groups worldwide. This paper presents a comprehensive and systematic overview of the autonomous research vessel Solgenia including the latest investigations and recently presented methods that contributed to the fields of autonomous systems, applied numerical optimization, nonlinear model predictive control, multi-extended-object-tracking, computer vision, and collision avoidance. These are considered to be the main components of autonomous water taxi applications. Autonomous water taxis have the potential to transform the traffic in cities close to the water into a more efficient, sustainable, and flexible future state. Regarding this transformation, the test platform Solgenia offers an opportunity to gain new insights by investigating novel methods in real-world experiments. An established test platform will strongly reduce the effort required for real-world experiments in the future.
We present a novel dataset captured from a VW station wagon for use in mobile robotics and autonomous driving research. In total, we recorded 6 hours of traffic scenarios at 10–100 Hz using a variety of sensor modalities such as high-resolution color and grayscale stereo cameras, a Velodyne 3D laser scanner and a high-precision GPS/IMU inertial navigation system. The scenarios are diverse, capturing real-world traffic situations, and range from freeways over rural areas to inner-city scenes with many static and dynamic objects. Our data is calibrated, synchronized and timestamped, and we provide the rectified and raw image sequences. Our dataset also contains object labels in the form of 3D tracklets, and we provide online benchmarks for stereo, optical flow, object detection and other tasks. This paper describes our recording platform, the data format and the utilities that we provide.
We explore the use of geometrical methods to tackle the non-negative independent component analysis (non-negative ICA) problem, without assuming the reader has an existing background in differential geometry. We concentrate on methods that achieve this by minimizing a cost function over the space of orthogonal matrices. We introduce the idea of the manifold and Lie group SO(n) of special orthogonal matrices that we wish to search over, and explain how this is related to the Lie algebra so(n) of skew-symmetric matrices. We describe how familiar optimization methods such as steepest-descent and conjugate gradients can be transformed into this Lie group setting, and how the Newton update step has an alternative Fourier version in SO(n). Finally we introduce the concept of a toral subgroup generated by a particular element of the Lie group or Lie algebra, and explore how this com- mutative subgroup might be used to simplify searches on our constraint surface. No proofs are presented in this article.
We present two classes of improved estimators for mutual information M(X,Y), from samples of random points distributed according to some joint probability density mu(x,y). In contrast to conventional estimators based on binnings, they are based on entropy estimates from k -nearest neighbor distances. This means that they are data efficient (with k=1 we resolve structures down to the smallest possible scales), adaptive (the resolution is higher where data are more numerous), and have minimal bias. Indeed, the bias of the underlying entropy estimates is mainly due to nonuniformity of the density at the smallest resolved scale, giving typically systematic errors which scale as functions of k/N for N points. Numerically, we find that both families become exact for independent distributions, i.e. the estimator M(X,Y) vanishes (up to statistical fluctuations) if mu(x,y)=mu(x)mu(y). This holds for all tested marginal distributions and for all dimensions of x and y. In addition, we give estimators for redundancies between more than two random variables. We compare our algorithms in detail with existing algorithms. Finally, we demonstrate the usefulness of our estimators for assessing the actual independence of components obtained from independent component analysis (ICA), for improving ICA, and for estimating the reliability of blind source separation.
Accurate LiDAR-camera online calibration is critical for modern autonomous vehicles and robot platforms. Dominant methods heavily rely on hand-crafted features, which are not scalable in practice. With the increasing popularity of deep learning (DL), a few recent efforts have demonstrated the advantages of DL for feature extraction on this task. However, their reported performances are not sufficiently satisfying yet. We believe one improvement can be the problem formulation with proper consideration of the underneath geometry. Besides, existing online calibration methods focus on optimizing the calibration error while overlooking the tolerance within the error bounds. To address the research gap, a DL-based LiDAR-camera calibration method, named as the RGGNet, is proposed by considering the Riemannian geometry and utilizing a deep generative model to learn an implicit tolerance model. When evaluated on KITTI benchmark datasets, the proposed method demonstrates superior performances to the state-of-the-art DL-based methods. The code will be publicly available at https://github.com/KleinYuan/RGGNet.
In this paper we present CMRNet, a realtime approach based on a Convolutional Neural Network to localize an RGB image of a scene in a map built from LiDAR data. Our network is not trained in the working area, i.e. CMRNet does not learn the map. Instead it learns to match an image to the map. We validate our approach on the KITTI dataset, processing each frame independently without any tracking procedure. CMRNet achieves 0.27m and 1.07deg median localization accuracy on the sequence 00 of the odometry dataset, starting from a rough pose estimate displaced up to 3.5m and 17deg. To the best of our knowledge this is the first CNN-based approach that learns to match images from a monocular camera to a given, preexisting 3D LiDAR-map.
Convolutional Neural Networks define an exceptionally powerful class of
models, but are still limited by the lack of ability to be spatially invariant
to the input data in a computationally and parameter efficient manner. In this
work we introduce a new learnable module, the Spatial Transformer, which
explicitly allows the spatial manipulation of data within the network. This
differentiable module can be inserted into existing convolutional
architectures, giving neural networks the ability to actively spatially
transform feature maps, conditional on the feature map itself, without any
extra training supervision or modification to the optimisation process. We show
that the use of spatial transformers results in models which learn invariance
to translation, scale, rotation and more generic warping, resulting in
state-of-the-art performance on several benchmarks, and for a number of classes
of transformations.
This paper reports on an algorithm for automatic, targetless, extrinsic calibration of a lidar and optical camera system based upon the maximization of mutual information between the sensor-measured surface intensities. The proposed method is completely data-driven and does not require any fiducial calibration targets—making in situ calibration easy. We calculate the Cramér-Rao lower bound (CRLB) of the estimated calibration parameter variance, and we show experimentally that the sample variance of the estimated parameters empirically approaches the CRLB when the amount of data used for calibration is sufficiently large. Furthermore, we compare the calibration results to independent ground-truth (where available) and observe that the mean error empirically approaches zero as the amount of data used for calibration is increased, thereby suggesting that the proposed estimator is a minimum variance unbiased estimate of the calibration parameters. Experimental results are presented for three different lidar-camera systems: (i) a three-dimensional (3D) lidar and omnidirectional camera, (ii) a 3D time-of-flight sensor and monocular camera, and (iii) a 2D lidar and monocular camera.
Half-title pageSeries pageTitle pageCopyright pageDedicationPrefaceAcknowledgementsContentsList of figuresHalf-title pageIndex
Learning deep representations by mutual information estimation and maximization
- R D Hjelm
- A Fedorov
- S Lavoie-Marchildon
- K Grewal
- P Bachman
- A Trischler
Automatic calibration of lidar and camera images using normalized mutual information
- Z Taylor
- J Nieto
Mutual information neural estimation
- M I Belghazi
- A Baratin
- S Rajeswar
- S Ozair
- Y Bengio
- A Courville
Calibrating lidar and camera using semantic mutual information
- P Jiang
- P Osteen
- S Saripalli
Soic: Semantic online initialization and calibration for lidar and camera
- W Wang
- S Nobuhara
- R Nakamura
- K Sakurada