Conference PaperPDF Available

Multi Sensor Fusion for Navigation and Mapping in Autonomous Vehicles: Accurate Localization in Urban Environments

Authors:

Abstract and Figures

The combination of data from multiple sensors, also known as sensor fusion or data fusion, is a key aspect in the design of autonomous robots. In particular, algorithms able to accommodate sensor fusion techniques enable increased accuracy, and are more resilient against the malfunction of individual sensors. The development of algorithms for autonomous navigation, mapping and localization have seen big advancements over the past two decades. Nonetheless, challenges remain in developing robust solutions for accurate localization in dense urban environments, where the so called last-mile delivery occurs. In these scenarios, local motion estimation is combined with the matching of real-time data with a detailed pre-built map. In this paper, we utilize data gathered with an autonomous delivery robot to compare different sensor fusion techniques and evaluate which are the algorithms providing the highest accuracy depending on the environment. The techniques we analyze and propose in this paper utilize 3D lidar data, inertial data, GNSS data and wheel encoder readings. We show how lidar scan matching combined with other sensor data can be used to increase the accuracy of the robot localization and, in consequence, its navigation. Moreover, we propose a strategy to reduce the impact on navigation performance when a change in the environment renders map data invalid or part of the available map is corrupted.
Content may be subject to copyright.
Multi Sensor Fusion for Navigation and Mapping in Autonomous
Vehicles: Accurate Localization in Urban Environments
L. Qingqing1,2, J. Pe ˜
na Queralta2, T. Nguyen Gia2, Z. Zou1and T. Westerlund2
1School of Information Science and Technology, Fudan Universtiy, China
2Department of Future Technologies, University of Turku, Finland
Emails: {qingqingli16, zhuo}@fudan.edu.cn, {jopequ, tunggi, tovewe}@utu.fi,
Abstract—The combination of data from multiple sensors, also
known as sensor fusion or data fusion, is a key aspect in the
design of autonomous robots. In particular, algorithms able to
accommodate sensor fusion techniques enable increased accuracy,
and are more resilient against the malfunction of individual
sensors. The development of algorithms for autonomous navi-
gation, mapping and localization have seen big advancements
over the past two decades. Nonetheless, challenges remain in
developing robust solutions for accurate localization in dense
urban environments, where the so called last-mile delivery occurs.
In these scenarios, local motion estimation is combined with
the matching of real-time data with a detailed pre-built map.
In this paper, we utilize data gathered with an autonomous
delivery robot to compare different sensor fusion techniques
and evaluate which are the algorithms providing the highest
accuracy depending on the environment. The techniques we
analyze and propose in this paper utilize 3D lidar data, inertial
data, GNSS data and wheel encoder readings. We show how
lidar scan matching combined with other sensor data can be
used to increase the accuracy of the robot localization and, in
consequence, its navigation. Moreover, we propose a strategy to
reduce the impact on navigation performance when a change in
the environment renders map data invalid or part of the available
map is corrupted.
Index Terms—SLAM; Sensor fusion; Navigation; Localization;
Mapping; Urban navigation; ROS; PCL; 3D Lidar; LOAM; Last-
mile delivery; Autonomous vehicles; Self-driving cars;
I. INTRODUCTION
Accurate mapping and localization is the cornerstone of
self-driving cars. In open roads or highways, lane-following
can partly replace localization while still allowing for au-
tonomous operation [1]. Nonetheless, in dense urban environ-
ments accurate localization is a paramount aspect of a robot’s
autonomous operation [2]. Smaller pathways and more dy-
namic environments pose significant technical challenges [3].
In addition, the robot mission often adds accuracy require-
ments, such as in autonomous post delivery [4]. Multiple
sensors can be utilized to facilitate autonomous navigation and
operation. Among those, visual data provides more semantic
and qualitative information [5], [6], lidar measurements are
more accurate and are able to accurately describe objects from
a geometric point of view [7].
The past decade has seen a boost in the development
of autonomous vehicles for civilian use. Google started the
development of its self-driving technology for cars in 2009 [8],
and since then a myriad of industry leaders [9], [10], start-
ups [11], and academic researchers [12] have joined the race in
Fig. 1. Illustration of the matching process between the pre-acquired map
(red) and current lidar scan (green-blue).
the technology sector, a race to make everything autonomous.
In any mobile robot or vehicle, SLAM algorithms are an
essential and crucial aspect of autonomous navigation [2], [13].
Autonomous robots or self-driving cars will potentially
disrupt the logistics industry worldwide [14]. Autonomous
trucks or autonomous cargo vessels are already in advanced
stages or development and might be seen in operation within
the next five or ten years [15]. However, both technological
and legal challenges remain within the so called ”last-mile”
delivery [16]. Last-mile refers to the last step in the delivery
of goods from a local logistics or supply center to the clients’
door. In this paper, we utilize data gathered using a small
delivery robot from Jingdong, one of the top two e-commerce
platform and B2C online retailer in China.
The development of simultaneous localization and mapping
(SLAM) algorithms has seen a rapid evolution over the past
two decades [17], [18]. In SLAM algorithms, information from
a wide range of sensors can be used to map the environment
and localize the vehicle or robot in real time. These include
inertial measurement units, monocular or binocular cameras,
GNSS sensors, lidars, ultrasonic sensors or radars, or wheel
encoders [19]. Detailed 3D maps in the form of point clouds
can be generated, for instance, from 3D lidars or with stereo
vision [7]. We focus on the study and comparison of different
localization methods for a small delivery robot in dense urban
environments. In these scenarios, an existing map of the
operating environment is obtained in advance and either pre-
loaded or accessible by the autonomous robot. The map is used
in order to obtain more accurate localization by matching each
scan with a certain area of the map in real-time [2], [20].
The local motion of a robot or vehicle can be estimated
directly by integrating data from inertial measurement units,
including accelerometers, gyroscopes and compasses. Alter-
natively, different odometry methods can be applied based
on non-inertial sensors. Visual odometry algorithms utilize
feature extraction and tracking, while lidar-based odometry
uses mostly geometric information [17], [18]. Inertial mea-
surement units can be easily combined with wheel encoders.
Differential GNSS measurements also provide accurate local
motion estimation [21]. Global localization can be estimated
either with GNSS data, or by comparing sensor data with
predefined maps or information gathered a priori. For instance,
different methods exist to match a lidar scan with section of a
3D point cloud that defines a map of the operational area [2],
[20]. Over the past decades, researchers from both industry
and academia have been exploring the utilization of these
methods and their combinations to obtain accurate mapping
and localization. More concretely, scholars often refer to the
combination of different sensor data as sensor fusion or data
fusion. In this paper, we compare different techniques and
provide arguments on the best sensor fusion techniques for a
small delivery robot for last-mile delivery.
The algorithms, analysis and results presented in this paper
were mostly developed during the JD Digital (JDD) Global-
ization Challenge Competition in ”Multi-sensor fusion local-
ization for autonomous driving”. The challenge was a global
competition, with 4 classification divisions depending on the
geographical location of the team. Our team ranked first in the
US division semi-final and classified for the 24h global final in
Beijing, China, where the 4 semi-final winners competed for
the first prize. The available sensor data during the competition
was GNSS and gyroscope data, wheel odometry and the
output from a 16-channel 3D lidar. A map of the area was
given as a 3D point cloud. Multiple datasets exist to test
and benchmark different localization algorithms. However, the
most accurate algorithms are obtained through fine tuning and
parameters specific for the dataset, with different parameters
being potentially necessary to achieve the optimal accuracy in
a different sensor or environment setup [22]. Therefore, in this
paper we have utilized the data provided as part of the JDD
competition as it was gathered from the sensors on-board the
vehicle in order to compare a variety of methods. This ensures
that our algorithms can be implemented on the same robot
without a significant impact to performance.
The main contributions of this work are the following:
(i) to analyze and compare different approaches for vehicle
localization estimation, and the definition of a sensor fusion
approach for accurate localization in urban environments; and
(ii) the introduction of a strategy for rebuilding an area of
a local map when data is corrupted or the environment has
undergone significant modifications.
The remainder of this paper is organized as follows: Section
II introduces related work in localization and odometry tech-
niques based on data available from the on-board sensors. Sec-
tion III describes the dataset used in the paper, delves into the
possibilities of sensor fusion approaches based on the available
information, and provides a strategy to operate in areas where
the available map is either outdated or corrupted. Then, section
IV shows experimental results on the localization accuracy
for four different sensor fusion approaches. Finally, Section V
concludes the work and outlines directions for future work.
II. REL ATED WORK
Autonomous navigatio through 3D mapping with lidars has
been an increasingly popular technology over the last decade,
as lidars can provide high accuracy range measurements when
compared to other sensors. Zhang et al. proposed a method
for lidar odometry and mapping. The authors approached the
odometry problem by extracting lidar point cloud features
from each sweep. Then, the transformation between two
consecutive sweeps can be estimated. In this setup, the lidar
is utilized as an odometry sensor [17]. For 3D mapping,
the most important problem is accurate position estimation,
and 3d lidar data for odometry still can produce accumulated
error after long distance walk. However, the accumulated error
is much lower when compared to camera-based odometry
(especially in unfavourable light conditions) or integration
of inertial data from accelerometers/gyroscopes. State-of-the
art solutions combine lidar and monochrome camera sensors
as visual-lidar odometry to improve the performance of ego-
motion estimation [18].
In SLAM algorithms, localization and mapping are done
concurrently in real time. However, in order to achieve lo-
calization with centimeter accuracy in an urban environment,
map matching techniques have emerged over the past few
years. Currently, one of the most widely used approaches for
3D point cloud matching is Normal Distributions Transform
(NDT) matching [23]. Introduced by Bibel et al., NDT has the
advantages of no requiring explicit assignments of relation-
ships between features or subsets of points, and the analytic
formulation of the algorithms. The former aspect increases
the robustness of the algorithm, while the latter reduces the
computational cost and accuracy of the implementation.
Multiple improvements of the NDT algorithm have been
proposed. Gonzalez Prieto et al. presented DENDT, an al-
gorithm for 3D-NDT scan matching with Differential Evo-
lution [20]. The authors utilize a differential evolution (DE)
algorithm in order to improve the optimization process for
finding the solution of the NDT method. Akai et al. presented
a robust localization method that uses intensity readings from
the lidar data in order to detect road markings and use them
for matching consecutive scans [2]. While the method is able
to provide very high accuracy in some environments, the ex-
istence of road markers significantly impacts the performance
and therefore this method cannot be used in all situations.
Wen et al. recently analyzed the performance of different
NDT-based SLAM algorithms in multiple scenarios in Hong
Kong [24]. A valuable conclusion from this work is that the
best performance was achieved in areas with more sparse point
clouds and nominal traffic conditions, while the performance
Fig. 2. On the left, a map with noise added in some areas to simulate corrupted data. On the right, one of
the two corrupted sections of the the map has been restored using GNSS, IMU and lidar odometry.
decreased in dense urban areas. In this work, we study how can
we combine different sensor information in order to improve
localization performance in an urban area.
III. LOCALIZATION AND MAP PIN G
In this section, we describe the dataset that we have utilized
and the different localization approaches. We also introduce a
strategy for situations where the existing map data is corrupted,
outdated, or part of the data is missing.
A. Dataset
The data utilized in this paper was provided as part of
the JD Discovery Global Digitalization Challenge from De-
cember 2018 to January 2019. The data was gathered using
JD’s autonomous last-mile delivery robot, depicted in fig. ??
on page ??. The data includes: (1) GNSS directional and
positional data referenced in the World Geodetic System
(WGS84) format; (2) lidar data as a 3D point cloud; (3) raw
accelerometer and gyroscope data; and (4) wheel speed meter.
Ground truth data is provided as well. The output of the 3D
lidar is given at 10 Hz, IMU data is acquired at 100 Hz and
GNSS data is updated at 5 Hz. In addition, a map of the
objective operation area is given. The map represented as a
point cloud is shown in red in fig. 1 on page 1. The dataset
contains sensor data recorded in an 800-second long closed
loop movement.
In order to both read and process data, ROS has been
utilized. ROS (Robot Operating system) is an open source op-
erating system for robots, which provides a publish-subscribe
communication framework that allows for rapid development
of distributed robotic systems [25]. ROS provides algorithm
reuse support for robot research and development, as well as
abstraction of data models for easier integration of different
modules. PCL (Point Cloud Library) is a cross-platform open
source C++ library, which implements common algorithms and
data structures of point clouds [26]. It can realize point cloud
acquisition, filtering, segmentation, registration, retrieval, fea-
ture extraction, recognition, tracking, surface reconstruction,
visualization and so on. If OpenCV is the crystallization of
2D visual data acquisition and processing, PCL has the same
position in the 3D geometrical data domain.
B. Localization methods
Based on the available sensor data, we have utilized five
different approaches to estimate the vehicle’s localization. In
each approach, we use a different combination of sensors and
describe how the robot position is calculated based on their
data.
GNSS-based localization
One of the most traditional methods for outdoor robot
localization is to use a global navigation satellite system.
In this case, data from multiple satellite constellations was
available and used for increase accuracy. GNSS data error
are mostly caused by the atmospheric conditions and multi-
path interference. The effect of the environment in a larger
scale and the atmospheric conditions can be minimized using
differential GNSS readings, and assuming that the real-time
error is equivalent to the error obtained in a near known
location with which the system is synchronized. However, in
this work we have not relied on differential GNSS.
GNSS+IMU localization
We can easily combine GNSS data with inertial data,
including both accelerometer and accuracy. As differential
GNSS has not been implemented in this case, instead, the
results labelled as ”IMU” utilize the IMU readings for local
motion estimation, and the GNSS reading for an initial global
estimation and estimations when the robot movement is almost
zero for a prolonged period of time.
Lidar odometry (LOAM)
Zhang et al. introduced lidar odometry as an alternative to
the more classical visual odometry techniques [17]. As with
many odometry approaches, features are extracted from data
and compared within consecutive frames, or scans in the case
of a lidar. Features extracted from lidar data are usually based
on geometrical aspects. These include corners and surfaces,
for instance. Because lidars are able to provide high accu-
racy distance measurements even for objects far away from
the sensors, lidar-based odometry is able to provide higher
accuracy than visual-based odometry in open space situations
with clearly differentiated objects. An implementation based
on Zhang’s algorithm has been used in this case.
NDT-based localization (NDT+)
The NDT algorithm is a kind of registration algorithm
that uses the existing high-precision point cloud map and
real-time 3D lidar point cloud data to achieve high-precision
localization.
NDT algorithm does not directly compare the distance
between points in point clouds map and points in lidar point
clouds. First, the NDT algorithm will transform the point cloud
map into the normal distribution in three-dimensional.
If a variable Xis normal distribution X(µ, δ), then it
can be described as:
f(x) = 1
δ2πe
(xµ)2
2δ2(1)
where µis the mean of the variable distribution and δ2is the
variance. For a multivariate normal distribution, its probability
density function can be expressed as:
f(~x) = 1
(2π)D
2p|P|e(~x~µ)TP1(~x~µ)(2)
Where ~x represents the mean vector and Prepresents the
covariance matrix.
The first step of the NDT algorithm is to divide the point
cloud into a 3D grid coordinate. For each cell, the probability
distribution function(PDF) is calculated based on the points
distribution density in the grid.
~µ =1
m
m
X
k=1
~yk(3)
Σ=1
mX
k=1
m(~yk~µ)( ~yk~µ)T(4)
Where ~yk= 1,2,3, m denotes all lidar points in a grid.
Then the PDF can be expressed as:
f(~x) = 1
(2π)3
2p|Σ|e(~x~µ)TP1(~x~µ)(5)
We use the normal distribution to represent the discrete
points of each grid. Each probability density function can be
considered as an approximation of a local surface. It not only
describes the location of the surface in space but also contains
information about the direction and smoothness of the surface.
After calculated the PDF of each grid, then our goal is
to find the best transformation. The lidar point cloud set is
X=~x1, ~x2, ..., ~xn, and the parameter of transformation is ~p.
The spatial transformation function T(~p, ~xk)represents using
transformation ~p to move point ~xk, combined with the previous
calculated state density function(the PDF of each grid), so
the best transformation ~p should be the transformation of
maximum likelihood function:
Likelihood :θ=
n
Y
k=1
f(T(~p, ~xk))
And the maximum likelihood is equivalent to minimum neg-
ative logarithmic likelihood:
log θ=
n
X
k=1
log f(T(~p, ~xk))
The task now becomes to minimize the negative logarithmic
likelihood by using an optimization algorithm to adjust the
transformation parameter ~p. we can use the Newton method
to optimize the parameters.
The main problem of the NDT approach is its stability
when used standalone. As indicated by the authors of previous
works, NDT alone has the disadvantage of being unstable
depending on the scenario [24]. Therefore, we utilize GNSS
data for setting the initial position as well as resetting the
NDT localization method when a sudden change in position
or orientation estimation is detected. In the results, we refer
to this method as NDT+, and a close implementation to the
one provided in existing NDT method has been used [24].
NDT+IMU localization (NDT++)
The final localization utilized in our experiments consisted
on integrating the IMU data into the NDT+ method described
above that uses lidar and GNSS data. With this approach,
we have been able to eradicate the instabilities of the NDT+
method and increase its accuracy.
The algorithm workflow is as follows: first, on system start-
up or reset, GNSS data is used in order to obtain an initial
estimation of the robot’s location. This estimation can be
utilized in order to reduce the area of the map in which the
NDT matching will be looked for. Second, when the robot
starts moving, an unscented Kalman filter that uses IMU data
as input serves as an estimation between lidar scans. The
Kalman filter output is then feeded to the NDT algorithm for
scan matching with the predefined map. The GNSS data is still
used to avoid instabilities, even though we have not detected
any in the dataset utilized in this paper.
The NDT+ and NDT++ approaches have an additional
benefit over the lidar odometry method. In autonomous robots
moving in an urban environment, it is essential to react on time
to obstacles and to have localization information as frequently
as possible. A lidar-only approach has the disadvantage of
receiving sensor updates at only 10Hz in this case. With IMU
readings having a refresh rate of 100Hz, the IMU can be
utilized to obtain local movement estimation between lidar
scan matches using the NDT approach. This minimizes the
possibilities of instabilities in the NDT algorithm as the match-
ing possibilities are reduced and the goal of the algorithm
partly shifts from coarse localization to increasing the accuracy
of IMU-based movement estimation.
C. Corrupted Map Reconstruction
In an urban scenario, it is impractical to propose a local-
ization method that has a high dependency on the existence
of an accurate map of the operational area without a strategy
for operating in case the map data is corrupted or outdated.
In fig. 2 on the preceding page, we show the map of the
0 200 400 600 800
0
2
4
time (s)
Translational Error (m)
LOAM
NDT++
NDT+
GNSS
Fig. 3. Translational errors of the proposed approaches over time.
0 200 400 600 800
5
0
5
time (s)
Rotational error (°)
LOAM
NDT++
NDT+
GNSS
IMU
Fig. 4. Rotational errors of the proposed approaches over time.
operation area (on the left, in black and white), with two areas
where the data has either been removed completely or noise
has been added to render the NDT algorithm unusable. When
the robot approaches these areas, we are able to detect them by
monitoring the difference between the NDT localization and
GNSS and IMU positioning. When part of the map cannot
be matched with current scans, we utilize lidar odometry and
mapping in order to rebuild the corrupted or missing data.
The result of this process is shown on the right side of fig. 2
on page 3, where one of the corrupted map areas has been
restored in real-time while the robot was travelling through
it using lidar odometry and mapping. Even though it is not
visible in the image, there is a relatively small mismatch in
the map in the area where the robot emerges again into a
mapped environment.
IV. EXP ERIMENT AND RESU LTS
We have applied the five approaches proposed to the given
dataset. The results showing the translational and rotational
localization and orientation error are shown in fig. 3 and fig. 4,
respectively. In these figures, the NDT++ method shows a
stable and very small error though time, both in position and
rotation estimation. The NDT+ without taking into account
inertial data shows a larger error but, more importantly, shows
several instabilities that are corrected from the GNSS data.
In order to be able to compare in more detail the different
methods, and to see whether there exist some background
error or drifting, we show the variability of the localization
error though two sets of boxplots in fig. 5 and fig. 6 on the
next page. The specific values are also listed in Table I. We
TABLE I
LOC ALIZ ATI ON ERR OR MEA N AND STA DARD D EVIATI ON
µrot. σrot µxσxµyσy
GNSS 1.52 0.79 -0.46 0.22 1040.18
IMU -1.24 0.62 n/a n/a N/A N/A
LOAM -0.50 1.88 0.40 0.49 0.10 0.49
NDT+ 0.03 0.87 -0.02 1.51 -0.05 1.13
NDT++ 1030.31 -0.01 0.10 -0.05 0.10
LOAM NDT+ NDT++ GNSS
1
0
1
2
Translational Error (m)
Fig. 5. Boxplots of translational errors for the different approaches: x-error
(red) and y-error (blue).
have omitted the location errors of standalone IMU motion
estimation as the error is significantly higher than the proposed
approaches. However, inertial data is still valuable for local
movement estimation and for orientation estimation.
The translational errors are shown in fig. 5, where the red
boxes show the error in the xcoordinate, and the blue boxes
refer to the ycoordinate error. We have separated the error
because in some cases the error mean differs between them.
That is the case of the GNSS data, which due to atmospheric or
environmental conditions shows a steady negative drift in the x
direction. If consistent through large periods of time, it can be
assumed that it is due to a sensing error in the device itself, or
to environment conditions such a specific multi-path occurring
in the operating area. Therefore, this value can be utilized to
decrease the sensing error in real time during operation. In
order to have a deeper understanding of the distribution of
the GNSS error, we show the histogram of the three errors
(two translational, one rotational) in fig. 7 on the next page.
Only the error in the ydirection has a mean of 0, while
the distribution of the xerror is symmetrical and narrow.
Therefore, it is possible to fix the drift while keeping the same
variance for both components of the translational error. In the
case of the rotational error, it is more complex to correct even
though the distribution is still symmetric.
From fig. 5 and fig. 6 on the next page we can see that the
most stable methods are NDT++ and GNSS, with the NDT+
method providing accurate results for rotation estimation.
However, the NDT+ is highly unstable for position estimation,
with the highest variance of all presented approaches. In
position estimation, all approaches have a relatively small
error after 800 seconds of movement, except for the LOAM
method, which error drifts away from 0 towards the end of
the available data set. Similarly, the IMU constantly drifts in
LOAM NDT+ NDT++ GNSS IMU
4
2
0
2
Rotational Error (°)
Fig. 6. Boxplots of rotational errors for the different approaches in degrees.
1 0 1 2
0
1,000
2,000
3,000
4,000
Error (m, degrees)
Absolute Frequency
GNSS x-error
GNSS y-error
GNSS yaw error
Fig. 7. Distribution of GNSS errors.
terms of orientation estimation but it provides a more stable
measurement than LOAM, HDL+ or GNSS.
In summary, lidar scan matching with a 3D map provides
the highest accuracy for localization, both in terms of position
and orientation. Nonetheless, it is essential to take into account
other sensor data in order to implement a more robust approach
that is less prone to instabilities and depends less on the
operational environment. GNSS and inertial data are essential
for increasing the localization accuracy but also for minimizing
the possibilities of unexpected behaviour in the algorithm.
V. CONCLUSION
Accurate localization in dense urban areas is paramount
in order to solve the autonomous last-mile delivery problem.
Nonetheless, it still presents important challenges. We have
explored the possibilities for localization in a city environment
using 3D lidar data complemented with GNSS and inertial data
using a delivery robot from JD. We have shown the accuracy
of different approaches, assuming that a map of the operation
area is given in the form of a point cloud. In addition, we
have presented a strategy for situations where the map might
be corrupted or the scenario might have undergone significant
changes that rendered the map outdated. We have shown that
3D scan matching is the best approach for localization when
properly complimented with IMU data within an unscented
Kalman filter, and GNSS data.
In future work, we will explore the possibilities of inte-
grating visual data for odometry, as well as using the lidar
odometry together with the inertial data within the Kalman
filter in order to increase the NDT localization accuracy.
ACKNOWLEDGMENT
This work has been supported by NFSC grant No.
61876039, and the Shanghai Platform for Neuromorphic and
AI Chip (NeuHeilium).
REFERENCES
[1] R. W. Wolcott. Visual localization within lidar maps for automated
urban driving. In 2014 IEEE/RSJ International Conference on Intelligent
Robots and Systems, pages 176–183, Sep. 2014.
[2] N. Akai et al. Robust localization using 3d ndt scan matching with
experimentally determined uncertainty and road marker matching. In
2017 IEEE Intelligent Vehicles Symposium (IV), June 2017.
[3] J. Levinson et al. Map-based precision vehicle localization in urban
environments. In Robotics: Science and Systems, volume 4, page 1.
Citeseer, 2007.
[4] A. Heinla et al. Method and system for autonomous or semi-autonomous
delivery, August 16 2018. US Patent App. 15/948,974.
[5] S. Se et al. Vision-based global localization and mapping for mobile
robots. IEEE Transactions on robotics, 21(3):364–375, 2005.
[6] E. Garcia-Fidalgo et al. Vision-based topological mapping and localiza-
tion methods: A survey. Robotics and Autonomous Systems, 64, 2015.
[7] C. Premebida et al. High-resolution lidar-based depth mapping using
bilateral filter. In 2016 IEEE 19th international conference on intelligent
transportation systems (ITSC), pages 2469–2474. IEEE, 2016.
[8] M. John. Google cars drive themselves. Traffic {WWW
Document}. The New York Times. URL http://www. nytimes.
com/2010/10/10/science/10google. html, 2010.
[9] M. Bojarski et al. End to end learning for self-driving cars. arXiv
preprint arXiv:1604.07316, 2016.
[10] M. Harris. Documents confirm apple is building self-driving car. The
Guardian, 14, 2015.
[11] N. A. Greenblatt. Self-driving cars and the law. IEEE spectrum,
53(2):46–51, 2016.
[12] S. Kato et al. An open approach to autonomous vehicles. IEEE Micro,
35(6):60–68, 2015.
[13] W. Zhang. Lidar-based road and road-edge detection. In 2010 IEEE
Intelligent Vehicles Symposium, pages 845–848, June 2010.
[14] E. Hofmann. Industry 4.0 and the current status as well as future
prospects on logistics. Computers in Industry, 89:23–34, 2017.
[15] H. Fl¨
amig. Autonomous vehicles and autonomous driving in freight
transport. In Autonomous driving, pages 365–385. Springer, 2016.
[16] N. Boysen et al. Scheduling last-mile deliveries with truck-based
autonomous robots. European Journal of Operational Research,
271(3):1085–1099, 2018.
[17] J. Zhang et al. Loam: Lidar odometry and mapping in real-time. In
Robotics: Science and Systems, 2014.
[18] J. Zhang et al. Visual-lidar odometry and mapping: Low-drift, robust,
and fast. In 2015 IEEE International Conference on Robotics and
Automation (ICRA). IEEE, 2015.
[19] H. Cho et al. A multi-sensor fusion system for moving object detection
and tracking in urban driving environments. In 2014 IEEE International
Conference on Robotics and Automation (ICRA), May 2014.
[20] P. G. Prieto et al. Dendt: 3d-ndt scan matching with differential
evolution. In 2017 25th Mediterranean Conference on Control and
Automation (MED), pages 719–724, July 2017.
[21] C. Rizos et al. Precise point positioning: is the era of differential gnss
positioning drawing to an end? 2012.
[22] The KITTI Vision Benchmark Suite. Visual odometry and slam
evaluation, 2012-2019.
[23] P. Biber et al. The normal distributions transform: A new approach
to laser scan matching. In Proceedings 2003 IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS 2003)(Cat. No.
03CH37453), volume 3, pages 2743–2748. IEEE, 2003.
[24] W. Wen et al. Performance analysis of ndt-based graph slam for
autonomous vehicle in diverse typical driving scenarios of hong kong.
Sensors, 18(11):3928, 2018.
[25] M. Quigley et al. Ros: an open-source robot operating system. In ICRA
workshop on open source software. Kobe, Japan, 2009.
[26] R. B. Rusu et al. 3D is here: Point cloud library (PCL). In 2011 IEEE
International Conference on Robotics and Automation, May 2011.
... There are multiple well-established frameworks and algorithms for autonomous driving in urban environments [8], as well as localization and mapping in roads and buildings [9]. Many of the solutions proposed in these areas have a strong dependency on lidar scanners [10], among other sensors. In the field of forest mapping and navigation, several researchers have utilized terrestrial laser scan (TLS) to build point-cloud maps [11][12][13][14][15]. ...
... Autonomous mobile robots meant to operate outdoors often rely on GNSS sensors as the basic source of global localization data, and then integrate other sensor data through sensor fusion techniques for local position estimation [10]. GNSS sensors alone do not provide enough accuracy in urban or dense environments, with accuracy often in the order of meters [22]. ...
Article
Full-text available
Autonomous harvesting and transportation is a long-term goal of the forest industry. One of the main challenges is the accurate localization of both vehicles and trees in a forest. Forests are unstructured environments where it is difficult to find a group of significant landmarks for current fast feature-based place recognition algorithms. This paper proposes a novel approach where local point clouds are matched to a global tree map using the Delaunay triangularization as the representation format. Instead of point cloud based matching methods, we utilize a topology-based method. First, tree trunk positions are registered at a prior run done by a forest harvester. Second, the resulting map is Delaunay triangularized. Third, a local submap of the autonomous robot is registered, triangularized and matched using triangular similarity maximization to estimate the position of the robot. We test our method on a dataset accumulated from a forestry site at Lieksa, Finland. A total length of 200 m of harvester path was recorded by an industrial harvester with a 3D laser scanner and a geolocation unit fixed to the frame. Our experiments show a 12 cm s.t.d. in the location accuracy and with real-time data processing for speeds not exceeding 0.5 m/s. The accuracy and speed limit are realistic during forest operations.
... The DashGo platform also provides wheel odometry from its differential drive system. The 3-D lidar was used to accurately localize the landmarks and provide ground truth (GT) with map-based localization algorithms for 3-D point clouds introduced in [38]. The camera was used to detect the QR codes and extract the encoded information in them. ...
Article
Full-text available
As autonomous robots are becoming more widespread, more attention is being paid to the security of robotic operations. Autonomous robots can be seen as cyber–physical systems: they can operate in virtual, physical, and human realms. Therefore, securing the operations of autonomous robots requires not only securing their data (e.g., sensor inputs and mission instructions) but securing their interactions with their environment. There is currently a deficiency of methods that would allow robots to securely ensure their sensors and actuators are operating correctly without external feedback. This article introduces an encoding method and end-to-end validation framework for the missions of autonomous robots. In particular, we present a proof of concept of a map encoding method, which allows robots to navigate realistic environments and validate operational instructions with almost zero a priori knowledge. We demonstrate our framework using two different encoded maps in experiments with simulated and real robots. Our encoded maps have the same advantages as typical landmark-based navigation, but with the added benefit of cryptographic hashes that enable end-to-end information validation. Our method is applicable to any aspect of the robotic operation in which there is a predefined set of actions or instructions given to the robot.
... Active research areas in TIERS include multi-robot coordination [1], [2], [3], [4], [5], swarm design [6], [7], [8], [9], UWB-based localization [10], [11], [12], [13], [14], [15], localization and navigation in unstructured environments [16], [17], [18], lightweight AI at the edge [19], [20], [21], [22], [23], distributed ledger technologies at the edge [24], [25], [26], [27], [28], [29], edge architectures [30], [31], [32], [33], [34], [35], offloading for mobile robots [36], [37], [38], [39], [40], [41], [42], LPWAN networks [43], [44], [45], [46], sensor fusion algorithms [47], [48], [49], and reinforcement and federated learning for multi-robot systems [50], [51], [52], [53]. ...
... 1) System Modeling: The process view indicates the interplay of logical components of localization services in a sequential order. Figure 6 and map data, the initial GPS position narrows the area of map matching which optimizes and stabilizes the position estimation [61]. The last part of the process view is the transmission of vehicle location data to the cloud services for vehicle tracking. ...
Preprint
Full-text available
The Internet of Vehicles (IoV) equips vehicles with connectivity to the Internet and the Internet of Things (IoT) to support modern applications such as autonomous driving. However, the consolidation of complex computing domains of vehicles, the Internet, and the IoT limits the applicability of tailored security solutions. In this paper, we propose a new methodology to quantitatively verify the security of single or system-level assets of the IoV infrastructure. In detail, our methodology decomposes assets of the IoV infrastructure with the help of reference sub-architectures and the 4+1 view model analysis to map identified assets into data, software, networking, and hardware categories. This analysis includes a custom threat modeling concept to perform parameterization of Common Vulnerability Scoring System (CVSS) scores per view model domain. As a result, our methodology is able to allocate assets from attack paths to view model domains. This equips assets of attack paths with our IoV-driven CVSS scores. Our CVSS scores assess the attack likelihood which we use for Markov Chain transition probabilities. This way, we quantitatively verify system-level security among a set of IoV assets. Our results show that our methodology applies to arbitrary IoV attack paths. Based on our parameterization of CVSS scores and our selection of use cases, remote attacks are less likely to compromise location data compared to attacks from close proximity for authorized and unauthorized attackers respectively.
... However, these sensors present limitations in challenging environments with low-light or low-visibility conditions. In dense urban environments, lidar-based odometry is the only viable solution for long-term autonomy if high-accuracy localization is required [115]. ...
Article
Full-text available
Search and rescue (SAR) operations can take significant advantage from supporting autonomous or teleoperated robots and multi-robot systems. These can aid in mapping and situational assessment, monitoring and surveillance, establishing communication networks, or searching for victims. This paper provides a review of multi-robot systems supporting SAR operations, with system-level considerations and focusing on the algorithmic perspectives for multi-robot coordination and perception. This is, to the best of our knowledge, the first survey paper to cover (i) heterogeneous SAR robots in different environments, (ii) active perception in multi-robot systems, while (iii) giving two complementary points of view from the multi-agent perception and control perspectives. We also discuss the most significant open research questions: shared autonomy, sim-to-real transferability of existing methods, awareness of victims' conditions, coordination and interoperability in heterogeneous multi-robot systems, and active perception. The different topics in the survey are put in the context of the different challenges and constraints that various types of robots (ground, aerial, surface, or underwater) encounter in different SAR environments (maritime, urban, wilderness, or other post-disaster scenarios). The objective of this survey is to serve as an entry point to the various aspects of multi-robot SAR systems to researchers in both the machine learning and control fields by giving a global overview of the main approaches being taken in the SAR robotics area.
... IV. ENCODED NAVIGATION GRAPH One of the most fundamental ways in which a robot interacts with its environment is by navigating it. Maps have long been utilized for autonomous navigation and exploration in mobile robots to increase the robustness of long-term autonomous operation [23]- [25]. Maps or landmarks provide robots means for localization in a known reference frame, while enabling the calibration and adjustment of on-board odometry and localization algorithms. ...
Preprint
As autonomous robots become increasingly ubiquitous, more attention is being paid to the security of robotic operation. Autonomous robots can be seen as cyber-physical systems that transverse the virtual realm and operate in the human dimension. As a consequence, securing the operation of autonomous robots goes beyond securing data, from sensor input to mission instructions, towards securing the interaction with their environment. There is a lack of research towards methods that would allow a robot to ensure that both its sensors and actuators are operating correctly without external feedback. This paper introduces a robotic mission encoding method that serves as an end-to-end validation framework for autonomous robots. In particular, we put our framework into practice with a proof of concept describing a novel map encoding method that allows robots to navigate an objective environment with almost-zero a priori knowledge of it, and to validate operational instructions. We also demonstrate the applicability of our framework through experiments with real robots for two different map encoding methods. The encoded maps inherit all the advantages of traditional landmark-based navigation, with the addition of cryptographic hashes that enable end-to-end information validation. This end-to-end validation can be applied to virtually any aspect of robotic operation where there is a predefined set of operations or instructions given to the robot.
... Accurate localization and mapping are two of the pillars behind fully autonomous systems [5,13]. Over the past two decades, much attention has been put into solving the simultaneous localization and mapping (SLAM) problem [26,29,30]. ...
Conference Paper
Full-text available
Fleets of autonomous mobile robots are becoming ubiquitous in industrial environments such as logistic warehouses. This ubiquity has led in the Internet of Things field towards more distributed network architectures, which have crystallized under the rising edge and fog computing paradigms. In this paper, we propose the combination of an edge computing approach with computational offload-ing for mobile robot navigation. As smaller and relatively simpler robots become more capable, their penetration in different domains rises. These large multi-robot systems are often characterized by constrained computational and sensing resources. An efficient computational offloading scheme has the potential to bring multiple operational enhancements. However, with the most cost-effective autonomous navigation method being visual-inertial odometry, streaming high-quality images can induce latency increments with a consequent negative impact on operational performance. In this paper, we analyze the impact that image quality and compression have on the state-of-the-art on visual inertial odometry. Our results indicate that over one order of magnitude in image size and network bandwidth can be reduced without compromising the accuracy of the odometry methods even in challenging environments .This opens the door to further optimization by dynamically assessing the trade-off between image quality, network load, latency and performance of the visual-inertial odometry and localization accuracy.
Conference Paper
Full-text available
This conceptual paper discusses how different aspects involving the autonomous operation of robots and vehicles will change when they have access to next-generation mobile networks. 5G and beyond connectivity is bringing together a myriad of technologies and industries under its umbrella. High-bandwidth, low-latency edge computing services through network slicing have the potential to support novel application scenarios in different domains including robotics, autonomous vehicles, and the Internet of Things. In particular, multi-tenant applications at the edge of the network will boost the development of autonomous robots and vehicles offering computational resources and intelligence through reliable offloading services. The integration of more distributed network architectures with distributed robotic systems can increase the degree of intelligence and level of autonomy of connected units. We argue that the last piece to put together a services framework with third-party integration will be next-generation low-latency blockchain networks. Blockchains will enable a transparent and secure way of providing services and managing resources at the Multi-Access Edge Computing (MEC) layer. We overview the state-of-the-art in MEC slicing, distributed robotic systems and blockchain technology to define a framework for services the MEC layer that will enhance the autonomous operations of connected robots and vehicles.
Article
Full-text available
Robust and lane-level positioning is essential for autonomous vehicles. As an irreplaceable sensor, Light detection and ranging (LiDAR) can provide continuous and high-frequency pose estimation by means of mapping, on condition that enough environment features are available. The error of mapping can accumulate over time. Therefore, LiDAR is usually integrated with other sensors. In diverse urban scenarios, the environment feature availability relies heavily on the traffic (moving and static objects) and the degree of urbanization. Common LiDAR-based simultaneous localization and mapping (SLAM) demonstrations tend to be studied in light traffic and less urbanized area. However, its performance can be severely challenged in deep urbanized cities, such as Hong Kong, Tokyo, and New York with dense traffic and tall buildings. This paper proposes to analyze the performance of standalone NDT-based graph SLAM and its reliability estimation in diverse urban scenarios to further evaluate the relationship between the performance of LiDAR-based SLAM and scenario conditions. The normal distribution transform (NDT) is employed to calculate the transformation between frames of point clouds. Then, the LiDAR odometry is performed based on the calculated continuous transformation. The state-of-the-art graph-based optimization is used to integrate the LiDAR odometry measurements to implement optimization. The 3D building models are generated and the definition of the degree of urbanization based on Skyplot is proposed. Experiments are implemented in different scenarios with different degrees of urbanization and traffic conditions. The results show that the performance of the LiDAR-based SLAM using NDT is strongly related to the traffic condition and degree of urbanization. The best performance is achieved in the sparse area with normal traffic and the worse performance is obtained in dense urban area with 3D positioning error (summation of horizontal and vertical) gradients of 0.024 m/s and 0.189 m/s, respectively. The analyzed results can be a comprehensive benchmark for evaluating the performance of standalone NDT-based graph SLAM in diverse scenarios which is significant for multi-sensor fusion of autonomous vehicle.
Conference Paper
Full-text available
In this paper, we present a localization approach that is based on a point-cloud matching method (normal distribution transform "NDT") and road-marker matching based on the light detection and ranging intensity. Point-cloud map-based localization methods enable autonomous vehicles to accurately estimate their own positions. However, accurate localization and ``matching error'' estimations cannot be performed when the appearance of the environment changes, and this is common in rural environments. To cope with these inaccuracies, in this work, we propose to estimate the error of NDT scan matching beforehand (off-line). Then, as the vehicle navigates in the environment, the appropriate uncertainty is assigned to the scan matching. 3D NDT scan matching utilizes the uncertainty information that is estimated off-line, and is combined with a road-marker matching approach using a particle-filtering algorithm. As a result, accurate localization can be performed in areas in which 3D NDT failed. In addition, the uncertainty of the localization is reduced. Experimental results show the performance of the proposed method.
Conference Paper
Full-text available
Scan matching is one of the most important tasks that must be solved to achieve simultaneous localization and mapping with autonomous mobile robots. This paper shows the development and results of a new 3D scan matching algorithm based on the Normal Distribution Transform and the bio-inspired optimization algorithm Differential Evolution. Different tests have been carried out to show the good performance of the new solution to the scan matching problem.
Conference Paper
Full-text available
High resolution depth-maps, obtained by upsampling sparse range data from a 3D-LIDAR, find applications in many fields ranging from sensory perception to semantic segmentation and object detection. Upsampling is often based on combining data from a monocular camera to compensate the low-resolution of a LIDAR. This paper, on the other hand, introduces a novel framework to obtain dense depth-map solely from a single LIDAR point cloud; which is a research direction that has been barely explored. The formulation behind the proposed depth-mapping process relies on local spatial interpolation, using sliding-window (mask) technique, and on the Bilateral Filter (BF) where the variable of interest, the distance from the sensor, is considered in the interpolation problem. In particular, the BF is conveniently modified to perform depth-map upsampling such that the edges (foreground-background discontinuities) are better preserved by means of a proposed method which influences the range-based weighting term. Other methods for spatial upsampling are discussed, evaluated and compared in terms of different error measures. This paper also researches the role of the mask's size in the performance of the implemented methods. Quantitative and qualitative results from experiments on the KITTI Database, using LIDAR point clouds only, show very satisfactory performance of the approach introduced in this work.
Chapter
Full-text available
The degree of vehicle automation is continuously rising in all modes of transport both on public traffic infrastructure and in-house transport within company grounds, in order to improve the productivity, reliability, and flexibility of transport.
Article
Full-text available
We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands. This end-to-end approach proved surprisingly powerful. With minimum training data from humans the system learns to drive in traffic on local roads with or without lane markings and on highways. It also operates in areas with unclear visual guidance such as in parking lots and on unpaved roads. The system automatically learns internal representations of the necessary processing steps such as detecting useful road features with only the human steering angle as the training signal. We never explicitly trained it to detect, for example, the outline of roads. Compared to explicit decomposition of the problem, such as lane marking detection, path planning, and control, our end-to-end system optimizes all processing steps simultaneously. We argue that this will eventually lead to better performance and smaller systems. Better performance will result because the internal components self-optimize to maximize overall system performance, instead of optimizing human-selected intermediate criteria, e.g., lane detection. Such criteria understandably are selected for ease of human interpretation which doesn't automatically guarantee maximum system performance. Smaller networks are possible because the system learns to solve the problem with the minimal number of processing steps. We used an NVIDIA DevBox and Torch 7 for training and an NVIDIA DRIVE(TM) PX self-driving car computer also running Torch 7 for determining where to drive. The system operates at 30 frames per second (FPS).
Article
To reduce the negative impact of excessive traffic in large urban areas, many innovative concepts for intelligent transportation of people and freight have recently been developed. One of these concepts relies on autonomous delivery robots launched from trucks. A truck loads the freight dedicated to a set of customers in a central depot and moves into the city center. Also on board are small autonomous robots which each can be loaded with the freight dedicated to a single customer and launched from the truck. Then, the autonomous robots move to their dedicated customers and, after delivery, autonomously return to some robot depot in the city center. The truck can replenish robots at these decentralized depots to launch further of them until all its customers are supplied. To assess the potential of this innovative concept, this paper develops scheduling procedures which determine the truck route along robot depots and drop-off points where robots are launched, such that the weighted number of late customer deliveries is minimized. We formulate the resulting scheduling problem, investigate computational complexity, and develop suited solution methods. Furthermore, we benchmark the truck-based robot delivery concept with conventional attended home delivery by truck to assess the potential of this novel last-mile concept.
Article
It is the year 2023, and for the first time, a self-driving car navigating city streets strikes and kills a pedestrian. A lawsuit is sure to follow. But exactly what laws will apply? Nobody knows. Today, the law is scrambling to keep up with the technology, which is moving forward at a breakneck pace, thanks to efforts by Apple, Audi, BMW, Ford, General Motors, Google, Honda, Mercedes, Nissan, Nvidia, Tesla, Toyota, and Volkswagen. Google's prototype self-driving cars, with test drivers always ready to take control, are already on city streets in Mountain View, Calif., and Austin, Texas. In the second half of 2015, Tesla Motors began allowing owners (not just test drivers) to switch on its Autopilot mode.