Conference PaperPDF Available

Visual Odometry Offloading in Internet of Vehicles with Compression at the Edge of the Network

Authors:

Abstract and Figures

A recent trend in the IoT is to shift from traditional cloud-centric applications towards more distributed approaches embracing the fog and edge computing paradigms. In autonomous robots and vehicles, much research has been put into the potential of offloading computationally intensive tasks to cloud computing. Visual odometry is a common example, as real-time analysis of one or multiple video feeds requires significant on-board computation. If this operations are offloaded, then the on-board hardware can be simplified, and the battery life extended. In the case of self-driving cars, efficient offloading can significantly decrease the price of the hardware. Nonetheless, offloading to cloud computing compromises the system's latency and poses serious reliability issues. Visual odometry offloading requires streaming of video-feeds in real-time. In a multi-vehicle scenario, enabling efficient data compression without compromising performance can help save bandwidth and increase reliability.
Content may be subject to copyright.
Visual Odometry Offloading in Internet of Vehicles with
Compression at the Edge of the Network
L. Qingqing1,2, J. Pe ˜
na Queralta2, T. N. Gia2, H. Tenhunen3, Z. Zou1and T. Westerlund2
1School of Information Science and Technology, Fudan Universtiy, China
2Department of Future Technologies, University of Turku, Finland
3Department of Electronics, KTH Royal Institute of Technology, Sweden
Emails: 1{qingqingli16, zhuo}@fudan.edu.cn, 1{jopequ, tunggi, tovewe}@utu.fi, 3hannu@kth.se
Abstract—A recent trend in the IoT is to shift from traditional
cloud-centric applications towards more distributed approaches
embracing the fog and edge computing paradigms. In au-
tonomous robots and vehicles, much research has been put into
the potential of offloading computationally intensive tasks to
cloud computing. Visual odometry is a common example, as real-
time analysis of one or multiple video feeds requires significant
on-board computation. If this operations are offloaded, then
the on-board hardware can be simplified, and the battery life
extended. In the case of self-driving cars, efficient offloading can
significantly decrease the price of the hardware. Nonetheless,
offloading to cloud computing compromises the system’s latency
and poses serious reliability issues. Visual odometry offloading
requires streaming of video-feeds in real-time. In a multi-
vehicle scenario, enabling efficient data compression without
compromising performance can help save bandwidth and increase
reliability.
Index Terms—Odometry; VSLAM; Visual Odometry; Visual
SLAM; Internet of Vehicles; IoV; Edge Computing;
I. INTRODUCTION
Accurate localization is one of the key pillars behind full
autonomy. It is also essential for wider types of advanced
intelligent systems, including those related to human-robot
interaction. In terms of self-driving cars, the future of au-
tonomous vehicles is also the future of connected vehicles [1].
This will come under the umbrella of Internet of Everything
(IoE) and, more concretely, the Internet of Vehicles (IoV)
paradigms [2]. In these paradigms, all vehicles share data with
each other in vehicle-to-vehicle (V2V) communication, and
any entity with information that might affect its operation in
vehicle-to-everything (V2X) communication.
In GNSS-denied environments, or in those applications
where high accuracy is necessary, localization often relied on
odometry information. Typical ways of obtaining odometry
information is through lidars or cameras. Visual odometry with
mono or stereo vision has been extensively studied and current
state-of-the-art methods provide robust solutions for accurate
localization in both indoors and outdoors scenarios.
As the first cars with self-driving capabilities are entering
the market, a significant part of the vehicle production cost
is in the hardware required to provide robust and reliable
autonomous operation. V2V and V2X will be one of the key
factors in obtaining the target in terms of reliability, road
safety, traffic efficiency, and energy savings. If we combine
this with intensive computational offloading to near infrastruc-
ture, there is potential for important savings in the on-board
complexity of both hardware and software [3]. This can be
implemented in vehicles with human-in-the-loop, where the
operation can change to manual if required. Nonetheless, in
any case, strict control must be maintained over the network
load in order to ensure that the bandwidth available for each
unit is enough to keep latency and delays within safe limits.
II. REL ATE D WORK
To the best of our knowledge, previous works that have
considered offloading visual odometry calculations have uti-
lized cloud-centric architectures. Yun et al. proposed a
cloud robotics platform named RSE-PF for distributed visual
SLAM [4]. The authors reported round-trip latency of around
150ms. This, compared to state-of-the-art methods able of
processing at 30 frames/second or more, might result in delays
that limit the potential application scenarios. While the authors
utilized websockets in order to save bandwidth compared
to HTTP, they did not report on the maximum number of
concurrent units that could be handled. Dey et al., while
still relying on cloud servers, also proposed offloading in
a multi-tier edge+cloud setup [5]. The authors put a focus
on finding the optimal offloading strategy to make best use
of the different network layers. They formulated an integer
linear programming problem and provided an initial approach
for dynamically deciding on the best offloading decision, in
which the network bandwidth was a variable. In contrast, we
put the focus on studying what is the maximum bandwidth
savings that we can obtain without sacrificing performance,
while maintaining a reliable service with minimal latency.
III. IMAG E COMPRESSION FOR VISUAL ODOMETRY
Visual odometry (VO) is an estimation of camera motion
method based on a series of sequential images. VO can be
applied in a verity applications. The general idea is to calculate
the position correspondences between the two views by finding
some invariant features. In this work we utilize an approach
to visual odometry consisting of 3D-2D correspondences: In
this method, the transformation matrix is calculated using the
Perspective-n-Point(PnP) method. Firstly, the features across
two neighbor frames obtained by the camera are detected
and matched. The best matching points will be obtained
TABLE I
COMPRESSION RATE AND PERFORMANCE IMPACT WITH DIFFERENT JPEG
COMPRESSION TECHNIQUES.
Bandwidth Savings Accuracy loss
JPEG50 22% 0.1%
JPEG10 71% 1.2%
after incorrect matches are discarded. Then 3D points are
obtained by triangulating. After that, we eliminate inaccurate
3D points twice and combined the optical flow method and
feature matching method to find more accurate 3D-to-2D
correspondences. Finally, camera pose will be solved through
these correspondences by PnP algorithm.
A major challenge in offloading visual odometry is the
amount of data that needs to be streamed over the network.
Compared to 2D lidar data or IMU data, a continuous stream
of images consumes significantly higher bandwidth. Therefore,
if images can be compressed without compromising the algo-
rithm’s performance, we can increment the efficiency of the
edge offloading scheme.
IV. EXP ERIMENT AND RESU LTS
We have analyzed how traditional JPEG compression affects
the performance of visual offloading algorithms for different
levels of image quality levels. We have utilized a subset of
the TUM dataset [6]. The experiments have been run with a
set of 3682 acquired over 186 seconds. We compare the errors
produced as a consequence of compressing the images at 10%
quality and 50% quality, which results in compressed sizes
of 0.78 and 0.29 times the original image size, respectively,
as shown in Table 1. In terms of localization accuracy, the
cumulative error is around 10 times smaller in the case of the
50% quality compression. The total and cumulative errors, and
the errors in each direction are shown in Figure 1. We can see
that, in this case, the performance varies across dimensions.
The most clear difference appears in the error in the z axis,
where the difference between the two compression methods
is evident. In general, we can conclude that we can utilize
images with 50% quality without compromising accuracy in
most applications, while a 10% quality can be utilized in
situations where the the accuracy requirements are not so tight.
It is worth mentioning that the drift in the localization error
is continuous in both cases, with a mostly constant average
error. Therefore, it these method should be combined with
loop-closure techniques that ensure that the localization error
can be reduced to near zero within certain intervals of time.
V. CONCLUSION AND FUTURE WORK
In this paper, we present preliminary results on the study of
how different degrees of image compression affect the perfor-
mance of visual odometry algorithms. We have concluded that
around 20% of the network bandwidth can be saved without
compromising accuracy, while a slight reduction in accuracy
can bring over 70% of network load reduction, enabling a
more flexible scaling of the computational offloading scheme.
0 100 200 300
0
2
4
6
·103
JPEG10
JPEG50
(a) Error in x axis (m)
0 50 100 150
0
2
4
·103
JPEG10
JPEG50
(b) Error in y axis (m)
0 100 200 300
0
2
4
·103
JPEG10
JPEG50
(c) Error in z axis (m)
0 100 200 300
0
2
4
6
·105
Instant Error Square
JPEG10 Err.
JPEG50 Err.
0 100 200 300
0
1
2
3
·102
JPEG10 Cumm. Err.
JPEG50 Cumm. Err.
Fig. 1. Performance comparison of different compression rates.
In future work, we will study a wider range of image
compression techniques, including ML-powered lossless com-
pression techniques and measure network conditions.
ACKNOWLEDGMENT
This work has been supported by NFSC grant No.
61876039, and the Shanghai Platform for Neuromorphic and
AI Chip (NeuHeilium).
REFERENCES
[1] A. Bazzi et al. On the performance of ieee 802.11 p and lte-v2v for
the cooperative awareness of connected vehicles. IEEE Transactions on
Vehicular Technology, 66(11):10419–10432, 2017.
[2] O. Kaiwartya et al. Internet of vehicles: Motivation, layered architecture,
network model, challenges, and future aspects. IEEE Access, 2016.
[3] V. K. Sarker et al. Offloading slam for indoor mobile robots with edge-
fog-cloud computing. In ICASERT, IEEE, 2019.
[4] P. Yun et al. Towards a cloud robotics platform for distributed visual
slam. In Computer Vision Systems. Springer, 2017.
[5] S. Dey et al. Robotic slam: A review from fog computing and mobile
edge computing perspective. In MOBIQUITOUS. ACM, 2016.
[6] J. Sturm et al. Evaluating egomotion and structure-from-motion ap-
proaches using the TUM RGB-D benchmark. In CDCFR, IROS, 2012.
... Accurate localization is an essential aspect of the navigation of autonomous robots in GNSS-Denied environments. Existing approaches can be classified among those relying on on-board sensors only, such as visual or lidar odometry [1], or fixed elements in the environment in known locations, such as visual markers [2], or wireless beacons [3]. In the case of aerial robots, which have gained momentum over the past decade, accurate localization poses additional ...
... Accurate localization is an essential aspect of the navigation of autonomous robots in GNSS-Denied environments. Existing approaches can be classified among those relying on on-board sensors only, such as visual or lidar odometry [1], or fixed elements in the environment in known locations, such as visual markers [2], or wireless beacons [3]. In the case of aerial robots, which have gained momentum over the past decade, accurate localization poses additional challenges due to the three-dimensional nature of their operational environment. ...
... However, in multiple application scenarios only GNSS-denied navigation is possible, from emergency and post-disaster situations [5], to navigation in industrial environments such as factories or warehouses [6]. Multiple challenges remain in the utilization of odometry methods based on onboard sensors for accurate and robust localization in long-term autonomy [1], while the main drawback of most existing wireless localization solutions is their lower accuracy [3]. One of the advantages of wireless methods is that they are less dependent on the environment, such as in low-visibility conditions or the presence or dust of smoke. ...
Article
Full-text available
Ultra-wideband technology has emerged in recent years as a robust solution for localization in GNSS denied environments. In particular, its high accuracy when compared to other wireless localization solutions is enabling a wider range of collaborative and multi-robot application scenarios, being able to replace more complex and expensive motion-capture areas for use cases where accuracy in the order of tens of centimeters is sufficient. We present the first survey of UWB-based localization focused on multi-UAV systems and heterogeneous multi-robot systems. We have found that previous literature reviews do not consider in-depth the challenges in both aerial navigation and navigation with multiple robots, but also in terms of heterogeneous multi-robot systems. In particular, this is, to the best of our knowledge, the first survey to review recent advances in UWB-based (i) methods that enable ad-hoc and dynamic deployments; (ii) collaborative localization techniques; and (iii) cooperative sensing and cooperative maneuvers such as UAV docking on mobile platforms. Finally, we also review existing datasets and discuss the potential of this technology for both localization in GNSS-denied environments and collaboration in multi-robot systems.
... An important aspect of UGV navigation in the follow-me mode is the process of determining the exact location of the guide. Currently used approaches are distinguished based solely on on-board sensors, which include vision systems or lidars [5], stationary environmental features of the recognized environment, such as vision markers [6], or radio navigation signals [7]. For robots operating in a devastated environment, determining the exact location creates additional challenges. ...
... This applies to both emergency situations and natural disasters [9] as well as navigation in an industrial environment, which can include factory and warehouse halls [10]. There are many applications related to the use of odometry based on onboard sensors to accurately and reliably locate autonomously moving vehicles [5], although the greatest disadvantage of most existing wireless location solutions is their increasing location error [8]. One of the advantages of radio methods is that they are less sensitive to environmental conditions related to poor visibility caused by fog or smoke [11]. ...
Article
Full-text available
Unmanned Ground Vehicles (UGV) are devices capable of performing basic working movements without the operator being in their immediate working environment. Their capabilities include but are not limited to the perception of the environment with the use of sensors, determining the platform’s position, and planning and executing its movement. Ultra Wideband (UWB) is one of the wireless communication technologies which is increasingly used in location systems. This article presents the use of UWB technology in developing a guide localization system for a UGV (one of the stages of implementing a follow-me system). The article describes tests carried out on the developed testbed. Their aim was to determine the hardware configuration of the anchor arrangement characterized by the minimum number of lost data packets during operation. In order to determine the influence of the analysed variables on the output values, the method of global sensitivity analysis for neural networks was used.
... One of the main objectives of the ETSI MEC ISG is to define the base technologies for distributed and multi-tenant clouds that are meant to be deployed at the edge of the radio access network (RAN) [9]. By deploying data aggregation and processing tasks directly at the edge of the network, MEC services can provide better reliability, lower latency and higher-throughput [29,30,7]. We will specifically discuss throughout this paper how blockchain technology can play a key role in terms of security and robustness for the resource management needed in a multitenant edge infrastructure, as well as enhance the services that MEC applications can provide [31,32,33,34]. ...
... In this paper, the blockchain MEC slice was the key slice managing the deployment of applications across other MEC slices supporting different verticals within the automotive sector. Further adoption of blockchain for computational offloading will require, however, higher-bandwidth and lower-latency blockchain frameworks enabling real-time sensor data to be streamed for applications such as autonomous mobile robots [30,106]. ...
Preprint
Mobile edge computing (MEC) and next-generation mobile networks are set to disrupt the way intelligent and autonomous systems are interconnected. This will have an effect on a wide range of domains, from the Internet of Things to autonomous mobile robots. The integration of such a variety of MEC services in a inherently distributed architecture requires a robust system for managing hardware resources, balancing the network load and securing the distributed applications. Blockchain technology has emerged a solution for managing MEC services, with consensus protocols and data integrity checks that enable transparent and efficient distributed decision-making. In addition to transparency, the benefits from a security point of view are evident. Nonetheless, blockchain technology faces significant challenges in terms of scalability. In this chapter, we review existing consensus protocols and scalability techniques in both well-established and next-generation blockchain architectures. From this, we evaluate the most suitable solutions for managing MEC services and discuss the benefits and drawbacks of the available alternatives.
... Active research areas in TIERS include multi-robot coordination [1], [2], [3], [4], [5], swarm design [6], [7], [8], [9], UWB-based localization [10], [11], [12], [13], [14], [15], localization and navigation in unstructured environments [16], [17], [18], lightweight AI at the edge [19], [20], [21], [22], [23], distributed ledger technologies at the edge [24], [25], [26], [27], [28], [29], edge architectures [30], [31], [32], [33], [34], [35], offloading for mobile robots [36], [37], [38], [39], [40], [41], [42], LPWAN networks [43], [44], [45], [46], sensor fusion algorithms [47], [48], [49], and reinforcement and federated learning for multi-robot systems [50], [51], [52], [53]. ...
... With the proposed architecture, data related to other vehicles on a street can be exchanged directly with a connected vehicle through edge gateways in the near infrastructure. Moreover, the Edge AI opens multiple possibilities for computational offloading [42,43]. The benefits of our proposed architecture are in the control of the use of private vehicle data by third parties. ...
Chapter
Full-text available
Traditional cloud-centric architectures for Internet-of-Things applications are being replaced by distributed approaches. The Edge and Fog computing paradigms crystallize the concept of moving computation towards the edge of the network, closer to where the data originates. This has important benefits in terms of energy efficiency, network load optimization and latency control. The combination of these paradigms with embedded artificial intelligence in edge devices, or Edge AI, enables further improvements. In turn, the development of blockchain technology and distributed architectures for peer-to-peer communication and trade allows for higher levels of security. This can have a significant impact on data-sensitive and mission-critical applications in the IoT. In this paper, we discuss the potential of an Edge AI capable system architecture for the Blockchain of Things. We show how this architecture can be utilized in health monitoring applications. Furthermore, by analyzing raw data directly at the edge layer, we inherently avoid the possibility of breaches of sensitive information, as raw data is never stored nor transferred outside of the local network.
Article
Full-text available
Cloud-based robotics systems leverage a wide range of Information Technologies (IT) to offer tangible benefits like cost reduction, powerful computational capabilities, data offloading, etc. However, the centralized nature of cloud computing is not well-suited for a multitude of Operational Technologies (OT) nowadays used in robotics systems that require strict real-time guarantees and security. Edge computing and fog computing are complementary approaches that aim at mitigating some of these challenges by providing computing capabilities closer to the users. The goal of this work is hence threefold: i) to analyze the current edge computing and fog computing landscape in the context of robotics systems, ii) to experimentally evaluate an end-to-end robotics system based on solutions proposed in the literature, and iii) to experimentally identify current benefits and open challenges of edge computing and fog computing. Results show that, in the case of an exemplary delivery application comprising two mobile robots, the robot coordination and range can be improved by consuming real-time radio information available at the edge. However, our evaluation highlights that the existing software, wireless and virtualization technologies still require substantial evolution to fully support edge-based robotics systems.
Chapter
Mobile edge computing (MEC) and next-generation mobile networks are set to disrupt the way intelligent and autonomous systems are interconnected. This will have an effect on a wide range of domains, from the Internet of Things to autonomous mobile robots. The integration of such a variety of MEC services in an inherently distributed architecture requires a robust system for managing hardware resources, balancing the network load and securing the distributed applications. Blockchain technology has emerged a solution for managing MEC services, with consensus protocols and data integrity checks that enable transparent and efficient distributed decision-making. In addition to transparency, the benefits from a security point of view are evident. Nonetheless, blockchain technology faces significant challenges in terms of scalability. In this chapter, we review existing consensus protocols and scalability techniques in both well-established and next-generation blockchain architectures. From this, we evaluate the most suitable solutions for managing MEC services and discuss the benefits and drawbacks of the available alternatives.
Preprint
Rescue vessels are the main actors in maritime safety and rescue operations. At the same time, aerial drones bring a significant advantage into this scenario. This paper presents the research directions of the AutoSOS project, where we work in the development of an autonomous multi-robot search and rescue assistance platform capable of sensor fusion and object detection in embedded devices using novel lightweight AI models. The platform is meant to perform reconnaissance missions for initial assessment of the environment using novel adaptive deep learning algorithms that efficiently use the available sensors and computational resources on drones and rescue vessel. When drones find potential objects, they will send their sensor data to the vessel to verity the findings with increased accuracy. The actual rescue and treatment operation are left as the responsibility of the rescue personnel. The drones will autonomously reconfigure their spatial distribution to enable multi-hop communication, when a direct connection between a drone transmitting information and the vessel is unavailable.
Conference Paper
Full-text available
Indoor mobile robots are widely used in industrial environments such as large logistic warehouses. They are often in charge of collecting or sorting products. For such robots, computation-intensive operations account for a significant percentage of the total energy consumption and consequently affect battery life. Besides, in order to keep both the power consumption and hardware complexity low, simple micro-controllers or single-board computers are used as onboard local control units. This limits the computational capabilities of robots and consequently their performance. Offloading heavy computation to Cloud servers has been a widely used approach to solve this problem for cases where large amounts of sensor data such as real-time video feeds need to be analyzed. More recently, Fog and Edge computing are being leveraged for offloading tasks such as image processing and complex navigation algorithms involving non-linear mathematical operations. In this paper, we present a system architecture for offloading computationally expensive localization and mapping tasks to smart Edge gateways which use Fog services. We show how Edge computing brings computational capabilities of the Cloud to the robot environment without compromising operational reliability due to connection issues. Furthermore, we analyze the power consumption of a prototype robot vehicle in different modes and show how battery life can be significantly improved by moving the processing of data to the Edge layer.
Conference Paper
Full-text available
Cloud computing allows robots to offload computation and share information as well as skills. Visual SLAM is one of the intensively computational tasks for mobile robots. It can benefit from the cloud. In this paper, we propose a novel cloud robotics platform named RSE-PF for distributed visual SLAM with close attention to the infrastructure of the cloud. We implement it with Amazon Web Services and OpenResty. We demonstrate the feasibility, robustness, and elasticity of the proposed platform with a use case of perspective-n-point solution. In this use case, the average round-trip delay is 153 ms, which meets the near real-time requirement of mobile robots.
Article
Full-text available
Internet of Things (IoT) is smartly changing various existing research areas into new themes including smart-health, smart-home, smart-industry and smart-transport. Relying on the basis of ‘Smart-Transport’, Internet of Vehicles (IoV) is evolving as a new theme of research and development from Vehicular Adhoc Networks (VANETs). This paper presents a comprehensive framework of IoV with emphasis on layered architecture, protocol stack, network model, challenges and future aspects. Specifically, following the background on evolution of VANETs and motivation on IoV, an overview of IoV is presented as a heterogeneous vehicular networks. The IoV includes five types of vehicular communications; namely, Vehicle-to-Vehicle, Vehicle-to-Roadside, Vehicle-to- Infrastructure of cellular networks, Vehicle-to-Personal devices and Vehicle-to-Sensors. A five layered architecture of IoV is proposed considering functionalities and representations of each layer. A protocol stack for the layered architecture is structured considering management, operational and security planes. A network model of IoV is proposed based on the three network elements including cloud, connection and client. The benefits of the design and development of IoV are highlighted by performing a qualitative comparison between IoV and VANETs. Finally, the challenges ahead for realizing IoV are discussed and future aspects of IoV are envisioned.
Article
Full-text available
In this paper, we present the TUM RGB-D bench-mark for visual odometry and SLAM evaluation and report on the first use-cases and users of it outside our own group. The benchmark contains a large set of image sequences recorded from a Microsoft Kinect associated with highly accurate and time-synchronized ground truth camera poses from an external motion capture system. The dataset consists in total of 39 sequences that were recorded in different environments and cover a large variety of scenes and camera motions. In this work, we discuss and briefly summarize the evaluation results of the first users from outside our group. Our goal with this analysis is to better understand (1) how other researcher use our dataset to date and (2) how to improve it further in the future.
Article
To improve safety on the roads, next generation vehicles will be equipped with short range communication technologies. The many applications enabled by such communication will be based on a continuous broadcast of information about the own status from each vehicle to the neighborhood, often referred as cooperative awareness or beaconing. Although the only standardized technology allowing direct vehicle-to-vehicle (V2V) communication has been IEEE 802.11p until now, the latest release of LTE is including advanced device-to-device features designed for the vehicular environment (LTE-V2V), making it a suitable alternative to IEEE 802.11p. Advantages and drawbacks are being considered for both technologies, and which one will be implemented is still under debate. The aim of this paper is thus to provide an insight into the performance of both technologies for cooperative awareness and to compare them. The investigation is performed analytically, through the implementation of novel models for both IEEE 802.11p and LTE-V2V able to address the same scenario, with consistent settings and focusing on the same output metrics. The proposed models take into account several aspects that are often neglected by related works, like hidden terminals and capture effect in IEEE 802.11p, the impact of imperfect knowledge of vehicles position on the resource allocation in LTE-V2V, and the various modulation and coding scheme combinations that are available in both technologies. Results show that IEEE 802.11p allows good performance at short distances, even in extreme density conditions, whereas the newer LTE-V2V becomes preferable if longer distances are targeted.
Conference Paper
Offloading computationally expensive Simultaneous Localization and Mapping (SLAM) task for mobile robots have attracted significant attention during the last few years. Lack of powerful on-board compute capability in these energy constrained mobile robots and rapid advancement in compute cloud access technologies laid the foundation for development of several Cloud Robotics platforms that enabled parallel execution of computationally expensive robotic algorithms, especially involving multiple robots. In this work the Cloud Robotics concept is extended to include the current emphasis of computing at the network edge nodes along with the Cloud. The requirements and advantages of using edge nodes for computation offloading over remote cloud or local robot clusters are discussed with reference to the ETSI 'Mobile-Edge Computing' initiative and OpenFog Consortium's 'OpenFog Architecture'. A Particle Filter algorithm for SLAM is modified and implemented for offloading in a multi-tier edge+cloud setup. Additionally a model is proposed for offloading decision in such a setup with experiments and results demonstrating the efficacy of the proposed dynamic offloading scheme over static offloading strategies.