Conference PaperPDF Available

Network enabled cooperation of autonomous vehicles: A communications perspective

Authors:
  • OceanScan - Marine Systems & Technology
  • Netherlands Red Cross
Network Enabled Cooperation of Autonomous Vehicles:
A Communications Perspective
Jos´
e Pinto1, Manuel Ribeiro1, Stefania Giodini2, Patrick L’Hoir3,
Alessandro Sartore4, J. M. Giron-Sierra5, Jos´
e Braga6, Jo˜
ao Sousa1
info@necsave.org
Abstract In this paper we present the NECSAVE sys-
tem which allows the control of heterogeneous unmanned
vehicles under challenging environments by a single
human operator by providing it with the desired high-
level objectives. The described software stack takes care
of decomposing the high-level objectives into assignable
tasks and (re)distributing the workload among team
members according to their announced capabilities and
eventual failures. NECSAVE is robust to system failures
by supporting decentralized planning and adapting allo-
cation in face of platform losses.
The system was validated in multiple field trials
where NECSAVE was used for the coordination of
heterogeneous vehicles under challenged communication
environments, namely the sea using radio (at surface)
and acoustic communications (underwater).
I. INTRODUCTION
EDA-NECSAVE (Network Enabled Cooperation
System of Autonomous Vehicles) is a 3 year long
project supported by the European Defense Agency
with the objectives of developing, testing and evaluat-
ing different tools and methodologies for the Network
Enabled Capability (NEC) of heterogeneous unmanned
vehicles for operations in challenging environments.
The constituent partners of the project are the Uni-
versity of Porto (Portugal), Port of Leix˜
oes (Portugal),
OceanScan (Portugal), Calzoni (Italy), UCM (Spain),
RMA (Belgium) and TNO (The Netherlands). In the
context of the EDA-NECSAVE project, a software
stack compatible with different onboard platforms has
been developed which supports coordination and plan-
ning for teams of heterogeneous robots.
In this paper we present the NECSAVE system
which allows human operators to task an entire net-
1LSTS-FEUP, Universidade do Porto, Portugal
2TNO, the Netherlands
3Royal Military Academy, Belgium
4L3 Calzoni, Italy
5Universidad Complutense de Madrid, Spain
6OceanScan - Marine Systems and Technology, Portugal
work of vehicles by providing high-level objectives
[1]. The NECSAVE software stack takes care of de-
composing high-level objectives into assignable tasks
and (re)distributing the workload among team members
according to their announced capabilities and eventual
failures. NECSAVE is robust to system failures by sup-
porting decentralized planning and adapting allocation
in face of platform losses.
The system has been tested in the field with teams of
heterogeneous vehicles including Autonomous Under-
water Vehicles (AUVs), Autonomous Surface Vehicles
(ASVs) and static communication gateways (no actua-
tion) from the different partners. Results validated the
applicability of NECSAVE to the coordination of het-
erogeneous vehicles under challenged communication
environments, namely the sea using radio (at surface)
and acoustic communications (underwater). NECSAVE
can be used to support multi-vehicle operations in
a range of scenarios that require coordination and
adaptation under challenged communications, such as
the following:
1) Mine hunting: Scan an area to find mines; mine
hunting can include marking and neutralization
of mines; requires sensors to detect and weapons
to neutralize mines.
2) Minesweeping: Work on the operation area to
demine using mine countermeasures; requires
sensors to detect mines and mine countermea-
sures to generate a ship like signature or to cut
moored cables.
3) Harbour protection: Patrol the area and monitor
the suspect activities from submarines, ships or
aircrafts; requires sensors to detect other vehicles
and deterrence weapons to protect the harbour.
4) Ship protection: Follow a ship and turn around
to detect and protect against submarines, ships or
aircrafts; requires: sensors to detect other vehicles
and deterrence weapons to protect the ship.
5) Search and rescue: Scan an area to detect humans
involved in an accident.
II. RELATED WOR K
There are various groups doing research in coop-
erating teams of heterogeneous vehicles. In [2], the
authors have used teams of ground and aerial robots
for mapping disaster areas. However, they have not
considered automatic resolution of vehicle failures and
assume good communication links from the base to all
robots which may not always be the case.
The work in [3] allows human operators to control
teams of heterogeneous underwater vehicles by having
a centralized planner that generates individual vehicle
plans based on user-provided (high-level) objectives.
However, the behaviors that are generated are executed
with no feedback to the central planner which is not
feasible in dynamic environments or when vehicle
failures are possible.
The approach in [4] addresses limited communica-
tions and uses a centralized planner to react to different
kinds of failures. However, correct behavior depends
on intermittent communications with the centralized
planner which still is a single point of failure for the
system.
Our work addresses unreliable communications and
robustness by allowing distributed and decentralized
planning. In fact, all nodes are symmetry in that they
all can plan actions based on local knowledge and last
known global objectives.
III. ARCHITECTURAL APP ROACH
One of the objectives of NECSAVE is to produce
a platform-agnostic system which is capable not only
of supporting different hardware but most importantly
exploit these differences in order to produce behaviors
which can benefit from the vehicle’s specific advan-
tages towards reaching the goals. For this, all platforms
run a similar piece of software which extracts the
hardware capabilities and disseminates them to other
platforms.
Hardware capabilities include the capability to per-
form certain maneuvers such as moving towards points
or scanning an area of interest using a range of sensors.
For each contained sensor the platforms indicate its ex-
istence as well as its coverage rate (how many squared
meters are covered in one hour, in average). This allows
the creation of plans which are time-optimized for the
vehicles currently available. The hardware capabilities
closely match the types of actions that can be included
Fig. 1. Communication between nodes is done via the Platform
modules of each node.
in behaviors requests that are part of a team-level plan.
How the actual actions are executed, however, are the
responsibility of the platform owners.
To support multiple platforms, the NECSAVE soft-
ware stack takes care of ad-hoc data dissemination,
planning and coordinated execution. As such, every
node in the network runs, at least, 4 NECSAVE mod-
ules: Mission Planner,Vehicle Planner,Perception
and Platform Adapter which all use the common
NECBus library for accessing shared functionalities
and communicating with other modules (Fig. 1).
A. NECBus Library
For inter-module communications inside a NEC-
SAVE stack, a base C++ library (NECBus) has been
developed which uses the ØMQ1framework to allow
both publish/subscribe and synchronous (RPC) com-
munication between the modules. The same library
also contains various utilitary functionalities such as
geographic calculations, configuration parsing and sim-
plified path planning.
Using a shared configuration parsing library is im-
portant across all modules as it allows using a single
configuration file for all the modules. Our approach,
inspired by the pAntler software which is part of
MOOS [5], allows using shared configuration files
across all platforms with common settings. Then, each
platform configuration file will override some specific
configurations such as the local platform identifier.
Having a normalized way of loading configurations
for the entire system greatly simplifies the process of
starting trials and simulations.
B. Mission Planner Module
This module is in charge of listening to the available
team capabilities and high-level objectives and produc-
1http://zeromq.org/
ing or adapting a team-level feasible plan whenever the
objectives or the perceived team capabilities change.
The produced plan is a (possibly empty) list of
behaviors to be executed by each platform. The plan
needs to be synchronized with all intervening platforms
either by communicating it reliably or checking that the
plans produced by other planning modules are similar.
C. Vehicle Planner Module
The vehicle planner is right down below from the
mission planner in control hierarchy. Its responsibility
is to decompose behaviors into platform-specific ac-
tions. For instance, if the underlying platform is not
able of coverage path planning, the vehicle planner
produces a series of Move actions that result in the
coverage of the area to be surveyed.
Also very importantly, the Vehicle Planner is in
charge of monitoring execution of the behaviors and
computing its progress. If the platform is stuck for some
reason, this can trigger a replanning by the Mission
Planner as long as the progress is correctly updated.
Mission Planner: In charge of producing and/or
negotiating a team-level plan.
Vehicle Planner: In charge of adapting and ex-
ecuting assigned tasks as well as supervising the
state of the underlying platform.
Perception: Keeps track of all information includ-
ing the local state and synchronization between
local and remote states.
Platform Adapter: In charge of interfacing the
hardware and implementing required actuation,
sensing and (inter-platform) communications.
By using a standard (ØMQ) communications mech-
anism, modules can also be implemented in other
languages to extend the NECSAVE system, as long
as they use the shared NECSAVE communications
protocol described next.
IV. COMMUNICATIONS IN NECSAVE
All modules use a well defined message-oriented
protocol (NMP) for exchanging information. The same
protocol is used for inter-module and for inter-platform
communication.
The NMP (NECSAVE Message Protocol) is a
message-oriented protocol where all messages have
a common header format and type-specific payload.
The protocol is defined in an XML file and from it
bindings can be generated for NECBus and also for
other languages (currently only Java and C++).
Message header includes the following fields:
1) Protocol Version: Denotes the packet API ver-
sion and can be used to deduce the byte order of
the sending host.
2) Message Type: The identification number of the
message contained in the payload. This field
is used for correct message interpretation and
deserialization.
3) Body size: Size of the body.
4) Time stamp: The time when the packet was sent,
as seen by the packet dispatcher.
5) Source Platform: Platform that generated this
message.
6) Source Module: Module (from the NECSAVE
stack) that generated this message.
7) Destination Platform: Platform this message is
addressed to.
8) Destination Module: Module this message is
addressed to.
9) Medium: Medium over which this message has
or is to be transmitted.
10) Expiration Time: After this time the message
will not be retransmitted as is discarded.
11) Original Source: When the message is retrans-
mitted (multi-hop networking), this field retains
the identifier of the original message source.
After the header information, other message-specific
information is encoded in the message. There are
messages that encode high-level objectives, platform
capabilities, behavior assignments, behavior execution
progress, platform state, etc.
Most of the messages are not transmitted between
platforms with the exception of the following:
1) PlatformState: This message is broadcasted pe-
riodically by every platform to disseminate its
state. State information includes the identifier
and location of the platform and a checksum of
the currently perceived global network capabilties
and plan version.
2) Plan: A plan is generated by a MissionPlanner
module and disseminated over the network using
this message. A plan is simply lists of behaviors
allocated to each platform. Whenever there is a
preceived failure in the system, the planner will
generate a new plan to be disseminated.
3) PlatformInfo: This message is transmitted by
each platform module to the planner (which can
be local or remote). The planner must be aware
of the capabilities of all the platforms in order to
allocate behaviors to the entire team. If a vehicle
fails (gracefully) it will update its capabilities and
disseminate them again to the planner. Capabili-
ties include both maneuvers that can be executed
as well as sensors and communication means
available.
4) PlatformPlanProgress: This message is used to
update all intervening platforms about the current
progress of execution of the platform. This al-
lows the planner(s) to update its global execution
progress and potentially replan accordingly.
The Perception module is in charge of deciding
when and how to exchange information with other
peers (different platforms) and for maintaining general
consensus about the network state which includes its
plan, capabilities and execution progress of the entire
team. The way this is done varies according to a
system-wide configurable topology and link availabil-
ity. Actual transmission of information between plat-
forms is delegated to the different platform adapters
and according to available communication capabilities.
Some information can be sent unreliably such as the
PlatformState which is broadcast periodically but other
such as the plan and platform capabilities are to be sent
reliably.
A. Inter-platform communications
In all topologies, when information needs to be
transmitted reliably the information is retransmitted
until there is an acknowledged transmission or after
some timeout. When a timeout occurs, the node is
removed from the list of peers and its capabilities are
emptied (assuming a failure).
Figure 2 depicts the interaction diagrams for the
different topologies. As it can be seen from the figure,
some messages like PlatformInfo are aggregated into a
team-level data type (AllPlatformInfo). All team-level
data-types need to be synchronized across the platforms
to maintain a correct perception of the network.
The type of interactions in mesh topology are similar
to those in tree topology. However, while in tree topol-
ogy follows a certain static hierarchy, in mesh topology
all nodes communicate with each other whenever they
notice their information is out of sync with their peers.
1) Star Topology: In this mode, there is a single
node in the network which is in charge of generating
the plan to the entire team of vehicles. Initially, the
Perception module checks if it’s running on ”slave”
node in which case sends the local capabilities to
the node announced as the ”master” using reliable
communications. It is then up to the ”master” to task
the entire network of vehicles by generating a team-
level plan. While executing the plan, all nodes keep
broadcasting their capabilities and execution progress
so that if there is a slow progress in one of the nodes
or its capabilities change, the ”master” can retask the
network by sending a new plan.
2) Tree Topology: The way this topology is imple-
mented is very similar to IV-A.1. Main difference is
that in this case there is one special node (gateway)
in charge of maintaining the communications between
the ”master” and other ”slave” nodes. As such, slaves
communicate reliably only with the gateway which then
fuses this information and retransmits it reliably to the
”master”.
3) Mesh Topology with Centralized Control: In this
case all nodes use a similar communications mech-
anism. They all broadcast periodically their version
of the plan, progress and capabilities and whenever
an older version of the information is detected, they
transmit the newer information to that node. In this
topology, the ”master” will update the plan when
needed (capability changes or change of objectives)
being the plan disseminated by all nodes.
4) Mesh Topology with Decentralized Control: In
this mode of operation, every platform is a ”master”
node. This means that all nodes produce plans for the
entire network. In order to reach consensus about the
plan that the network shall execute, we have devised
an information merging algorithm that generates in a
deterministic way a new, conflict-free plan whenever
new information is received. This allows adding and
removing platforms during execution as the newly
added platforms will merge their (empty) plan with the
one received by any of the peers. Plan merging is done
in a way that also shares workload among the nodes.
B. Decentralized state consensus
When using Mesh topology with decentralized con-
trol, all platforms need to keep an updated view of
the overall network state (capabilities and execution
progress). For this, we have created a compact state
representation format that can be broadcasted periodi-
cally. The compact state includes information such as
the Platform Identifier,known allocation version of
each platform and allocation state of each behavior:
unallocated, finished or allocated to a specific platform.
If a vehicle faces a failure, it marks all its assigned
behaviors as unallocated and updates its allocation
version. This means that the information about its own
allocations is more up-to-date than any other allocations
in the network and shall be considered. Whenever
Fig. 2. Interaction diagrams for the different topologies
another platform receives a state with behaviors un-
allocated, it will allocate the behaviors to itself (when
they are feasible).
If, due to communication limitations, two platforms
become allocated to overlapping behaviors and they
become aware of it (receive that information), there is a
deterministic way to evenly split the behaviors between
them based on the relative order of their identifiers and
allocation versions of other peers. As a result, they
always reach a conflict-free allocation.
As a result of all the previous rules, there is a
deterministic way of merging all incoming compressed
platform states which means that information can be
sent in broadcast mode to the entire network. This type
of state updates is based upon the rules of Conflict-Free
Replicating Data Types [6].
V. FI ELD TRIALS
The NECSAVE system prototype has been tested
multiple times at the Porto harbor using LAUV vehicles
with different payload capabilities and all equipped
with Evologics S2CR18/34 acoustic modems. More-
over, the same system was tested at Lisbon Naval
base with AUVs and ASVs from the different partners
for both heterogeneous sidescan surveys and pattern
Fig. 3. NECSAVE-compatible vehicles used for the sea trials. On
top-left the autonomous zodiac from UCM, on top-right Ocean-
scan’s DURIUS ASV, on bottom-left 3 LAUV vehicles from UPorto
and on bottom-right a Calzoni’s mini-ranger ASV.
formation behaviors with success (Fig. 3). All platforms
were capable of using Wi-Fi communications but only
the AUVs and gateways were capable of acoustic
communications.
To control the network, a Neptus [7] plug-in com-
patible with NECSAVE (using ØMQ) was developed
Fig. 4. Snapshot of the Neptus interface showing 2 ASVs and 1
AUV covering a shared area which was initally split evenly by the
planner. Data from AUV arriving only via acoustic mode as it was
underwater.
that is able to receive all broadcast communications
and request high-level objectives. The plug-in allows
visualization of the states being reported by the network
and also simulate failures (force capability changes).
Fig. 4 shows the interface developed for NECSAVE
where different behaviors are represented in the map
and color-coded to each vehicle. If a vehicle fails during
the survey, the area left to survey is reallocated to the
remaining vehicles.
VI. RESULTS
The NECSAVE system has been validated in the field
in harbor, open sea and rivering estuaries. Acoustic and
wi-fi communications reliability varied a lot among the
different sites. Results have shown that reliable com-
munications of large data underwater (such as global
plans) imposes communication latencies in the orders
of minutes due to retransmissions and send/receive
collisions. However, when vehicles surface sufficiently
close they can quickly transmit all missing information
using Wi-Fi. Moreover, when using compressed state
representations and conflict-free merging of data sent
in broadcast, the network was capable of adapting very
fast (less than 30 seconds) to failures and new platforms
entering the system.
Regarding the different network topologies, if using
reliable communications Start topology is the more
reliable and faster approach as it requires the least
number of messages to be transmitted. However, for
large network deployments or deployments where some
links are not always available (obstacles), the Mesh
topology is more robust to failures and thus the most
effective.
Centralized control again is applicable only when
there is reliable communication between the master and
all the slaves. In case of a master failure, the entire
system breaks down. Decentralized control, and most
importantly when all nodes are masters using mesh
topology is the most fault-tolerant approach and, when
using, CRDT data types for state synchronization also
one of the faster approaches to converge. The only
limitation of this last approach is that all the assignable
behaviors or high-level objectives need to be known by
the network before any platform dives.
VII. CONCLUSIONS AND FUTURE WOR K
The EDA-NECSAVE project has produced a very
successful base software for the coordination of hetero-
geneous platforms. Its application was already demon-
strated for mine-hunting and mine-sweeping scenarios
using AUVs and ASVs together. In the future, the
partners want to explore the applicability of the same
system to other more complex scenarios such as harbor
protection and anti-piracy. Moreover, we would also
like to explore more advanced planning techniques
producing time-windowed behaviors so that commu-
nications are not only opportunistic but also planned.
REFERENCES
[1] The NECSAVE Consortium, “Network enabled cooperation
system of autonomous vehicles,EDA Project, 2013–2017.
[2] N. Michael, S. Shen, K. Mohta, Y. Mulgaonkar, V. Kumar,
K. Nagatani, Y. Okada, S. Kiribayashi, K. Otake, K. Yoshida
et al., “Collaborative mapping of an earthquake-damaged build-
ing via ground and aerial robots,” Journal of Field Robotics,
vol. 29, no. 5, pp. 832–841, 2012.
[3] L. Chrpa, J. Pinto, M. A. Ribeiro, F. Py, J. Sousa, and K. Rajan,
“On mixed-initiative planning and control for autonomous
underwater vehicles,” in Intelligent Robots and Systems (IROS),
2015 IEEE/RSJ International Conference on. IEEE, 2015, pp.
1685–1690.
[4] F. Py, J. Pinto, M. A. Silva, T. A. Johansen, J. B. de Sousa,
and K. Rajan, “Europtus: A mixed-initiative controller for
multi-vehicle oceanographic field experiments,” in Int. Symp.
Experimental Robotics, 2016.
[5] P. M. Newman, “Moos-mission orientated operating suite,”
Massachusetts Institute of Technology, Tech. Rep, vol. 2299,
no. 08, 2008.
[6] M. Letia, N. M. Preguic¸a, and M. Shapiro, “Crdts: Consistency
without concurrency control,CoRR, vol. abs/0907.0929, 2009.
[Online]. Available: http://arxiv.org/abs/0907.0929
[7] P. S. Dias, R. M. Gomes, and J. Pinto, “Mission planning
and specification in the neptus framework,” in Robotics and
Automation, 2006. ICRA 2006. Proceedings 2006 IEEE Inter-
national Conference on. IEEE, 2006, pp. 3220–3225.
Chapter
Navies, Industries and Research are investigating how to move to future mine counter measure operations where of multiple autonomous underwater vehicles work collaboratively with minimal human interaction. Modelling and simulation is proving to be a capable support to test autonomy by the adoption, in particular, on interoperable environments that may comprise of models, hardware and software-in-the-loop. Industry and NATO standards exist for federated simulation but the standards do not provide detailed guidance to carry out lean, comprehensive and consistent design, development and testing of distributed simulation for autonomy.
Conference Paper
Full-text available
Our research concerns the mixed-initiative coordination of air and underwater vehicles interacting over inter-operated radio and underwater communication networks for novel oceanographic field studies. In such an environment, operating multiple vehicles to observe dynamic oceanographic events such as fronts, plumes, blooms and cetaceans has required that we design, implement and operate software, methods and processes which can support ephemeral and unpredictable observations (including those of moving animals) in real-world settings with substantial constraints. We articulate an approach for coordinated measurements using such platforms, which relate directly to task outcomes. We show the use and operational value of a new Artificial Intelligence (AI) based mixed-initiative system, EUROPtus, for handling multiple platforms from a recent field experiment in open waters of the mid-Atlantic.
Conference Paper
Full-text available
Our research concerns the mixed-initiative coordination of air and underwater vehicles interacting over inter-operated radio and underwater communication networks for novel oceanographic field studies. In such an environment, operating multiple vehicles to observe dynamic oceanographic events such as fronts, plumes, blooms and cetaceans has required that we design, implement and operate software, methods and processes which can support ephemeral and unpredictable observations (including those of moving animals) in real-world settings with substantial constraints. We articulate an approach for coordinated measurements using such platforms, which relate directly to task outcomes. We show the use and operational value of a new Artificial Intelligence (AI) based mixed-initiative system, EUROPtus, for handling multiple platforms from a recent field experiment in open waters of the mid-Atlantic.
Conference Paper
Full-text available
Supervision and control of Autonomous underwater vehicles (AUVs) has traditionally been focused on an operator determining a priori the sequence of waypoints of a single vehicle for a mission. As AUVs become more ubiquitous as a scientific tool, we envision the need for controlling multiple vehicles which would impose less cognitive burden on the operator with a more abstract form of human-in-the-loop control. Such mixed-initiative methods in goal-oriented commanding are new for the oceanographic domain and we describe the motivations and preliminary experiments with multiple vehicles operating simultaneously in the water, using a shore-based automated planner.
Conference Paper
Full-text available
The C3I (command, control, communication and information) Neptus framework which is being developed at the Underwater Systems and Technology Laboratory (USTL/LSTS) is presented. Neptus is a modular mixed initiative framework (human operators in the control loop) for the operation of heterogeneous teams of vehicles such as autonomous and remotely operated underwater, surface, land, and air vehicles. Neptus is composed of mission and vehicle planning, supervision, and post-mission analysis modules which are provided as services across a network. This paper focus mainly on the mission definition module with the presentation of MDL - a XML based language for mission definition
Article
Full-text available
A CRDT is a data type whose operations commute when they are concurrent. Replicas of a CRDT eventually converge without any complex concurrency control. As an existence proof, we exhibit a non-trivial CRDT: a shared edit buffer called Treedoc. We outline the design, implementation and performance of Treedoc. We discuss how the CRDT concept can be generalised, and its limitations.
Article
This paper is about simple to use, extensible software for mobile robotic research. It is concerned with a project called MOOS – an acronym for Mission Oriented Operating Suite. MOOS is an umbrella term for a set of libraries and applications designed to fa-cilitate research in the mobile robotic domain. The spectrum of functionality provided ranges over low-level, multi-platform communications, dynamic control, high precision navigation and path planning, concurrent mission task arbitration and execution, mis-sion logging and playback. The first part of the paper describes the underlying philosophy of MOOS and the resulting perceived benefits. The work then moves on to describe the details of the design and implementations of core system components. There then follows a set of high level descriptions of principal mission-oriented MOOS processes. Collectively these processes constitute a resilient, distributed and coordinated suite of software suitable for in-the-field deployment of sub-sea and land research robots.
Article
We report recent results from field experiments conducted with a team of ground and aerial robots engaged in the collaborative mapping of an earthquake-damaged building. The goal of the experimental exercise is the generation of three-dimensional maps that capture the layout of a multifloor environment. The experiments took place in the top three floors of a structurally compromised building at Tohoku University in Sendai, Japan that was damaged during the 2011 Tohoku earthquake. We provide details of the approach to the collaborative mapping and report results from the experiments in the form of maps generated by the individual robots and as a team. We conclude by discussing observations from the experiments and future research topics. © 2012 Wiley Periodicals, Inc.
Network enabled cooperation system of autonomous vehicles
  • Necsave The
  • Consortium
The NECSAVE Consortium, "Network enabled cooperation system of autonomous vehicles," EDA Project, 2013-2017.