Conference PaperPDF Available

The Skyeye Team Participation in the 2020 Mohamed Bin Zayed International Robotics Challenge

Authors:

Abstract and Figures

This article presents a detailed report of the systems developed by the Skyeye team for the 2020 Mohamed Bin Zayed International Robotics Challenge (MBZIRC). This biennial competition aims to encourage researchers to develop fully autonomous systems for state-of-the-art applications in the area of field robotics. In this work, the main hardware and software components for the Unmanned Aerial Vehicle (UAV) and Unmanned Ground Vehicle (UGV) platforms to be used in the MBZIRC competition are described. These components are based on multidisciplinary fields of expertise encompassing computer vision, state estimation, planning, and control techniques. A series of preliminary tests demonstrate that the proposed system strategies are capable of achieving the overall tasks required by Challenge 1 and Challenge 3 of the competition. The main results include autonomously detecting, interacting , and retrieving tethered items from UAVs in motion; and heterogeneous robotic cooperation for autonomous fire detection and extinguishment using UAVs and UGVs.
Content may be subject to copyright.
Noname manuscript No.
(will be inserted by the editor)
The Skyeye Team Participation in the 2020
Mohamed Bin Zayed International Robotics
Challenge
Ramon A. Suarez Fernandez ·Alejandro
Rodriguez-Ramos ·Adrian Alvarez ·
Javier Rodr´ıguez-V´azquez ·Hriday
Bavle ·Lu Liang ·Miguel Fernandez ·
Alberto Rodelgo ·Jose Antonio Cobano ·
David Alejo ·Domingo Acedo ·Rafael
Rey ·Simon Martinez-Rozas ·Martin
Molina ·Luis Merino ·Fernando
Caballero ·Pascual Campoy
Received: date / Accepted: date
Abstract This article presents a detailed report of the systems developed
by the Skyeye team for the 2020 Mohamed Bin Zayed International Robotics
Challenge (MBZIRC). This biennial competition aims to encourage researchers
to develop fully autonomous systems for state-of-the-art applications in the
area of field robotics. In this work, the main hardware and software compo-
nents for the Unmanned Aerial Vehicle (UAV) and Unmanned Ground Vehicle
(UGV) platforms to be used in the MBZIRC competition are described. These
components are based on multi-disciplinary fields of expertise encompassing
computer vision, state estimation, planning, and control techniques. A series
of preliminary tests demonstrate that the proposed system strategies are ca-
pable of achieving the overall tasks required by Challenge 1 and Challenge 3 of
the competition. The main results include autonomously detecting, interact-
ing, and retrieving tethered items from UAVs in motion; and heterogeneous
robotic cooperation for autonomous fire detection and extinguishment using
UAVs and UGVs.
Keywords Autonomous Systems ·MBZIRC ·UAV ·UGV ·Skyeye
R. A. Suarez Fernandez, A. Rodriguez-Ramos, A. Alvarez, J. Rodriguez-Vazquez, H. Bavle,
L. Liang, M. Fernandez, A. Rodelgo, M. Molina, P. Campoy
Universidad Polit´ecnica de Madrid, c/ Jose Gutierrez Abascal 2, 28006, Madrid, Spain.
E-mail: fernandez.suarez.ramon@gmail.com
J.A. Cobano, D. Alejo, D. Acevedo, R. Rey, S. Martinez-Rozas, L. Merino, F. Caballero
Universidad Pablo Olavide, Ctra. de Utrera, 1, 41013 Sevilla, Spain.
2 Ramon A. Suarez Fernandez et al.
1 Introduction
In the past decades, a wide variety of competitions, such as the International
Micro Aerial Vehicle competition [1] and the International Aerial Robotics
Competition [2], have served as an important motivation for researchers to de-
velop state-of-the-art technological solutions in the field of unmanned robotic
systems. Similar to other major competitions, the Mohamed Bin Zayed In-
ternational Robotics Challenge (MBZIRC) celebrates a biennial international
robotics competition that addresses a set of current technologically demanding
challenges, in an effort to extend the state-of-the-art in robotic applications
and demonstrate the capabilities of autonomous robotic systems.
The first MBZIRC competition was held at the YAS Marina Circuit in
Abu Dhabi, UAE, in March 2017, with 27 international teams and an ac-
colade of 5 Million USD for prize and team sponsorship. The first series of
challenges consisted of several scenarios focusing on autonomous landing on
moving platforms using Unmanned Aerial Vehicles (UAVs) and robot collab-
oration between UAVs and Unmanned Ground Vehicles (UGVs).
The latest edition of the MBZIRC will take place on February 20-25, 2020
at the Abu Dhabi National Exhibition Center in Abu Dhabi, UAE. It will con-
sist of three individual challenges and a single Grand Challenge. In Challenge
1, a team of UAVs will have to autonomously track and interact with a set of
objects both static and dynamic. Challenge 2 requires a team of UAVs and a
UGV to collaborate and construct a series of structures by picking and trans-
porting brick-shaped objects. For Challenge 3, a team of UAVs and a UGV
will have to autonomously extinguish a series of simulated fires in an urban
high rise building firefighting scenario. Finally, the Grand Challenge requires
UAVs and the UGV to simultaneously perform the tasks performed in the
previous challenges.
Given such a complex amount of tasks to be performed for each challenge,
a multi-disciplinary consortium of three research groups has been created to
tackle the MBZIRC competition. This paper is a systems article providing a de-
tailed report of the solutions proposed for challenge 1 by the Computer Vision
and Aerial Robotics (CVAR) group at the Universidad Polit´ecnica de Madrid,
as well as the solutions proposed for challenge 3 by the Service Robotics Lab
(SRL) group from the Universidad Pablo Olavide in Sevilla, Spain. In this
work, the integration and development of the major hardware and software
components in the areas of computer vision, state estimation, 3D mapping,
planning, and control for challenges 1 and 3 will be presented.
The rest of this paper is organized as follows: Sections 2 and 3 present the
overall mechanical and hardware designs for the UAV and UGV platforms to
be used, as well as the different approaches developed to solve the requirements
of the individual challenges 1 and 3, respectively. Thereafter, in Section 4, the
preliminary results of experimental tests are shown, whilst in Section 5 the
conclusions are presented.
The Skyeye Team Participation in the 2020 MBZIRC 3
(a) (b)
Fig. 1 Aerial vehicle platform used for challenge 1. (a) Modified DJI Matrice 210 RTK v2
with custom gripping mechanism. (b) 3D model of the custom gripping mechanism designed
for item retrieval.
2 Challenge 1: Autonomous Interaction with Moving Objects
The overall objective in challenge 1 of the MBZIRC competition is to au-
tonomously perform a series of interaction tasks using aerial vehicles. First,
a UAV (or a team of UAVs) must detach and retrieve an item (foam ball of
approximately 11 cm in diameter), which will be tethered to a target UAV in
motion. The target UAV will start flying from a random location within a 100
m×40 m outdoor flight arena, following a lemniscate of Bernoulli 3D curve
with a maximum velocity of 8 m/s. Once the item has been removed from
the tether, it must then be delivered to a pre-specified 10 m ×10 m landing
location inside the arena.
Furthermore, the UAV (or a team of UAVs) will have to locate and physi-
cally interact with a set of static items (balloons of 45 cm in diameter), which
are of known color, randomly distributed within the outdoor flight arena,
and tethered to the ground. For added difficulty, once these static items are
detected, the UAV will have to navigate and destroy each item using any com-
ponent installed on-board. In this section, the main system components that
have been developed and integrated to solve the key problems encountered in
challenge 1 will be presented.
2.1 Aerial Vehicle Platform
The aerial platform used for challenge 1 of the MBZIRC competition will be a
modified version of the DJI Matrice 210 RTK v2 UAV, shown in Fig. 1(a). This
robust platform was chosen as the main aerial vehicle due to the demanding
flight performance required by the different tasks, e.g., flights more than 10
m/s, large payload, and extended flight times. The UAV will be equipped with
a Real-time kinematic (RTK) Global Positioning System (GPS), a guidance
system, and a DJI Zenmuse X4S on-board camera featuring a 20MP, 1” sensor
4 Ramon A. Suarez Fernandez et al.
and 84 degree FOV high-resolution lens. Additionally, the aerial platform will
be equipped with the Single Board Computer (SBC) NVIDIA Jetson AGX
Xavier with an 8-core ARM v8.2, 64-bit CPU running Ubuntu Linux 18.04
Bionic Beaver for on-board computing and image processing. All communica-
tions will be handled using the Melodic Morenia distribution of the Robotic
Operating System (ROS) middleware.
As previously mentioned, the objective of challenge 1 is to retrieve an
item which is tethered to a separate target UAV in motion. To this effect,
the DJI Matrice 210 UAV will be equipped with a custom designed gripping
mechanism, shown in Fig. 1(b), which will enable grasping of the item. The
gripper is divided in two main parts: a grasping base and a securing arm. The
grasping base is composed of 4 ”fingers”, 3D printed in ABS plastic, interlocked
by carbon fibre and nylon threads, which will release the item from the tether.
Once the item has been retrieved, the securing arm will close to maintain the
item safe inside the grasping base.
2.2 System Architecture
In order to perform the complex tasks encountered in challenge 1 of the
MBZIRC, a robust and versatile system architecture is essential for efficient
development and validation of both software modules and hardware interfaces.
Therefore, based on the previous results obtained in autonomous platform de-
velopment [6][7], the system architecture that is proposed in this work has
been developed using the Aerostack1.
1Aerostack Webpage: https://github.com/Vision4UAV/Aerostack
Fig. 2 System architecture design used for challenge 1 (based of the Aerostack software
framework). Each layer contains different functionalities that have been newly integrated
into the software framework.
The Skyeye Team Participation in the 2020 MBZIRC 5
Aerostack is an open-source multi-purpose software framework for the de-
velopment of autonomous multi-robot unmanned aerial systems created by
the Computer Vision and Aerial Robotics (CVAR) group at the Universi-
dad Polit´ecnica de Madrid. The Aerostack modules are mainly implemented
in Python and C++ language and is based on ROS for inter-communication
between the different components and layers. To provide higher degrees of au-
tonomy, Aerostack’s software framework integrates state-of-the-art concepts
of intelligent, cognitive and social robotics, based on six main layers: physical,
reactive, executive, deliberative, reflective, and social. For a deeper under-
standing of Aerostack, we refer the reader to [8].
Using the Aerostack software framework, a new system architecture design
was developed for the tasks presented in challenge 1. Fig. 2 shows the new
functionalities that have been implemented in this work for extending the and
improving several systems in different layers of the software framework, such as
the Feature Extraction, Situation Awareness, Planning, Executive and Motor
systems.
2.3 Feature Extraction System
A critical element required to achieve the tasks proposed in challenge 1 will
be the capability of robustly detecting the target UAV and items, as well as
accurately estimating their location with respect to the aerial platform.
The Feature Extraction System (FES) in the Reactive Layer (see Fig. 2) of
the system architecture incorporates all of the perception software modules to
be used for challenge 1, which are based on computer vision and deep learning
techniques. The FES can be divided into three base modules, which are used
during different stages of the challenge: (i) the Far UAV Detector, (ii) the
Close UAV Detector, and (iii) the Item Detector.
2.3.1 Far UAV Detector
As mentioned in the previous section, the challenge 1 of the MBZIRC compe-
tition will be carried out in a 100 m ×40 m outdoor flight arena. Therefore, in
order to detect the target UAV in motion, with such a wide scope area, a Far
UAV detector module has been implemented. The Far detector is in charge of
the detection of moving objects when they appear small in the image plane,
e.g., the case of a small UAV flying in a far distance from the camera.
In these type of scenarios, where there are no semantics that can be ex-
ploited, other techniques such as background motion estimation and moving
object detection can be incorporated, such as in [9]. In the first stage, the
image frames acquired from the on-board camera are used to estimate the
background motion between frames. Considering a globally smooth motion
with camera projection, the background motion can be calculated using a per-
spective transform model. Since the background and the moving objects in
6 Ramon A. Suarez Fernandez et al.
Fig. 3 Diagram of the proposed method for the Far UAV detector in challenge 1.
the image can be assumed to have fairly different motion models, the mov-
ing objects in a far plane can be obtained by compensating the motion of the
background. After subtracting the background from the original image, a deep
learning classifier can be used to detect the candidates for moving objects in
the highlighted patches of the background subtracted image. Finally, to find
the local motion of the moving objects, a Lucas-Kanade optical flow algorithm
can be applied to the output of the deep learning classifier, and spatio-temporal
characteristics can be used to identify actual UAVs in the image.
2.3.2 Close UAV & Item Detector
The Far detector presented in the previous section will be used to find and
approach the target UAV during the challenge. Nevertheless, the techniques
used to detect objects in a far plane are not suitable for short distances. For
object detection on close behavior (less than 20-15 meters) a modified version
of Tiny YOLO v3 [3] will be used to increase accuracy on smaller objects. Tiny
YOLO v3 is an extremely fast fully convolutional one shot object detector,
which can use a full image (at 720p resolution) to output a set of bounding
boxes corresponding to the objects in the image along with a confidence value.
Since Tiny YOLO v3 can be used as a multi-classifier, the same detector
can be used to identify the target UAV, as well as items of interest (foam ball
and balloons). Three classes are then used: UAV, ball and balloon. For the
training procedure, two steps are performed:
First, the model is pre-trained with a manually labeled dataset containing
nearly 4000 images of UAVs gathered online, and 1000 images of balls and
balloons.
Once the detector is performing properly, a second training step of fine
tuning is performed using a second dataset. This dataset is composed of
1000 images of the target UAV that is used for the tests (DJI m100 and
DJI f450) and 500 images of balls and balloons recorded during real flights
and manually labeled. This procedure lowers the domain gap between our
original dataset and the real environment.
The results of the training procedure showed that the Tiny YOLO v3 de-
tector was able to achieve 85 mAP@0.5 (a common metric in object detection)
with this model running at 25 Hz in the NVIDIA Jetson Xavier.
The Skyeye Team Participation in the 2020 MBZIRC 7
(a) (b) (c)
Fig. 4 Diagram of the proposed method UAV pose estimation in challenge 1. (a) An el-
lipsoid surrounding the UAV is considered, and the resulting pose is calculated using the
bounding box of the detection in image coordinates. (b) Close up diagram of image refer-
ences. (c) Close up diagram of distance estimation.
2.3.3 Pose estimation
The target UAV pose estimation is performed using a basic trigonometric cal-
culation to estimate an approximate 3D position relative to the aerial vehicle.
For a given height and width of an ellipsoid around the target UAV, and
the bounding box coordinates in the image (Figs. 4(a), 4(b), 4(c)), the polar
coordinates to the object are calculated using the following equations.
F=F OV
2(1)
β=α1α2(2)
R=resolution
2(3)
f=R
tan(F)(4)
γ=tan1(x×tan(F)
R) (5)
d=r
tan(β
2)(6)
In the case of the item (ball and balloon) pose estimation, a EPnP opencv
algorithm2[2] is used. The four corners of the bounding box in image are
paired with the respective points in world (centering the axes at the center of
the item). Afterwards the plane which dissects the item is used, whose corners
are {(-size/2,size/2,0), (size/2,size/2,0),...}, to acquire the translation to the
camera axes. Finally, a linear Kalman Filter (KF) is integrated in order to filter
the raw position estimates and make them usable by other components in the
architecture. Everything is shared through tf channels in the ROS framework.
2https://github.com/cvlab-epfl/EPnP
8 Ramon A. Suarez Fernandez et al.
2.3.4 Component Execution
In the execution of the mission, the FES components share a policy to decide
at which time slot each detector is executed. Fig. 5 shows the execution flow
of this policy. At the start of the mission, the Far detector is executed when
an image frame is received (since the location of the target UAV is unknown
and assumed at a far distance) and the amount of image frames is recorded
using a counter (NUM FRAMES).
Fig. 5 Feature extraction system component execution flowchart.
After the execution of the Far detector, the modulus operation is applied
to NUM FRAMES with the divisor F t. As such, after every F t frames are
received, the Close UAV and Item detector is executed to evaluate if a detection
is made. Once the number of consecutive frames where there is a detection
(NUM CON DET) equals a minimum threshold (D min), the Close UAV and
Item detector is continuously executed until the amount of frames without a
detection (MISSED FRAMES) reaches a maximum value of allowed misses
(M max).
The Skyeye Team Participation in the 2020 MBZIRC 9
2.4 Situation Awareness System
Challenge 1 requires the aerial vehicle to perform aggressive maneuvers of fly-
ing at high speeds (between 5 and 8 m/s) and precise maneuvers to interact
with objects during the mission. Thus, precise and robust state estimation is a
key element of the overall autonomous system solution. The Situation Aware-
ness System encompasses any modules that could be used for state estimation
of the UAV platform during a mission.
Fig. 6 Block diagram for the Situational Awareness System Multi Sensor Fusion state
estimation module.
The state estimation block of our system, shown in Fig. 6, consists of the
standard Multi-Sensor Fusion (MSF) framework in ROS [5]. The framework is
based on an Extended Kalman Filter (EKF) and has the capability to integrate
an unlimited number of sensors, as well as to compensate for any delayed sensor
measurements with on-line self-calibration. The state prediction is performed
by fusing the measurements received from the IMU sensors, which provide the
measurements of linear accelerations and angular velocities in the 3 axis at
a frequency of 400 Hz, and the position estimates about the x,y and z axis
provided by the RTK GPS sensor at 10 Hz. For additional robustness, the
framework has been modified in order to integrate the linear velocities about
x, y and z axis of the aerial robot estimated by an on-board Visual Inertial
Odometry (VIO) algorithm running at 50 Hz and the altitude of the aerial
robot estimated by a sonar sensor. The MSF is thus able to estimate the
complete high-frequency odometry information of the aerial robot at 400
Hz.
2.5 Planning System
The Planning System is in charge of defining a path along which the aerial
vehicle will have to navigate. A modified CHOMP planner is used for planning
a collision-free path for approaching and following the target UAV and items.
The goal points for the planner are given by the Executive System and the
initial point is taken as the current pose of the UAV, which is given by the
MSF odometry measurements.
10 Ramon A. Suarez Fernandez et al.
First, the line segment between the initial and goal points is interpolated
into finite points, where every point can be thought of as a state. The cost
function of the CHOMP planner is composed of an obstacle objective function
fo(ξ), a smooth objective function fs(ξ), and the boundary objective functions
fb(ξ) for X, Y, and Z. The final objective function can be expressed as
f(ξ) = w1fs(ξ) + w2fo(ξ) + w3fb(ξ) (7)
with,
fb(ξ) = fxmax (ξ) + fxmin (ξ) + fymax (ξ) + fymin (ξ) + fzmax (ξ) + fzmin (ξ) (8)
fs(ξ) = 1
2Z1
0kd
dsξ(s)k2ds(9)
fo(ξ) = Z1
0
co(ξ(s))kd
dsξ(s)kds(10)
where w1,w2and w3are the weights for each one of the objective functions,
ξis the path and ξ(s) is a function mapping the path length sto the UAV
configurations. Given that the target UAV is considered as a cylinder obstacle
the obstacle cost function parameters are expressed as,
co(ξ(s)) = (w0r(ξ(s))
3(1 dist(ξ(s))
r(ξ(s)) )3, dist(ξ(s)) r(ξ(s))
0, dist(ξ(s)) > r(ξ(s)) (11)
r(ξ(s)) = R
cos(arctan(distz(ξ(s),c)
dist(ξ(s))).(12)
In (11) and (12), the distance from current position of the UAV to center of
the cylinder obstacle is denoted by dist(ξ(s)), Ris the radius of the obstacle,
cis the position of the center of the obstacle, and distz(ξ(s), c) is the altitude
distance from the current position of the UAV to the center of the obstacle.
With these functions, the cost functional gradients of every state is calcu-
lated and a gradient descent approach is used to do path optimization. After-
wards, a ROS trajectory generation package is used to generate the desired
trajectory to the target UAV or item, given the path of the CHOMP planner.
In order to achieve different velocities in certain stages of the mission, function
to estimate the segment times is modified as,
Tseg(ξk1, ξk) = Tseg (ξk1, ξk)
fv
(13)
where Tseg(ξk1, ξk) is the segment time between ξk1and ξkand fvis the
velocity factor which is given by Executive system. An example from simula-
tion tests can be seen in Fig. 7. In this simulation 5 points are used as inputs
and the Planning system generates the desires trajectories required to avoid
the obstacles present in the path while reaching the goal point.
The Skyeye Team Participation in the 2020 MBZIRC 11
(a) (b) (c)
(d) (e) (f )
Fig. 7 Planning system execution example. The planner is given 5 waypoints starting from
(a) and finishing in (f).
2.6 Motor System
The Motor System is in charge of generating and communicating the necessary
actuator commands for the UAV to perform the desired trajectories, generated
by the aforementioned planning system. The main component of this system
is the motion controller. Given the complex requirements of challenge 1, re-
liable trajectory tracking control is a key element of the overall system. The
motion controller scheme implemented for challenge 1 is a modified version
of a trajectory tracking Model Predictive Controller (MPC) developed by the
Autonomous Systems Lab at ETH Zurich [10][11].
The trajectory tracking controller based on MPC is a robust, high band-
width control system which tracks the reference trajectories generated by the
Planning System and compensates for changes in mass and wind with an
on-line EKF disturbance observer. Since the DJI Matrice 210 v2 does not sup-
port directly sending angular rate commands, the MPC controller outputs are
smooth pitch and roll commands for horizontal motion and altitude. Although
though the position and heading of the UAV can be controlled simultaneously
by the MPC, we decouple these degrees-of-freedom (DOF). Thus, for simplic-
ity, we use proportional control scheme for the yaw-axis.
12 Ramon A. Suarez Fernandez et al.
2.7 Executive System
In order to organize and schedule the execution of the overall system mod-
ules, a high-level Finite State Machine (FSM) was implemented to integrate
the challenge 1 tasks. The main states of the FSM are shown in Fig. 8 and
explained below.
Fig. 8 Task-level Finite State Machine implemented for challenge 1 of the MBZIRC.
FOCUS Once the UAV has reached the TAKE OFF altitude, in this mode,
the UAV will move quickly to the center of the arena in search of the tar-
get UAV. In the FOCUS mode, the Far detector is disabled because of low
efficiency when the UAV moves at high speeds.
SEARCH In this mode, the UAV will initiate a predefined trajectory (ellipse
shape in the middle of the flight arena) in search of the target UAV. In
the SEARCH mode, the Far detector will be activated and the UAV will
perform slow maneuvers.
FOLLOW FAR When the Far detector detects the target UAV, the vehicle
is commanded to move 5 meters in the target UAV’s direction, until the
Close detector is able to track the target UAV. In this mode, if the Far
detector is not capable of detecting the target UAV during a predefined
time, then (UAV lost) the state returns to the FOCUS mode.
The Skyeye Team Participation in the 2020 MBZIRC 13
FOLLOW CLOSE Once the Close detector is activated, the UAV will move
in the direction of the target UAV whilst maintaining a lower altitude than
the target UAV. In this mode, if the Close detector is not capable of detect-
ing the target UAV during a predefined time, then (UAV lost) the state
shifts to the RECOVER mode. If the UAV is close enough to the target
UAV to detect the item, the state will change to the FOLLOW ITEM state.
FOLLOW ITEM In this state, the UAV will follow a trajectory where the
gripper will collide with the target item.
GRASP ITEM When the item is at a short distance from the UAV, the
gripper will automatically close. After this action, if the item is grasped,
a sensor will activate. If the UAV fails to grasp the item, the state will
change to the RECOVER mode.
RECOVER Depending on what was the previous state and the last detec-
tion of the target UAV or item, in this mode an action to recover the visual
contact will be triggered. If the target UAV or item are not detected before
a timeout is reached, the FSM will change to the FOCUS mode.
LAND The UAV will be commanded to land either if the battery is low, the
time for the challenge is over, or if there is a successful grasping of the
item. The LAND state will take place in a predefined GPS position for the
return of the target item.
3 Challenge 3: Autonomous Fire Extinction with UAV and UGV
The main objective of Challenge 3 is to autonomously extinguish a series of
simulated fires in an urban high rise building firefighting scenario. In particular,
a team of robots has to extinguish a set of fires using water, including fires
inside the building and located in its facade. Furthermore, fires located at
ground level outside the building must be extinguished using fire blankets
carried by the robots.
To accomplish the mission, we use a team of up to 3 UAVs and 1 UGV. In
the following, we describe the proposed robotic team and their main modules.
Then, we describe the cooperation in the team.
3.1 The robot team
3.1.1 Unmanned ground vehicle
The ground platform used for Challenge 3 is a 6-wheeled robot with indepen-
dent traction system and a pan&tilt mechanism on top. The robot is able to
navigate at a maximum speed of 0.7 m/s and it has battery autonomy for
14 Ramon A. Suarez Fernandez et al.
more than 4 hours of operation (motion and sensing/computation). The robot
integrates an i7 computer, local sensing and a 3D LIDAR. The hardware used
in the ground platform is (see Fig. 9):
Onboard computer: 5NUC5i7RYB.
GPS sensor UBlox 8.
IMU: ArduIMUv3.
Infrared camera: A compact SeekThermal camera mounted on a pan&tilt
mechanism on top of the robot. The images are used for fire detection.
3D LIDAR: A 3D OS-1-16 LIDAR manufactured by Ouster has been in-
stalled on top of the robot. The information gathered by this sensor is used
for localization and mapping.
2D LIDAR: A Hokuyo UTM-30LX sensor is installed. The information of
this sensor is used for local obstacle detection/avoidance.
Orbbec ASTRA camera. Used for providing the operator with images to
allow remotely operating the platform. It is also used for indoor local ob-
stacle avoidance.
Fig. 9 Ground platform used for Challenge 3.
3.1.2 Unmanned aerial vehicles
Regarding to the aerial platforms, the aerial robot models used for the com-
petition will be adaptations of the DJI Matrice 210 V2 and the DJI Matrice
600 Pro. The aerial platforms are also equipped with a OS-1-16 LIDAR manu-
factured by Ouster, installed to the bottom of the robots. The LIDAR is able
to provide 320k 3D points per second, ranging from 0.2 to 120 meters. The
purpose of this sensor is threefold:
Environment mapping and odometry. This sensor is the basis for the gen-
eration of the 3D map of the environment. It is also used to compute an
accurate robot odometry in the UAVs.
Robot localization. The 3D LIDAR is used as main sensor for robot local-
ization in all robots, UAVs and UGV.
Obstacle detection for robot navigation. The robot local planner uses this
information to detect mid/long range obstacles (more than 2 meters) in
order to properly react on time.
The Skyeye Team Participation in the 2020 MBZIRC 15
Fig. 10 shows the DJI Matrice 600 Pro used for the preliminary experi-
ments.
Fig. 10 DJI Matrice 600: one of the aerial platforms used for Challenge 3.
3.2 3D Mapping and Localization
One of the main issues in Challenge 3 is the localization of the robots. They
should localize seamlessly and accurately in GPS and GPS-denied environment
(inside the building), with a smooth transition among them.
The mapping and localization system proposed consists of two main mod-
ules (see Fig. 11): a LIDAR-based odometry and an online map-based global
localization module. These systems update the global map during the map-
ping phase, and estimates the global position of the robot into the map during
the mission. Next sections describe each module of the implemented system.
While there are differences in the UAV and UGV, both are based on the same
principles.
3.2.1 3D Mapping and multi-sensor odometry
The base for the mapping and localization systems is an odometry system that
allows to estimate the relative motion of the robot. The LIDAR-based odome-
try module considers IMU and 3D LIDAR in order to compute a reliable 6DoF
robot odometry in the environment. This module provides short-term aligned
scans that can be used for local mapping and a map of the environment. It
integrates the IMU estimation together with a LIDAR-based mapping process
to provide global consistency to the mapping process.
The LIDAR-based odometry module implements an advanced LOAM sys-
tem [14], which decomposes the original problem into scan registration and
16 Ramon A. Suarez Fernandez et al.
Fig. 11 Mapping and localization system implemented in the all the platforms.
Fig. 12 Example of 3D map computed outdoors with data provided by one robot.
mapping. The first problem is solved as online motion estimation. The sec-
ond one conducts mapping as batch optimization (similar to iterative closest
point, ICP [15]). The system computes high-precision motion estimates and
maps. Therefore, the accuracy of the localization of the robot and of the map
is considerably improved.
The system implemented fuses the information from 3D LIDAR and IMU
data. Some modifications have been implemented with respect to the original
LOAM system [14]. The original one uses the point cloud from a 2D LIDAR
sensor. In our implementation, the first motion estimation is computed from
3D point cloud and acceleration values from IMU sensor.
During the mapping stage, other sensors as GPS are also recorded. This
information is later on used to compute the relative transformation between
the computed 3D map and the GPS coordinate system for multi-sensor local-
ization.
3.2.2 Multi-sensor robot localization
Once the map is generated, another module is in charge of localizing the
robot within the map built during the execution of the different missions.
This module provides a robust global position of the robot in the map. The
The Skyeye Team Participation in the 2020 MBZIRC 17
algorithm is based on Monte Carlo localization (MCL) approach and extends
previous author work presented in [19].
The starting point of the algorithm is an initial belief of the robot pose
probability distribution, which determines the distribution of a set of position
hypotheses around such pose. We call these hypotheses particles, and they are
propagated following the robot motion model each time its pose changes. Every
time new sensor readings are received, each particle evaluates its accuracy by
checking how likely it would receive such sensor reading at its current pose.
The next step of the algorithm is redistributing (resampling) the particles to
new poses that are more likely to be accurate. This is an iterative process
that involves moving, sensing, and resampling, while all the particles should
converge to a single cluster near the true pose of the robot.
One of the main benefits of the MCL approach is the reduced computation
required. The approach only evaluates the sensors when the robot moves above
some given thresholds in both translation and rotation. When the robot moves
above one or both thresholds, it launches one update cycle: predicting robot
position according to odometry, updating position hypotheses based on sensors
and re-sampling the hypotheses space when required.
Thus, the particle filter consists of Nparticles pi, each one with the fol-
lowing state vector pi= [x, y, z, Ψ ]t. where Ψrefers to the yaw angle of the
robot. Note how the robot roll and pitch angles are not included into the robot
pose definition. These angles are available and accurate enough in the robot
through the use of its onboard IMU. They are fully observable and their values
are usually accurate. This greatly reduces the computational complexity of the
algorithm, allowing for real-time onboard computation. Each particle has an
associated weight wisuch as ΣN
i(wi) = 1.
The particles are initialized by setting the initial position and distributing
them in the hypotheses space. The information from the LIDAR-based odom-
etry provides increments of the robot state vector and it is used to compute
a priori distribution of the particles. Thus, considering that the robot is a
non-holonomic systems, the state of the particles will evolve according to the
following expressions for particle i:
xt+1
i=xt
i+∆x ×cos(Ψt
i)∆x ×sin(Ψt
i) (14)
yt+1
i=yt
i+∆x ×sin(Ψt
i)∆x ×cos(Ψt
i) (15)
zt+1
i=zt
i+∆z (16)
Ψt+1
i=Ψt
i+∆Ψ (17)
The values of ∆x,∆y,∆z, and ∆Ψ are drawn randomly following a normal
distribution centered in their actual values and standard deviations propor-
tional to each increment itself.
The following sensor reading are used to update the weight associated to
each particles hypothese: The point clouds provided by the 3D LIDAR sensor,
the 3D Position provided by the GPS, the robot roll, pitch and yaw provided
by the IMU and the height above the ground provided by the altitude sensor.
18 Ramon A. Suarez Fernandez et al.
These sensor are integrated into the MCL differently, according to the
following criteria:
The point clouds are transformed to each particle pose in order to find
correspondences between the cloud and what the map should look like
from that particle’s pose. Since this is very expensive computationally, we
first compute a 3D probability grid as in [22], in which each position stores
a value of how likely it is that such position falls within an occupied point
of the map, instead of storing binary information about occupancy as in the
provided map. Each 3D position piof the grid is then filled with probability
values according to a specific Gaussian distribution centered in the closest
occupied point in the map from pi,mapi, and whose variance σ2depends
on the sensor noise used in the approach.
grid(pi) = 1
2πσ2e−||pimapi||2/2σ2(18)
Such probability grid only needs to be computed once, it is not required
to be updated for a given environment, and relieves from performing nu-
merous distance computations between each cloud point for each particle
and its closest occupied point in the map. Besides, each point cloud is
first transformed according to the current roll and pitch provided by the
on-board IMU. This transformation is done just once per update, reduc-
ing the computational requirements as well. Then, for every point of the
transformed cloud, we access its corresponding value in the 3D probability
grid. Such value would be an indicator of how likely is that point to be
part of the map. By doing this with every point of the cloud and adding
all the probability values, we obtain a figure of how well that particle fits
the true location of the aerial robot according to the map.
Finally, the weight wiof each particle piis computed. Assuming that the
point cloud is composed of M3D points cj, the weight is computed by
adding all the associated probability grid values as follows:
wmap
i=1
M
M
X
j=1
grid(pi(cj)) (19)
where pi(cj) stands for the transformation of the point to the particle’s
state, and grid(pi(cj)) is the evaluation of the probability grid in such
transformed position.
The position measurements provided by the GPS are used to check how
well each particle hypothesis matches the sensor reading. First, the GPS
position is transformed into a local metric estimation into the map. Then,
awgps is computed for each particle. Assuming a GPS measurement xgps,
the associated weight wgps
iis is computed as follows:
wgps
i=1
2πσ2e−||pixg ps||2/2σ2(20)
The Skyeye Team Participation in the 2020 MBZIRC 19
Where the distance computation dismisses the altitude contribution of the
GPS, which is subject to significant noise. Also, the standard deviation
of the GPS measurement σis computed empirically to account for GPS
errors in single configuration.
The height above the ground is integrated into the MCL through the re-
sampling stage. Instead of including the information as a regular update,
we decided to include this information into the re-sampling, forcing the
hypotheses to be distributed around the altitude provided by the robot
altimeter. This way, we can reduce the dispersion of the particles around
Z axis, which is easily observable by means of different sensors as laser
altimeter, barometer or GPS altitude.
IMU yaw angle is also integrated through the re-sampling stage. The ratio-
nale behind this decision is also based on the computational optimization
and the nature of the yaw angle. In the experiments area, the yaw angle is
assumed to be slightly distorted by the environment, so that we can trust
the estimation from the magnetometer. Thus, everytime a new re-sampling
is performed, the orientation of the particles are drawn from a Gaussian
distribution centered in the latest yaw value provided by the IMU.
Finally, GPS and point-cloud updates are integrated into the filter. Given
the distinct nature of the two technologies involved, we calculate separate
weights for each sensing modality. A weighted average is used to obtain the
final weight of each particle:
wi=αwmap
i+ (1 α)wgps
i(21)
where αis chosen depending on the particularities of the indoor environment
where the robot is going to operate. If the map used in the MCL does not
contain the full environment or its accuracy is not enough to trust the map
matching, αshould be lower than 0.5. Whereas if GPS measurements are not
accurate, αshould be higher.
This approach is used for both robot systems, ground and aerial. However
some parameters are different between both platforms:
The dispersion in Z axis is very small in the predictive model of the ground
robot. This relates with the fact that the UGV moves in 2D.
The UGV does not integrates height sensors, for the same reason.
The yaw angle is not integrated into the UGV.
3.3 Autonomous robot navigation
After having a reliable localization, tasks such as where to go, follow a path
or reach a goal should be addressed. The proposed autonomous navigation
system consists of global and local planner. It generates safe paths from two
configurations of the robot. A 3D planner has been implemented in the aerial
platforms and a 2D planner in the ground platform. The implemented planner
20 Ramon A. Suarez Fernandez et al.
Fig. 13 Example of tra jectory computed by the Lazy Theta* algorithm. Initial position
(black square) is in the top and final position at the bottom of the image.
for the MBZIRC competition is an extended version of the planner presented
in [21].
The Global Planner computes a path from the global map. The inputs are
the mission plan generated by the operator, the map of the environment and
the robot localization given by the MCL approach. The output is a feasible
and safe path that should be followed in order to fulfill the mission plan.
The planner is based on a variation of the any-angle path planning Theta*
called Lazy Theta* [16]. An advantage of this algorithm is the optimization of
the computational load. Other advantages of this graph-based algorithm with
respect to sampling-based algorithms is that its result is closed the shortest
path and its solution is always the same. Furthermore, it gets a very low
computation time and a high repeatability. These characteristics make this
algorithm suitable to address the tasks of the MBZIRC competition.
Fig. 13 shows some of these advantages in a complex scenario where the
solution is difficult to find. Note that Lazy Theta* algorithm computes quickly
the shortest path.
Lazy Theta* algorithm digs into optimizations of Theta*, reducing the
number of line-of-sight checks required and optimizing the algorithm. It is a
variant of Theta* and thus it propagates information along graph edges (to
achieve a short runtime) without constraining the paths to graph edges (to find
”any-angle” paths). The algorithm is based on the A* algorithm, and this last
one is basically the Dijkstra algorithm with Euclidean distance as heuristic.
Theta* takes into account the line of sight between neighbours allowing any-
The Skyeye Team Participation in the 2020 MBZIRC 21
Fig. 14 Fire segmentation on the infrared images during an extinguishing action. From left
to right it can be seen how the temperature decreases when applying water.
angle paths, and finally Lazy Theta* reduces the line of sight checks improving
computation times by keeping the same paths as Theta*.
On the other hand, the Local planner creates a plan locally dealing with
dynamic obstacles thanks to the measurements obtained from the onboard 3D
LIDAR. It is in charge of following the global path computed by the global
planner and should also replan quickly a safe trajectory when an unexpected
event takes place such as an unknown obstacle. It takes as input the global
path generated by the global planner and the local map built by using the
measurements of the onboard sensors in order to make the crawler executes
the input path. As output, it provides a free-collision trajectory to the path
tracker module.
3.4 Fire perception and extinguishing
In order to accomplish the mission, we need to detect fires and estimate their
positions by using sensors on board the robots.
3.4.1 Fire segmentation
The main sensor related to fire perception is the miniature infrared thermal
camera mentioned above. The camera can detect temperatures between -40o
to 330oCelsius.
Fire detection is carried out by analyzing the infrared images provided by
the cameras. A calibration procedure approximately transforms the raw data
obtained by the cameras to temperature ranges. Then, given the clear differ-
ence of temperatures between the fire and the surroundings, a thresholding
operation is performed to segment the fire pixels.
As a result, the fire detection module provides coordinates on the image
plane of fire spots. Fig. 14 shows an example of the results obtained in local
tests considering the IR device used in the challenge to emulate the fire IR
signature.
3.4.2 Fire position estimation
The fire segmentation results mentioned above will be used for the visual
servoing final approach to extinguish the fire (see below). However, during the
22 Ramon A. Suarez Fernandez et al.
Fig. 15 Information from the camera pose and LIDAR is fused in time to estimate the
position of the fire.
exploration of the scenario we need to estimate the 3D position of the fire
spots to plan the approximation goals for fire extinguishing.
From the fire position in pixel coordinates it is not possible to the estimate
the 3D position of the fires, as there is no information about the range to
the fires. In order to obtain the range, the LIDAR information is considered.
The LIDAR values are mapped to the image coordinates and associated to
the fire object, so that a range can be estimated. This is not always possible
given the resolution of the LIDAR (and depending on the actual distance
to the fire). In that case, the range is estimated considering the maximum
distances according to the 3D map of the environment. Each measurement is
associated with a covariance matrix that reflects the uncertainty on the range,
which is adapted according to the estimated range (in the latter case, a large
uncertainty is associated to the range), and other aspects like the position of
the robot.
Then, an Information Filter is used to fuse in time the measurements about
the position of the fire from different view points in order to triangulate the fire
position, as in [17]. Fig. 15 shows the estimated position of a fire and the asso-
ciated uncertainty in a mission inside a building. This filter allows to consider
also measurements from multiple robots for the triangulation procedure.
3.4.3 Fire extinguishing
The fire position estimation can be used to approach the fire sources from an
adequate vantage point. Once the robot is located at a attacking distance to
the fire, a fire extinguishing task can be launched. This consists on aiming
adequately the extinguishing device, checking the presence of the fire and
releasing the water, for the fires in the building.
The Skyeye Team Participation in the 2020 MBZIRC 23
Fig. 16 Fire extinguishing systems installed on ground platform (left) and a DJI Matrice
M210 v2 (right).
The fire extinguishing system consists of a set of water pumps that can be
activated and deactivated through ROS messages (see Fig. 16). In the case of
the ground robot, the extinguishing device can be aimed by using a pan and
tilt unit. When a fire extinguishing task is called, a surveying motion is carried
out, and the fire is locked using visual servoing from the measurements from
the infrared camera if the fire is confirmed. In the case of the UAVs, the device
is fixed, and the UAV frame is used to aim it towards the fire, by controlling
only the yaw angle in the inside fires, or motions parallel to the wall for the
fires in the facade.
For the fire blankets, given the estimated position of the fire, a trajectory
is planned over the fire to release the blanket from the UAV, after checking
the actual presence of fire.
3.5 Executive layer
We employ Behaviour Trees (BTs) as the framework for task execution and
monitoring for the robots of the team. BTs are a way to structure the flow
and control of multiple tasks in the context of decision-making applications.
They are usually presented as an alternative to FSM. First originated as a
tool for NPC AI development in the video game industry, BTs have become
widespread in robotics, mainly due to its modularity and simplicity [12,13].
While a full description of the BT framework is out of the scope of this
paper, here we summarize its main elements. BT model behaviors as hierar-
chical trees made up of nodes (an example can be seen in Fig. 17). Trees are
traversed from top to bottom at a given rate following a set of well-defined
rules and executing the tasks/commands that are encountered while doing
so. Tasks and commands statuses are reported back up in the chain and the
flow changes accordingly. A status can be either success, failure or running.
According to their functionality, nodes can be classified as:
24 Ramon A. Suarez Fernandez et al.
Fig. 17 A simplified BT for a fire exploration mission of a single robot (there are aspects
-like fail-safe checks on time and batteries- removed for the sake of clarity). The robot
executes an exploration path while looking for a fire. If the fire detection is successful, the
robot will approach the fire and command an extinguishing task. Regardless of the result,
the robot will go back to the home position.
– Composite: control the flow through the tree itself and are similar to
control structures in traditional programming languages.
– Decorator: process or modify the status it receives from its child.
– Leaf: this is where the actual task is performed, the atomic tasks that the
robot can carry out, or other functionalities. As such, these nodes cannot
have any children.
As it can be seen from the classification above, BT decouple logic from
actual tasks in a natural way. When developing a tree one only should care
about the leaf nodes. The flow can later be defined and re-arranged constantly,
creating new behaviors and expanding on what is already done. One important
benefit of BTs are their composability (due to the hierarchical nature of the
trees) and re-usability.
The behavior of the individual robots of the team are created using BTs. In
the current case, the leaves are the associated atomic tasks of navigation, fire
detection and fire extinguishing as described above, and are integrated using
the actionlib interface of ROS. Fig. 17 presents an example of the behaviour
The Skyeye Team Participation in the 2020 MBZIRC 25
of one robot. We have used a publicly available BT implementation3, that we
have extended to consider the actions and particularities of the application.
3.6 Cooperation and Coordination
The approach to Challenge 3 takes advantage of the use of a heterogeneous
robot team (mixing UGV and different types of UAVs) for the coordination
of the fleet.
On the one hand, each robot specializes on one of the tasks for the Chal-
lenge. This means that we consider a loose cooperation between the robots.
In particular, we devote the ground robot for the ground floor indoor fire, and
each UAV to a different floor (with one UAV for the outside fires and the
ground floor fire on the facade).
Thus, the main aspect to consider is the coordination in time and space
to avoid collisions. The task allocation mitigates the risks, as the different
vehicles will fly at different heights. In any case, it is important to avoid close
encounters between the vehicles. For that, a priority level is assigned to the
different UAVs, and the different facades are visited by each vehicle at different
time instants, so that at every moment there no UAV overflying another one.
Before proceeding to a new facade, each vehicle checks if a higher priority
vehicle is still there.
Regarding the software architecture, we employ ROS, with different ROS
master nodes that run onboard each robot. We use the software described
in [18] to share between robots only vital information through the wireless
network, including the position of each robot for the checks indicated above.
4 Experiments and Results
In order to prepare for the upcoming challenges of the MBZIRC competi-
tion, several experimental tests have been performed independently by both
research groups to test, debug, and validate all of the proposed solutions which
have been integrated in the system pipelines. In this section, the major results
achieved for challenges 1 and 3 will be presented.
4.1 Challenge 1 Preliminary Results
The location chosen by the CVAR group as the test site for the tasks required
in challenge 1 is located in the outskirts of Illescas (Toledo, Spain). Since the
challenges were to be performed in an outdoor environment, an open airfield
provided a workspace large enough to test several scenarios that could be en-
countered during the real competition trials. The tests performed were focused
on gradually validating the individual components of the system architecture,
presented in Section 2.2.
3https://github.com/BehaviorTree
26 Ramon A. Suarez Fernandez et al.
4.1.1 Trajectory Tracking and Planning Tests
The initial tests were carried out to tune and validate the performance of the
Situation Awareness, Planning and Motor systems. The technical specifica-
tions for challenge 1 state that the target UAV will be flying in a lemniscate of
Bernoulli curve at a maximum velocity of 8 m/s. Thus, the Planning system
was used to command a trajectory to the Motor system from a specified series
of waypoints following a lemniscate shape.
0 5 10 15 20 25 30 35 40 45 50
-15
-10
-5
0
5
10
15
(a)
0 5 10 15 20 25 30 35 40 45 50
-15
-10
-5
0
5
10
(b)
0 5 10 15 20 25 30 35 40 45 50
-6
-4
-2
0
2
4
(c)
0 5 10 15 20 25 30 35 40 45 50
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
(d)
Fig. 18 Trajectory tracking results for preliminary flight tests performed at the outdoor
airfield. (a)-(b) UAV positions and commanded trajectory in X and Y axes, respectively.
(c)-(d) UAV velocities and commanded references in X and Y axes, respectively.
The results of the trajectory tracking test are presented in Fig. 18. As
previously mentioned, the MSF module from the Situation Awareness system
fuses the measurements from the RTK, VIO and IMU sensors to output an
estimation of the pose of the aerial vehicle. Using the fused measurements from
the MSF module, the MPC module of the Motor system was able to perform
the desired flight trajectories even in non-ideal conditions compensating for
wind gusts of up to 15 m/s.
The Skyeye Team Participation in the 2020 MBZIRC 27
4.1.2 Feature Extraction System Tests
The Feature Extraction system modules are considered the key elements re-
quired to successfully accomplish the tasks proposed in challenge 1. As such,
extensive testing was carried out to evaluate the the different components of
the FES. Fig. 19 through Fig. 21 show the results of the Far, Close and Item
detector tests performed with the aerial vehicle in the outdoor airfield. In these
tests, the aerial vehicle was manually flown towards and away from the target
UAV, where the FES components would activate depending on the detector
execution flowchart presented in Section 2.3.4.
(a) (b)
Fig. 19 Target UAV detection results from the Far detector module.
The aerial vehicle was initially flown to a distance of 50 m from the target
UAV. As can be seen in Fig. 19(a) and Fig. 19(b), at this distance the Far
detector was able to successfully detect the target UAV in the image frame
and an approximate pose was obtained, where the magenta Region-Of-Interest
(ROI) indicates the detection of the target UAV in the image frame.
(a) (b)
Fig. 20 Target UAV and Item detection results from the Close and Item detector module.
28 Ramon A. Suarez Fernandez et al.
Once the Far detector was validated, the aerial vehicle was brought closer
to the target UAV and maintained at distance of 10 m. At this point, shown
in Fig. 20(a) and Fig. 20(b), the Close and Item detector was able to detect
and identify both the target UAV and the item (ball) tethered to the vehicle.
From these detections, the pose estimation algorithm implemented in the FES
was able to produce the relative poses of both the target UAV and the item
to be used by the Planning system to execute the grasping maneuver with the
gripper.
(a) (b)
Fig. 21 Balloon item detection using the Close and Item detector module.
Finally, the aerial vehicle was flown through the outdoor airfield in order
to locate and return a position estimate of the remaining items (balloons),
which were tethered to the ground at a height of 2 m. The results of this
detector can be seen in Fig. 21(a) and Fig. 21(b). The results obtained in
these tests demonstrate that the perception components developed for the
FES are capable of detecting the target UAV and items with varying lighting
conditions, in addition to validating the FES component execution and pose
estimation modules.
4.1.3 Grasping Maneuver Tests
The most challenging task of challenge 1 will be to grasp the item which is
tethered to the target UAV in motion. Therefore, a rehearsal scenario was
implemented to validate the performance of the overall proposed strategy in
a fully autonomous real-world grasping maneuver mission.
In this scenario, a DJI Matrice M100 UAV was used as a target UAV which
has a red item (a balloon for safety) tethered to the end of a 2 m long carbon
fiber rod. As in the actual mission, the aerial platform would have to search,
detect, follow and interact with the target UAV in order to retrieve the desired
item.
The Skyeye Team Participation in the 2020 MBZIRC 29
(a) (b)
(c) (d)
(e) (f)
25
20
515
45
10
40
15
35 30 10
20
25 20
(g)
Fig. 22 Results from the grasping maneuver test. (a)(c)(e) Sequence view of the aerial
vehicle grasping maneuver. (b)(d)(f) Images from the on-board camera used for detection
of the target UAV and item. (g) 3D tra jectory of the UAV and pose of the item during
grasping maneuver.
30 Ramon A. Suarez Fernandez et al.
First, the target UAV was flown to a distance of approximately 50 m from
the aerial platform in flight. Afterwards, the FSM implemented in the Exe-
cution system (presented in Section 2.7) was activated to perform the fully
autonomous mission. The first states that were executed performed a grid
search of the target UAV in the open airspace. Once the Far detector achieved
the minimum number of detections necessary, the aerial vehicle proceeded to
move towards the target UAV in order to acquire a lock with the Close and
Item detector.
Once the Close and Item detector obtained a desired number of consec-
utive detections, the estimated pose of the tethered item was continuously
commanded to the Planning system as to generate the desired trajectories
needed to intersect the Item with the gripping mechanism. The results of the
grasping maneuver test performed with the proposed system strategy are pre-
sented in Fig. 22. Figs. 22(a) - 22(f) show a sequence of images of the aerial
vehicle performing the item grasping maneuver during the test flight, whilst
Fig. 22(g) shows the 3D trajectory of the aerial vehicle and pose of the Item.
As can be seen, the overall autonomous system strategy proposed and imple-
mented for the grasping task of challenge 1 is able to retrieve the item from
the target UAV successfully. Thus, the module components of the system ar-
chitecture presented in this work can be considered validated and the overall
solution ready for deployment in the actial challenge 1 of the MBZIRC.
4.2 Challenge 3 Preliminary Results
Many preliminary experiments with the ground and aerial platforms have been
performed in the airfield located in Guillena (Seville, Spain). These tests are
focused on the mapping, localization and navigation for the both ground and
aerial platforms, and the fire detection and extinguishing capabilities.
Fig. 23 Scenario built to replicate the scenario considered at the Abu Dhabi National
Exhibition Center in Abu Dhabi, (UAE): ground platform and one the aerial platforms.
The Skyeye Team Participation in the 2020 MBZIRC 31
A reduced version of the building and scenario considered at the final event
at the Abu Dhabi National Exhibition Center in Abu Dhabi, (UAE) has been
built in the airfield. Fig. 23 shows this scenario.
4.2.1 Mapping and odometry estimation
An accurate map is very important, as it is the base for the localization of the
system. Even if a previous model is available, we performed tests to evaluate
the accuracy of the maps.
Fig. 24 3D map of the scenario generated by the modified LOAM system and used by
ground platform and one the aerial platform, and the trajectory followed by the UAV (green
line).
Also, we have to evaluate the quality of the odometry. Fig. 24 represents
the 3D map generated from the modified LOAM system and the trajectory
followed by the UAV to generate the map (green line). The robot is localized
from the LIDAR-based odometry module described in Section 3.2. Fig. 25
shows another example.
4.2.2 Autonomous navigation experiments
We have also performed experiments to evaluate the navigation of the plat-
forms using the built maps. Several tests have been performed in this regard.
Fig. 26 shows a task in which the UAV has to visit waypoints and explore
the environment. It shows the local trajectory computed to reach a given
waypoint. The localization system provides an accurate localization to safely
navigate. The computation times of the global and local planner are 1 second
and 150 milliseconds respectively.
32 Ramon A. Suarez Fernandez et al.
Fig. 25 Estimated tra jectory according to odometry, and GPS comparison.
Fig. 26 Global and local trajectory generated by the planners during a flight of the DJI
Matrice M600 on the building (right) and an image of the infrared thermal camera (left).
Simulation tests to access to the building in the first floor and navigate
inside have also been performed. Fig. 27 shows the inflated obstacles from the
3D point cloud. The local planner computes the local trajectory in this space
at 5Hz. Fig. 28 and 29 show local trajectories computed to navigate inside. The
position of the UAV is shown with the frame and the local trajectory computed
avoids obstacles detected in each case in order to ensure a safe navigation of
the UAV.
4.2.3 Mission execution
We have also tested the integration of the different functionalities using Be-
haviour Trees (see Section 3.5) to achieve a fire extinguishing task. Here we
present an example with the ground platform. The task presented is to extin-
guish a fire inside the building. The mission includes several waypoints that
The Skyeye Team Participation in the 2020 MBZIRC 33
Fig. 27 Local costmap from the local sensors.
Fig. 28 UAV Local trajectory computed (red line), position of the UAV and scenario
considered.
Fig. 29 UAV Local trajectory computed (red line) to enter the building.
the ground platform should visit to explore the scenario. Once the ground
platform reaches one inspection waypoint, it performs an scan for nearby fires.
The mission includes localization using the previously built map, navigation
and fire detection and extinguishing.
Fig. 30 shows different snapshots of the mission. Fig. 30-A, B and D show
when the robot is closed to the door. Fig. 30-A and B shows the 3D point cloud
obtained from the LIDAR and the global (green) and local (red) trajectories
in Rviz. Fig. 30-E shows the ground platform passing through the door and
Fig. 30-C and F the robot inside. The computation times of the local planner
in this environment is 100 milliseconds. It is limited by the frecuency of the
LIDAR (10Hz).
Once the waypoint is reached, infrared thermal camera should check if a
fire exists. Figure 14 presents how the fire is detected. Figure 15-left shows
a the position of the fire. Finally, the fire extinguishing system is activated
to extinguish the fire (see Fig. 31-right). Then, the ground platform continues
exploring the environment and finally it returns to the HOME position (initial
position).
34 Ramon A. Suarez Fernandez et al.
Fig. 30 Mission to extinguish a fire inside the building: A-B-C show image of the camera,
3D point cloud, global trajectory (green line), local tra jectory (red line) and position of the
robot: D-E-F show images of the robot coming into the building.
Fig. 31 Fire detection inside of the building carried out by the infrared thermal cameras
(left) and fire extinguishing system working to extinguish the fire (right).
5 Conclusions
In this work, the hardware and software systems that are implemented as
the proposed strategies to solve Challenge 1 and Challenge 3 of the 2020
Mohamed Bin Zayed International Robotics Challenge (MBZIRC) have been
detailed. The contributions presented have been developed by the Skyeye team,
a consortium of research groups from the Universidad Polit´ecnica de Madrid
and the Universidad Pablo Olavide. The 2020 MBZIRC challenges present
complex environments and scenarios, which must be completed using fully
autonomous robotic systems. As such, a diverse knowledge of various fields is
necessary to implement efficient and robust solutions.
The aim of challenge 1 will be to test the capabilities of autonomous UAV
systems for detection and interaction with static and dynamic objects. The
results obtained from the experimental tests demonstrate the functionality
and validity of the overall strategy proposed for challenge 1. In these tests,
a UAV platform was able to autonomously execute complex tasks in outdoor
conditions to locate, follow and retrieve an object tethered to a target UAV
in flight. Nevertheless, given that the overall system is primarily based on
computer vision techniques, several complication can arise when extremely
varying lighting conditions are present. Moreover, the objective of Challenge
The Skyeye Team Participation in the 2020 MBZIRC 35
3 is to use several robots (ground and aerial platforms) to extinguish fires
in an urban-like scenario. The preliminary experiments have been presented
showing the main functionalities of the system, such as mapping, localization,
navigation, fire detection and extinguish, including integrated full missions
from the individual vehicles.
The future work to be performed entails the continuation of the flight tests
to explore the most outdoor scenarios that could possibly be found in challenge
1. Additionally, the challenge 3 work will focus on the coordination between
ground and aerial vehicles, and performing final experiments on the indoor
navigation of the UAVs in a scenario built to replicate the real environment
to be found at the final event before attending the MBZIRC competition
February 20-25 at the Abu Dhabi National Exhibition Center in Abu Dhabi,
(UAE).
Acknowledgements This work has been supported by the Mohamed Bin Zayed Inter-
national Robotics Challenge 2020. UPO would like to acknowledge Paulo Alvito and the
IDMind4company for the support with the ground robot hardware, the AeroGuillena air-
field for the support during the tests, and Danilo Tardioli, from the University of Zaragoza,
Spain, for his multi-master ROS software for multi-robot teams.
References
1. C. Sampedro et al. (2017) A fully-autonomous aerial robotic solution for the 2016 In-
ternational Micro Air Vehicle competition. 2017 International Conference on Unmanned
Aircraft Systems (ICUAS), Miami, FL, USA, 2017, pp. 989-998.
2. J. L. Sanchez-Lopez, J. Pestana, J. Collumeau, R. Suarez-Fernandez, P. Campoy and
M. Molina (2015) A vision based aerial robot solution for the Mission 7 of the Interna-
tional Aerial Robotics Competition. 2015 International Conference on Unmanned Aircraft
Systems (ICUAS), Denver, CO, 2015, pp. 1391-1400.
3. J. Redmon and A. Farhadi (2018) YOLOv3: An Incremental Improvement. arXiv.
https://arxiv.org/abs/1804.02767. Accessed February 2nd 2020.
4. V. Lepetit, F. Moreno-Noguer and P. Fua (2009) EPnP: An accurate O(n) solution to the
PnP problem. International Journal of Computer Vision. https://doi.org/10.1007/s11263-
008-0152-6.
5. S. Lynen, M. W. Achtelik, S. Weiss, M. Chli and R. Siegwart (2013) A robust and modular
multi-sensor fusion approach applied to MAV navigation. 2013 IEEE/RSJ International
Conference on Intelligent Robots and Systems, Tokyo, 2013, pp. 3923-3929.
6. R. A. S. Fern´andez, J. L. Sanchez-Lopez, C. Sampedro, H. Bavle, M. Molina and P.
Campoy (2016) Natural user interfaces for human-drone multi-modal interaction. 2016
International Conference on Unmanned Aircraft Systems (ICUAS), Arlington, VA, 2016,
pp. 1013-1022.
7. C. Sampedro et al. (2016) A flexible and dynamic mission planning architecture for
UAV swarm coordination. 2016 International Conference on Unmanned Aircraft Systems
(ICUAS), Arlington, VA, 2016, pp. 355-363.
8. J. L. Sanchez-Lopez et al. (2016) AEROSTACK: An architecture and open-source soft-
ware framework for aerial robotics. 2016 International Conference on Unmanned Aircraft
Systems (ICUAS), Arlington, VA, 2016, pp. 332-341.
9. Li, Jing, et al. (2016) Multi-target detection and tracking from a single camera in Un-
manned Aerial Vehicles (UAVs). 2016 IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS). IEEE, 2016.
4https://www.idmind.pt
36 Ramon A. Suarez Fernandez et al.
10. M. Kamel, T. Stastny, K. Alexis and R. Siegwart (2017) Model Predictive Control for
Trajectory Tracking of Unmanned Aerial Vehicles Using Robot Operating System. Robot
Operating System (ROS) The Complete Reference Volume 2. Springer 2017.
11. M. Kamel, M. Burri and R. Siegwart (2017) Linear vs Nonlinear MPC for Trajectory
Tracking Applied to Rotary Wing Micro Aerial Vehicles. IFAC-PapersOnLine, Volume 50,
Issue 1, 2017, Pages 3463-3469.
12. M. Colledanchise and P. Ogren (2017) How Behavior Trees Modularize Hybrid Control
Systems and Generalize Sequential Behavior Compositions, the Subsumption Architec-
ture, and Decision Trees. IEEE Transactions on Robotics, vol. 33, no. 2, pp. 372–389,
April 2017.
13. C. Paxton, A. Hundt, F. Jonathan, K. Guerin, and G. D. Hager (2017) Costar: In-
structing collaborative robots with behavior trees and vision. 2017 IEEE International
Conference on Robotics and Automation (ICRA), IEEE, 2017, pp. 564–571.
14. Zhang and S. Singh (2014) LOAM: Lidar Odometry and Mapping in Real-time.
Robotics: Science and Systems Conference (RSS), 2014.
15. S. Rusinkiewicz and M. Levoy (2001) Efficient variants of the ICP algorithm. Third
International Conference on 3D Digital Imaging and Modeling (3DIM), Quebec City,
Canada, June 2001.
16. Nash, S. Koenig and C. Tovey (2010) Lazy Theta*: Any-angle path planning and path
length analysis in 3D. Proceedings of the Twenty-Fourth AAAI Conference on Artificial
Intelligence (AAAI). pp. 147–154, 2010.
17. A. Dias, J. Capitan, L. Merino, J. Almeida, P. Lima, and E. Silva (2015) Decentralized
Target Tracking based on Multi-Robot Cooperative Triangulation. In Proceedings of the
IEEE International Conference on Robotics and Automation, ICRA, pp. 3449–3455, 2015.
18. D. Tardioli, R. Parasuraman, and P. ¨
Ogren (2019) Pound: A multi-master ROS node
for reducing delay and jitter in wireless multi-robot networks. Robotics and Autonomous
Systems, vol. 111, pp. 73-87, 2019.
19. F.J. Perez-Grau, F. Caballero, L. Merino, and A. Viguria (2017) Multi-modal Map-
ping and Localization of Unmanned Aerial Robots based on Ultra-Wideband and RGB-D
sensing. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
20. F.J. Perez-Grau, F. Caballero, A. Viguria, and A. Ollero (2017) Multi-sensor three-
dimensional Monte Carlo localization for long-term aerial robot navigation. International
Journal of Advanced Robotic Systems 14 (5).
21. F.J. Perez-Grau, R Ragel, F. Caballero, A. Viguria, and A. Ollero (2018) An architecture
for robust UAV navigation in GPS-denied areas. Journal of Field Robotics 35 (1), 121-145.
22. Armin. Hornung, K. M. Wurm, and M. Bennewitz (2010) Humanoid Robot Localiza-
tion in Complex Indoor Environments. IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS),
... In contrast, autonomous execution was the goal for Challenge 3 of the MBZIRC 2020. Team Skyeye (Suarez Fernandez et al., 2020) used DJI Matrice 600 and DJI Matrice 210 v2 UAVs equipped with color and thermal cameras, GPS, and LiDAR. A map is prebuilt from LiDAR, IMU, and GPS data to allow online Monte Carlo localization and path planning with Lazy Theta * . ...
Preprint
The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020 posed diverse challenges for unmanned aerial vehicles (UAVs). We present our four tailored UAVs, specifically developed for individual aerial-robot tasks of MBZIRC, including custom hardware- and software components. In Challenge 1, a target UAV is pursued using a high-efficiency, onboard object detection pipeline to capture a ball from the target UAV. A second UAV uses a similar detection method to find and pop balloons scattered throughout the arena. For Challenge 2, we demonstrate a larger UAV capable of autonomous aerial manipulation: Bricks are found and tracked from camera images. Subsequently, they are approached, picked, transported, and placed on a wall. Finally, in Challenge 3, our UAV autonomously finds fires using LiDAR and thermal cameras. It extinguishes the fires with an onboard fire extinguisher. While every robot features task-specific subsystems, all UAVs rely on a standard software stack developed for this particular and future competitions. We present our mostly open-source software solutions, including tools for system configuration, monitoring, robust wireless communication, high-level control, and agile trajectory generation. For solving the MBZIRC 2020 tasks, we advanced the state of the art in multiple research areas like machine vision and trajectory generation. We present our scientific contributions that constitute the foundation for our algorithms and systems and analyze the results from the MBZIRC competition 2020 in Abu Dhabi, where our systems reached second place in the Grand Challenge. Furthermore, we discuss lessons learned from our participation in this complex robotic challenge.
... In contrast, autonomous execution was the goal for Challenge 3 of the MBZIRC 2020. Team Skyeye (Suarez Fernandez et al., 2020) used DJI Matrice 600 and DJI Matrice 210 v2 UAVs equipped with color and thermal cameras, GPS, and LiDAR. A map is prebuilt from LiDAR, IMU, and GPS data to allow online Monte Carlo localization and path planning with Lazy Theta * . ...
Article
Full-text available
The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020 posed diverse challenges for unmanned aerial vehicles (UAVs). We present our four tailored UAVs, specifically developed for individual aerial-robot tasks of MBZIRC, including custom hardware- and software components. In Challenge 1, a target UAV is pursued using a high-efficiency, onboard object detection pipeline to capture a ball from the target UAV. A second UAV uses a similar detection method to find and pop balloons scattered throughout the arena. For Challenge 2, we demonstrate a larger UAV capable of autonomous aerial manipulation: Bricks are found and tracked from camera images. Subsequently, they are approached, picked, transported, and placed on a wall. Finally, in Challenge 3, our UAV autonomously finds fires using LiDAR and thermal cameras. It extinguishes the fires with an onboard fire extinguisher. While every robot features task-specific subsystems, all UAVs rely on a standard software stack developed for this particular and future competitions. We present our mostly open-source software solutions, including tools for system configuration, monitoring, robust wireless communication, high-level control, and agile trajectory generation. For solving the MBZIRC 2020 tasks, we advanced the state of the art in multiple research areas like machine vision and trajectory generation. We present our scientific contributions that constitute the foundation for our algorithms and systems and analyze the results from the MBZIRC competition 2020 in Abu Dhabi, where our systems reached second place in the Grand Challenge. Furthermore, we discuss lessons learned from our participation in this complex robotic challenge.
... Team Skyeye [17] used a 6-wheeled UGV and DJI Matrice 600 and 210 v2 UAVs equipped with color and thermal cameras, GPS and LiDAR. A map is prebuilt from LiDAR, IMU, and GPS data to allow online Monte Carlo localization and path planning with Lazy Theta * . ...
Conference Paper
Full-text available
Every day, burning buildings threaten the lives of occupants and first responders trying to save them. Quick action is of essence, but some areas might not be accessible or too dangerous to enter. Robotic systems have become a promising addition to firefighting, but at this stage, they are mostly manually controlled, which is error-prone and requires specially trained personal. We present two systems for autonomous firefighting from air and ground we developed for the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020. The systems use LiDAR for reliable localization within narrow, potentially GNSS-restricted environments while maneuvering close to obstacles. Measurements from LiDAR and thermal cameras are fused to track fires, while relative navigation ensures successful extinguishing. We analyze and discuss our successful participation during the MBZIRC 2020, present further experiments, and provide insights into our lessons learned from the competition.
... Team Skyeye [17] used a 6-wheeled UGV and DJI Matrice 600 and 210 v2 UAVs equipped with color and thermal cameras, GPS and LiDAR. A map is prebuilt from LiDAR, IMU, and GPS data to allow online Monte Carlo localization and path planning with Lazy Theta * . ...
Preprint
Every day, burning buildings threaten the lives of occupants and first responders trying to save them. Quick action is of essence, but some areas might not be accessible or too dangerous to enter. Robotic systems have become a promising addition to firefighting, but at this stage, they are mostly manually controlled, which is error-prone and requires specially trained personal. We present two systems for autonomous firefighting from air and ground we developed for the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020. The systems use LiDAR for reliable localization within narrow, potentially GNSS-restricted environments while maneuvering close to obstacles. Measurements from LiDAR and thermal cameras are fused to track fires, while relative navigation ensures successful extinguishing. We analyze and discuss our successful participation during the MBZIRC 2020, present further experiments, and provide insights into our lessons learned from the competition.
... Vision-based techniques, as in [18], to estimate marker locations, intercepted UAV states, or landing platform location are quite commonly used, especially with attention paid to robustifying approaches, as in [19]. The approaches to the landing problem, available in the literature, as in [20][21][22] or [23], do not focus on optimization of the landing task itself, but on building control laws to land the UAV in the autonomous mode based on, e.g., detected markers. ...
Article
Full-text available
The paper addresses the loop shaping problem in the altitude control of an unmanned aerial vehicle to land the flying robot with a specific landing scenario adopted. The proposed solution is optimal, in the sense of the selected performance indices, namely minimum-time, minimum-energy, and velocity-penalized related functions, achieving their minimal values, with numerous experiments conducted throughout the development and preparation to the Mohamed Bin Zayed International Robotics Challenge (MBZIRC 2020). A novel approach to generation of a reference altitude trajectory is presented, which is then tracked in a standard, though optimized, control loop. Three landing scenarios are considered, namely: minimum-time, minimum-energy, and velocity-penalized landing scenarios. The experimental results obtained with the use of the Simulink Support Package for Parrot Minidrones, and the OptiTrack motion capture system proved the effectiveness of the proposed approach.
Article
Full-text available
Autonomous robotic systems for various applications including transport, mobile manipulation , and disaster response are becoming more and more complex. Evaluating and analyzing such systems is challenging. Robotic competitions are designed to benchmark complete robotic systems on complex state-of-the-art tasks. Participants compete in defined scenarios under equal conditions. We present our UGV solution developed for the Mohamed Bin Zayed International Robotics Challenge 2020. Our hardware and software components to address the challenge tasks of wall-building and firefighting are integrated into a fully autonomous system. The robot consists of a wheeled, omnidirectional base, a 6 DoF manipulator arm equipped with a magnetic gripper, a highly efficient storage system to transport box-shaped objects, and a water-spraying system to extinguish fires. The robot perceives its environment using 3D LiDAR, as well as RGB and thermal camera-based perception modules and is capable of picking box-shaped objects and constructing a pre-defined wall structure. Its sensor modules also facilitate detecting and localizing heat sources to extinguish potential fires. A high-level planner coordinates and applies the robot's skills to complete the Challenge tasks. We analyze and discuss our successful participation during the MBZIRC 2020 finals, present further experiments, and provide insights to our lessons learned.
Preprint
Autonomous robotic systems for various applications including transport, mobile manipulation, and disaster response are becoming more and more complex. Evaluating and analyzing such systems is challenging. Robotic competitions are designed to benchmark complete robotic systems on complex state-of-the-art tasks. Participants compete in defined scenarios under equal conditions. We present our UGV solution developed for the Mohamed Bin Zayed International Robotics Challenge 2020. Our hard- and software components to address the challenge tasks of wall building and fire fighting are integrated into a fully autonomous system. The robot consists of a wheeled omnidirectional base, a 6 DoF manipulator arm equipped with a magnetic gripper, a highly efficient storage system to transport box-shaped objects, and a water spraying system to fight fires. The robot perceives its environment using 3D LiDAR as well as RGB and thermal camera-based perception modules, is capable of picking box-shaped objects and constructing a pre-defined wall structure, as well as detecting and localizing heat sources in order to extinguish potential fires. A high-level planner solves the challenge tasks using the robot skills. We analyze and discuss our successful participation during the MBZIRC 2020 finals, present further experiments, and provide insights to our lessons learned.
Article
Full-text available
We present a multirotor Unmanned Aerial Vehicle (UAV) control and estimation system for supporting replicable research through realistic simulations and real-world experiments. We propose a unique multi-frame localization paradigm for estimating the states of a UAV in various frames of reference using multiple sensors simultaneously. The system enables complex missions in GNSS and GNSS-denied environments, including outdoor-indoor transitions and the execution of redundant estimators for backing up unreliable localization sources. Two feedback control designs are presented: one for precise and aggressive maneuvers, and the other for stable and smooth flight with a noisy state estimate. The proposed control and estimation pipeline are constructed without using the Euler/Tait-Bryan angle representation of orientation in 3D. Instead, we rely on rotation matrices and a novel heading-based convention to represent the one free rotational degree-of-freedom in 3D of a standard multirotor helicopter. We provide an actively maintained and well-documented open-source implementation, including realistic simulation of UAV, sensors, and localization systems. The proposed system is the product of years of applied research on multi-robot systems, aerial swarms, aerial manipulation, motion planning, and remote sensing. All our results have been supported by real-world system deployment that subsequently shaped the system into the form presented here. In addition, the system was utilized during the participation of our team from the Czech Technical University in Prague in the prestigious MBZIRC 2017 and 2020 robotics competitions, and also in the DARPA Subterranean challenge. Each time, our team was able to secure top places among the best competitors from all over the world.
Conference Paper
Full-text available
This paper presents a methodology for mapping and localization of Unmanned Aerial Vehicles (UAVs) based on the integration of sensors from different modalities. Particularly, we integrate distance estimations to Ultra-Wideband (UWB) sensors and 3D point-clouds from RGB-D sensors. First, a novel approach for environment mapping is introduced, exploiting the synergies between UWB sensors and point-clouds to produce a multi-modal 3D map that integrates the estimated UWB sensors position. This map is further integrated into a Monte Carlo Localization method to robustly estimate the UAV pose. Finally, the full approach is tested with real indoor flights and validated with a motion tracking system.
Article
Full-text available
This article presents a software architecture for safe and reliable autonomous navigation of aerial robots in GPS-denied areas. The techniques employed within key modules from this architecture are explained in detail, such as a six-dimensional localization approach based on visual odometry and Monte Carlo localization, or a variant of the Lazy Theta* algorithm for motion planning. The aerial robot used to demonstrate this approach has been extensively tested over the past 2 years for localization and state estimation without any external positioning systems, autonomous local obstacle avoidance, and local path planning among other tasks. This article describes the architecture and main algorithms used to achieve these goals to build a robust autonomous system.
Article
Full-text available
This article presents an enhanced version of the Monte Carlo localization algorithm, commonly used for robot navigation in indoor environments, which is suitable for aerial robots moving in a three-dimentional environment and makes use of a combination of measurements from an Red,Green,Blue-Depth (RGB-D) sensor, distances to several radio-tags placed in the environment, and an inertial measurement unit. The approach is demonstrated with an unmanned aerial vehicle flying for 10 min indoors and validated with a very precise motion tracking system. The approach has been implemented using the robot operating system framework and works smoothly on a regular i7 computer, leaving plenty of computational capacity for other navigation tasks such as motion planning or control.
Conference Paper
Full-text available
In this paper, a fully-autonomous quadrotor aerial robot for solving the different missions proposed in the 2016 International Micro Air Vehicle (IMAV) Indoor Competition is presented. The missions proposed in the IMAV 2016 competition involve the execution of high-level missions such as entering and exiting a building, exploring an unknown indoor environment, recognizing and interacting with objects, landing autonomously on a moving platform, etc. For solving the aforementioned missions, a fully-autonomous quadrotor aerial robot has been designed, based on a complete hardware configuration and a versatile software architecture, which allows the aerial robot to complete all the missions in a fully autonomous and consecutive manner. A thorough evaluation of the proposed system has been carried out in both simulated flights, using the Gazebo simulator in combination with PX4 Software-In-The-Loop, and real flights, demonstrating the appropriate capabilities of the proposed system for performing high-level missions and its flexibility for being adapted to a wide variety of applications.
Chapter
Full-text available
In this chapter, strategies for Model Predictive Control (MPC) design and implementation for Unmanned Aerial Vehicles (UAVs) are discussed. This chapter is divided into two main sections. In the first section, modelling, controller design and implementation of MPC for multi-rotor systems is presented. In the second section, we show modelling and controller design techniques for fixed-wing UAVs. System identification techniques are used to derive an estimate of the system model, while state of the art solvers are employed to solve the optimization problem online. By the end of this chapter, the reader should be able to implement an MPC to achieve trajectory tracking for both multi-rotor systems and fixed-wing UAVs.
Article
Full-text available
Behavior trees (BTs) are a way of organizing the switching structure of a hybrid dynamical system (HDS), which was originally introduced in the computer game programming community. In this paper, we analyze how the BT representation increases the modularity of an HDS and how key system properties are preserved over compositions of such systems, in terms of combining two BTs into a larger one. We also show how BTs can be seen as a generalization of sequential behavior compositions, the subsumption architecture, and decisions trees. These three tools are powerful but quite different, and the fact that they are unified in a natural way in BTs might be a reason for their popularity in the gaming community. We conclude the paper by giving a set of examples illustrating how the proposed analysis tools can be applied to robot control BTs.
Article
The Robot Operating System (ROS) is a popular and widely used software framework for building robotics systems. With the growth of its popularity, it has started to be used in multi-robot systems as well. However, the TCP connections that the platform relies on for connecting the so-called ROS nodes presents several issues regarding limited-bandwidth, delays, and jitter, when used in wireless multi-hop networks. In this paper, we present a thorough analysis of the problem and propose a new ROS node called Pound to improve the wireless communication performance by reducing delay and jitter in data exchanges, especially in multi-hop networks. Pound allows the use of multiple ROS masters (roscores), features data compression, and importantly, introduces a priority scheme that allows favoring more important flows over less important ones. We compare Pound to the state-of-the-art solutions through extensive experiments and show that it performs equally well, or better in all the test cases, including a control-over-network example.