Conference PaperPDF Available

SkyCore: Moving Core to the Edge for Untethered and Reliable UAV-based LTE Networks

Authors:

Abstract and Figures

The advances in unmanned aerial vehicle (UAV) technology have empowered mobile operators to deploy LTE base stations (BSs) on UAVs, and provide on-demand, adaptive connectivity to hotspot venues as well as emergency scenarios. However, today’s evolved packet core (EPC) that orchestrates the LTE RAN faces fundamental limitations in catering to such a challenging, wireless and mobile UAV envi- ronment, particularly in the presence of multiple BSs (UAVs). In this work, we argue for and propose an alternate, radical edge EPC design, called SkyCore that pushes the EPC functionality to the extreme edge of the core network – collapses the EPC into a single, light-weight, self-contained entity that is co-located with each of the UAV BS. SkyCore incorporates elements that are designed to address the unique challenges facing such a distributed design in the UAV environment, namely the resource-constraints of UAV platforms, and the distributed management of pronounced UAV and UE mobility. We build and deploy a fully functional version of SkyCore on a two-UAV LTE network and showcase its (i) ability to interoperate with commercial LTE BSs as well as smartphones, (ii) support for both hotspot and standalone multi-UAV de- ployments, and (iii) superior control and data plane performance compared to other EPC variants in this environment.
Content may be subject to copyright.
SkyCore: Moving Core to the Edge for Untethered
and Reliable UAV-based LTE Networks
Mehrdad Moradi
University of Michigan
Karthikeyan Sundaresan
NEC Labs America
Eugene Chai
NEC Labs America
Sampath Rangarajan
NEC Labs America
Z. Morley Mao
University of Michigan
ABSTRACT
The advances in unmanned aerial vehicle (UAV) technol-
ogy have empowered mobile operators to deploy LTE base
stations (BSs) on UAVs, and provide on-demand, adaptive
connectivity to hotspot venues as well as emergency sce-
narios. However, today’s evolved packet core (EPC) that
orchestrates the LTE RAN faces fundamental limitations in
catering to such a challenging, wireless and mobile UAV envi-
ronment, particularly in the presence of multiple BSs (UAVs).
In this work, we argue for and propose an alternate, radical
edge EPC design, called SkyCore that pushes the EPC func-
tionality to the extreme edge of the core network collapses
the EPC into a single, light-weight, self-contained entity that
is co-located with each of the UAV BS. SkyCore incorporates
elements that are designed to address the unique challenges
facing such a distributed design in the UAV environment,
namely the resource-constraints of UAV platforms, and the
distributed management of pronounced UAV and UE mobil-
ity. We build and deploy a fully functional version of SkyCore
on a two-UAV LTE network and showcase its (i) ability to in-
teroperate with commercial LTE BSs as well as smartphones,
(ii) support for both hotspot and standalone multi-UAV de-
ployments, and (iii) superior control and data plane perfor-
mance compared to other EPC variants in this environment.
KEYWORDS
5G; UAV; LTE; EPC; Edge EPC; UAV-based LTE networks;
SDN; Drone; Resource-Challenged; Mobility Management
ACM Reference Format:
Mehrdad Moradi, Karthikeyan Sundaresan, Eugene Chai, Sampath
Rangarajan, and Z. Morley Mao. 2018. SkyCore: Moving Core to
the Edge for Untethered and Reliable UAV-based LTE Networks. In
The 24th Annual International Conference on Mobile Computing and
The contact author: moradi@umich.edu
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are not
made or distributed for prot or commercial advantage and that copies bear
this notice and the full citation on the rst page. Copyrights for components
of this work owned by others than ACM must be honored. Abstracting with
credit is permitted. To copy otherwise, or republish, to post on servers or to
redistribute to lists, requires prior specic permission and/or a fee. Request
permissions from permissions@acm.org.
MobiCom ’18, October 29-November 2, 2018, New Delhi, India
©2018 Association for Computing Machinery.
ACM ISBN 978-1-4503-5903-0/18/10. . . $15.00
https://doi.org/10.1145/3241539.3241549
Mobility
LTE UEs
LTE UAV
Command & Control (C2)
LTE
Ground Station
Satellite
Fig. 1: UAV-based LTE networks.
Networking (MobiCom ’18), October 29-November 2, 2018, New Delhi,
India. ACM, New York, NY, USA, 15 pages. https://doi.org/10.1145/
3241539.3241549
1 INTRODUCTION
LTE networks that are ubiquitous today are deployed after
sucient RF planning in a region. However, the static nature
of LTE base station (BS) deployments limits their ability to
cater to certain key 5G use cases surging trac demands
in hotspots (e.g., stadiums, event centers), as well as their
availability in emergency situations (e.g., natural disasters),
where the infrastructure could itself be compromised (sec-
tion 2). Providing an additional degree of freedom for base
stations, namely mobility, allows them to break away from
such limitations.
UAV-driven LTE networks.
In this regard, recent ad-
vances in unmanned aerial vehicle (UAV) technology have
empowered operators to take on-demand, outdoor connec-
tivity to another level, by allowing their base stations to be
deployed aerially on UAVs (Fig. 1), thereby oering complete
exibility in their deployment and optimization. Mobile op-
erators like AT&T and Verizon have both conducted trials
with LTE base stations mounted on UAVs [
15
,
16
] (helicopter
and xed-wing aircraft respectively, Fig. 3). AT&T also pro-
vided LTE network services from its UAV in the aftermath
of hurricane Maria in Puerto Rico last year [
7
]. Further, with
the availability of shared access spectrum like CBRS [
4
] in
3.5 GHz, this also opens the door for smaller, greeneld
operators to deploy and provide on-demand, private LTE
connectivity services without the heavy cost associated with
spectrum and deployment.
Limitations of the legacy EPC.
A typical LTE network
requires the deployment of two essential components: a radio
access network (RAN) consisting of multiple base stations
(BSs) that provide wide-area wireless connectivity to clients
(UEs), and a high-speed, wired core network of gateways
(evolved packet core, EPC) that sits behind the RAN and is
responsible for all the mobility, management and control
functions, as well as routing user trac to/from the Inter-
net. Realizing a multi-UAV-driven RAN (BSs deployed on
UAVs) with an EPC on the ground or in the cloud is one way
to directly apply today’s EPC architecture to the UAV envi-
ronment (as shown in Fig. 5). Based on publicly available
information [
5
,
7
,
14
16
], this has been the case with the
current operator-driven UAV eorts. However, this faces
signicant limitations in delivering real value to this chal-
lenging environment. Specically, while a
tethered setup
(EPC-UAV link being wired, Fig. 5a, and possibly in Fig. 3a)
signicantly limits the UAV’s mobility and ability to scale to
multiple UAVs,a
wireless setup
(EPC-UAV link being wire-
less/mobile, Figs. 3, 5b) incurs all the vagaries of the wireless
channel. For the latter, the choice of the wireless technology
becomes critical given that the EPC is responsible for setting
up, routing, and tearing down all voice/data bearers. It is
essential for the EPC to reliably reach all the UAVs wirelessly,
including those that are potentially far away in the presence
of non-line-of-sight conditions (e.g., buildings, foliage, etc.).
Further, it must deliver sucient capacity to support the
trac demands in the RAN. It is extremely challenging for a
wireless technology, be it lower frequency (sub-6 GHz like
LTE, WiFi, and etc.) or higher frequency (mmWave, satellite),
to simultaneously satisfy the needs of range, reliability/ro-
bustness, and capacity that the UAV environment demands
from the critical EPC-RAN link.
Core at the Edge.
Given the fundamental limitations in
deploying an EPC on the ground or in the cloud to support
a multi-UAV RAN, we advocate for a radical, yet standards-
compliant re-design of the EPC, namely the Edge-EPC archi-
tecture, to suit the UAV environment. As the name suggests,
we aim to push the entire EPC functionality to the extreme
edge of the core network, by collapsing and locating the EPC
as a single, light-weight, self-contained entity on each of the
UAVs (BSs) as shown in Fig. 7. Being completely distributed
at the very edge of the network, such an architecture com-
pletely eliminates wireless on the critical EPC-RAN path and
hence the crippling drawbacks faced by the legacy architec-
ture in this environment.
While denitely promising at the outset, realizing this rad-
ical design is not without its own set of challenges that are
unique to the UAV environment. In particular, (i) Resource-
challenged environment: The compute resources consumed
by the numerous network functions in the EPC is apprecia-
ble and becomes a concern when all the EPC functionality
is placed into a single node, and deployed directly on a UAV
platform the latter being highly resource-challenged to
begin with. This could signicantly aect both the UAV’s op-
erational lifetime as well as the processing (control and data
plane) latency of its trac, thereby resulting in a reduced
trac capacity. (ii) Mobility management: The hierarchical
nature of the legacy EPC architecture, gives a single network
gateway (like the mobility management entity, the MME)
a consolidated view of multiple BSs, thereby allowing it to
eciently manage handos during mobility of active UEs
as well as tracking/paging mobile UEs that are in idle mode.
Mobility of both active (handos) and idle UEs (tracking/-
paging) becomes a critical challenge, when the entire EPC
is located at each of the UAVs, thereby restricting their view
of events to only those that are local to the UAV.
Our Proposal SkyCore.
Towards our vision of building
untethered yet reliable UAV-based LTE networks, we present
our novel EPC design, SkyCore. SkyCore embodies the Edge-
EPC architecture while introducing two key pillars in its
design to address the associated challenges a complete soft-
ware refactoring of the EPC for compute-ecient deployment
on a UAV, and a new inter-EPC communication interface to
enable fully functional operation in a multi-UAV environ-
ment. Through software refactoring, SkyCore eliminates the
distributed EPC interfaces and collapses all distributed func-
tionalities into a single logical entity (agent) by transforming
the latter into a series of switching ow tables and associ-
ated switching actions. It also reduces control plane signaling
and latency by precomputing and storing (in-memory) sev-
eral key attributes (security keys, QoS prole, etc.) for UEs
that can be accessed quickly in real time without any com-
putation. To ensure complete EPC functionality, SkyCore
manages mobility right at the edge of the network it en-
ables a new control/data interface through software-dened
networking (SDN) to realize ecient inter-EPC signaling and
communication directly between UAVs. This allows the Sky-
Core agents on each UAV to proactively synchronize their
states with each other, thereby avoiding the real-time impact
of wireless (UAV-UAV) links on critical control functions
results in fast and seamless handoof active-mode UEs as
well as tracking of idle-mode UEs across multiple UAVs. Our
design decisions are ecient and scalable for airborne LTE
networks consisting of a few to at most tens of UAVs.
Real-world prototype:
We have built a complete version
of SkyCore on a single board server with a small compute
and energy footprint, and deployed it on DJI Matrice 600 Pro
rotary-wing drones to create a two-UAV LTE network. To
the best of our knowledge, this is the rst realization of a
self-contained Edge-EPC solution that can support a multi-
UAV network and is a direct armation of SkyCore’s design.
SkyCore’s feasibility and functionality are validated by seam-
less integration and operation with a commercial LTE RAN
(BS) from ip.access and o-the-shelf UEs (Moto G and Nexus
smartphones). We demonstrate SkyCore UAVs to operate
both as LTE hotspots that allow for better UE connectivity
to the Internet by extending coverage of a terrestrial LTE
network, as well as standalone LTE networks for connectiv-
ity of geographically separated UEs through two dierent
UAVs (e.g., rst responders in emergency scenarios), while
also allowing for handos. Our real-world evaluations of
SkyCore and its comparison with a state-of-the-art software
EPC (OpenEPC [
12
]) on UAV clearly showcases SkyCore’s
PCRF
MME
[NAS, S1AP, SCTP]
Packet Network
PGW
GTP-U
HSS
RAN EPC
SGW
Data plane Control plane Signaling Data
eNB
UE
Diameter
Diameter
Diameter
GTP-C
GTP-U
IP
GTP-C
Fig. 2: Legacy EPC architecture.
superior performance and scalability SkyCore provides an
order of magnitude lower control plane latencies, incurs 5
lower CPU utilization, and provides data plane rates that
currently scale up to a Gbps.
Our two key contributions in this work include,
A novel Edge-EPC solution, SkyCore that can reliably
and scalably support a multi-UAV LTE network deploy-
ment that was not possible earlier.
A real-world implementation and evaluation that show-
case both its feasibility and its superior performance.
Broader implications:
SkyCore’s underlying design is driven
by the observation that when connectivity between core net-
work functions, which are on the critical path, is unreliable
(wireless and mobile), the merits of pushing functionality to
the edge of the network signicantly outweighs the associ-
ated drawbacks. Hence, although designed for a multi-UAV
environment, SkyCore’s design can also benet other deploy-
ments, where distributed critical functions have to communi-
cate over unreliable links (e.g., distributed enterprise RANs).
2 MOTIVATION
(a) (b)
Fig. 3: (a) AT&T’s and (b) Verizon’s Cell on Wings (CoW).
Both adopt the legacy EPC architecture with wireless con-
nectivity (e.g., using a satellite relay or directly based on
other technologies) between UAVs (eNBs) and the EPC de-
ployed on the ground (e.g., on trailers hosting the core [3])
or in their cloud network. AT&T used a tether to provide a
ber data connection (possibly to support a wired EPC on
the ground if available) and power to the UAV for unlim-
ited ight time. In Verizon’s tests, there was no tether to the
ground but the EPC was still on/in the ground/cloud.
2.1 UAV-based LTE Networks
We consider low-altitude UAV networks, such as those con-
sidered by mobile operators [
15
,
16
] for on-demand LTE
network deployments. These are envisioned to provide tem-
porary LTE coverage from the sky to a designated area on
the ground. One use case is to enhance connectivity at big
events (e.g., music festivals, football tournaments) by creat-
ing a hotspot UAV-based LTE network extending capacity/-
coverage of local terrestrial LTE networks (macrocells). The
second use case is to establish a standalone UAV-based LTE
network for rst responders and the general public battling a
disaster where no infrastructure is available. Such a network
can keep them connected through local group communica-
tion and peer-to-peer (P2P) services, e.g., push-to-video.
The relevance of our work to high-altitude, long-endurance
platforms like Google’s Loon [
8
] and Facebook’s Aquila [
6
]
is discussed in section 9. In a UAV-based LTE network, an
LTE BS (eNB) is directly deployed on each UAV, and multiple
of them together provide wireless connectivity to UEs over
a desired wide area as shown in Fig. 1. However, not much
thought has been paid towards the deployment of an EPC to
support such a RAN. Deploying and managing a traditional
LTE EPC is a challenge in its own right. Designing one to
support an LTE RAN on UAVs, which are highly restrictive in
their compute capabilities, endurance and payload capacity,
further amplies the associated challenges.
To foster a better understanding, we rst provide a primer
on EPC’s key functionality, followed by the limitations of
today’s EPC for our target environment, and the benets
and drawbacks of an “alternate" Edge-EPC architecture.
2.2 EPC Primer
Fig. 2 shows the network architecture of the EPC, which is a
distributed system of dierent nodes, each consisting diverse
network functions (NFs) that are required to manage the LTE
network. The EPC consists of data and control data planes:
the data plane enforces operator policies (e.g., DPI, QoS
classes, accounting) on data trac to/from the user equip-
ment (UE), while the control plane provides key control and
management functions such as access control, mobility, and
security management. eNodeBs or eNBs (RANs) are grouped
into logical serving areas and connected to serving gateways
(SGWs). The SGW is connected to an external packet net-
work (e.g., the internet) via the packet data network gateway
(PGW). The PGW enforces most of the data plane policies
(e.g., NAT, DPI) and may connect the core to other IP network
services (e.g., Web servers). The EPC forwards each UE’s data
trac between the eNodeB and PGW using a separate GTP-U
(GPRS tunneling protocol) tunnel. The mobility management
entity (MME) is responsible for access control, security and
mobility functions (e.g., attach/detach, paging/hando) in
conjunction with the home subscriber server (HSS) database.
2.3 Limitations of Legacy EPC Architecture
The straightforward way to apply the EPC to our UAV net-
work would be to collapse all the EPC network functions
into a single node (EPC-in-a-box) and deploy this EPC node
on a resource-capable machine on/in the ground/cloud that
can support multiple UAV BSs. Based on publicly available
information [
5
,
7
,
14
16
], this is the approach adopted by
operators like AT&T and Verizon in their recent trials (Fig. 3).
(a) (b)
Fig. 4: (a) Degraded throughput on the EPC-RAN link (10
MHz LTE link) when an LTE UAV ies in LOS and NLOS
(over a building) trajectory with a 3GPP-compliant EPC be-
ing on the ground (detail on the EPC and RAN in section 6).
(b) # of SCTP/TCP (signaling/data) retransmissions in NLOS.
2.3.1 Tethered Deployment (Wired EPC-UAV link). In to-
day’s traditional LTE networks, the connectivity between
the EPC and eNBs (RAN) is a reliable, wired network provi-
sioned with sucient bandwidth for catering to the UE trac
demands in both downlink and uplink. A similar approach
can be adopted in our UAV network, where the RAN runs on
the UAV, which is tethered by a wire to a ground station run-
ning the EPC (Fig. 5a and possibly in Fig. 3a). However, such
an approach signicantly limits the potential and exibility
of the UAV to y and re-position itself to cater to network
trac requirements, not to mention the associated safety
concerns and the infeasibility of scaling such a setup to sup-
port a network of UAVs. With UAV technology advancing at
a rapid pace to provide longer operational times [1], such a
tethered EPC-on-ground does not oer a viable, future-proof
solution.
(a) Legacy Wired EPC
RAN RAN
EPC EPC
RAN RAN
(b) Legacy Wireless EPC
Wireless
Wire
Fig. 5: Legacy EPC variants for UAV-based LTE networks.
2.3.2 Un-tethered Deployment (Wireless EPC-UAV link).
The other alternative is where the connectivity between the
EPC and eNBs (UAVs) is wireless (Figs. 3, 5b). In this design,
the EPC could be placed on the ground (e.g., on trailers
hosting the core [
3
]) or in the operator’s cloud. The backhaul
between UAVs and the EPC could be routed through satellite
(in case of the cloud or ground EPC) or another wireless
technology, e.g., mmWave, directly between UAV and EPC
(in case of the ground EPC). While relying on the satellite
backhaul is an expensive option that may not scale to large
and particularly time-sensitive trac demands, the ground-
EPC option also has its share of drawbacks that we elaborate.
Reliability vs. range vs. capacity:
The wireless channel
is inherently an unreliable medium, and is subject to wire-
less artifacts such as shadowing (building, trees, obstacles),
multipath fading, etc. that can signicantly degrade signal
quality (by as much as 70% in our experiments, Fig. 4a) and
cause high packet retransmissions (more than 100 SCTP/TCP
retransmissions, Fig. 4b) and potentially cause disconnec-
tions. The choice of the wireless technology also plays an
important role. Using lower frequencies like 700 MHz, 1 GHz,
etc. allows for better penetration and hence longer commu-
nication ranges and better reliability but signicantly lesser
bandwidth (capacity of few tens of MHz). In contrast, higher
frequencies like mmWave (28 GHz, 60 GHz, etc.) oer sig-
nicantly more bandwidth (hundreds of MHz to a GHz) but
suer from higher attenuation and hence lower range. While
the latter can employ beamforming to cope with attenuation,
they are limited by line-of-sight requirements and the need
to constantly track the beam direction with respect to each
UAV as they move impediment for reliable operation in
low-altitude deployments. Thus, it is extremely challeng-
ing to identify a wireless modality for the critical EPC-RAN
(ground to UAV) link that can oer the simultaneous features
of reliable connectivity, increased communication range, and
capacity, that is warranted by this EPC deployment.
Single-point bottleneck:
The EPC node on the ground
becomes the routing focal point that ferries trac not only
between UEs and the Internet but also between UEs within
the UAV network. Hence, even if the UAV backhaul (connec-
tivity between UAVs) is well-provisioned, having a small set
of ground EPC nodes, concentrates all the trac on the UAV
backhaul towards these ground nodes, which in turn become
the bottleneck. This would signicantly degrade the capacity
of the network as a whole. For a low-altitude UAV network
deployed to provide on-demand connectivity to a small geo-
graphic region, the bulk of the tracmight be local e.g.,
between users and content servers in events, or between rst
responders and/or aected people in emergencies. In such
scenarios, incurring the wireless capacity bottleneck due to
the EPC on the ground is unwarranted.
A simple illustration in Fig. 6 shows that the capacity
oered by an EPC-on-ground architecture even for a small 4-
UAV network can be 6
lower than if the local trac were to
be served directly between the UAVs. Assume the capacity of
the EPC-UAV link (shared among all UAVs) is
X
and similar
to that of each UAV-UAV link. If there are
N
user sessions
between every pair of UAVs, each having bandwidth request
N/X
, the EPC-on-ground can satisfy
N/
6of the sessions as
opposed to the whole in the distributed EPC version.
In addition, UAV and UE mobility is highly pronounced
in these networks, which also leads to increased control
signaling and associated latency over multiple wireless hops
between the ground EPC node and the UAVs.
One option is to deploy multiple EPC nodes on the ground
to allow for more reliable connectivity to all UAVs and to add
capacity (akin to provisioning multiple gateways in wireless
mesh networks [
17
]). However, this adds to both the cost
as well as reliance on ground deployments, working against
the exibility oered by UAVs in the rst place.
2.4 Challenges in Edge EPC Architecture
To counteract the challenges in deploying a legacy EPC archi-
tecture, we focus our attention on a radically dierent “edge"
EPC
RAN
RAN
RAN
RAN
x
x
xx
x
x
x
xLink Capacity Wireless Link
Fig. 6: The capacity bottleneck
in the legacy wireless EPC.
RAN RAN
EPC EPC
Wireless Backhaul
User data (IP traffic)
Fig. 7: Edge-EPC for UAV-
based LTE networks.
EPC architecture (referred to as Edge-EPC). Here, the entire
EPC is collapsed and located as a single, self-contained entity
on each of the UAVs as shown in Fig. 7. Being completely
distributed at the edge of the network, such an architec-
ture would completely eliminate the crippling drawbacks
faced by the previous architecture resulting from wireless
connectivity between the EPC and eNBs (UAVs). 12
While the Edge-EPC idea is denitely promising at the
outset, it does encounter a dierent set of challenges in its
realization.
(a) (b)
Fig. 8: Edge-EPC has high overheads on UAVs and results in
performance bottlenecks and degraded user experience.
2.4.1 Resource-challenged. An EPC consists of multiple
network functions along with the complex interfaces and
tunneling protocols between them (Fig. 2). Further, most
of these are stateful network functions and consist of both
control and data plane functionality. These network func-
tions, which used to be deployed by operators on specialized
hardware, are now slowly migrating to a virtualization envi-
ronment with the recent advances in NFV (network function
virtualization [
11
,
18
,
45
]). Nevertheless, the compute re-
sources consumed by these network functions is appreciable
and becomes a concern when all the EPC functionality is
collapsed onto a single node. Deploying an EPC node on the
1
In Edge-EPC, the hotspot use case is realized by connecting the local
EPC hosted on a subset of UAVs that are in proximity of macrocells to the
Internet. To create a standalone network, content servers are co-located
with and attached to the local EPC on a desired subset of UAVs. In both
modes, the EPC-RAN (EPC-UAVs) link is highly reliable (being co-located
on the same UAV), unlike the drawbacks experienced by the legacy wireless
EPC architecture with a wireless EPC-RAN link.
2
Implementing the EPC functionality at macrocells is not a favorable option
for realizing Edge-EPC. Due to reliance on the xed infrastructure, it cannot
be applied to the standalone use case (e.g., deployment in remote areas).
Similarly, it is not viable for the hotspot use case due to lack of compute
resources for running EPC on cell towers.
Fig. 9: Edge-EPC fails in seamlessly handling increased
handos in our LTE UAV environment.
UAV could signicantly aect both its operational lifetime
as well as the processing (control and data plane) latency
of its trac, thereby resulting in a highly reduced trac
capacity. This can be observed in Figs. 8a, 8b, where the
latency and CPU utilization of control plane functions can
be an order higher in Edge-EPC, when the platform (such as
that on a UAV) is resource-constrained (experimental details
in section 6). Given the EPC is a complex distributed sys-
tem, identifying the key sources of resource consumption
in the EPC and to the extent that each source contributes to
resource consumption is out of the scope of this work. We
refer the reader to previous works reporting other relevant
issues in the EPC [23, 30, 36].
2.4.2 Handling Mobility at the Edge. The conventional
EPC has a hierarchical structure, where a single PGW spans
multiple SGWs, and a single SGW spans multiple eNBs. As
the UE (in active mode) moves from one eNB to another
(hando), this is handled locally by its SGW. Further, every
UE has a tracking area (TA, set of neighboring eNBs) associ-
ated with it, which the EPC will use to page (all eNBs in its
TA) to locate it when in idle mode. When the UE moves out
of its current TA, it noties the EPC of its updated TA. Thus,
UE mobility is handled seamlessly in the legacy EPC.
Active-mode mobility (Handos):
Network dynamics
in the form of UE and/or UAV mobility forms a signicant
part of our operating environment. However, with the col-
lapse of the hierarchical architecture in Edge-EPC, one needs
to now enable communication between the EPC entities on
individual UAVs to enable seamless handoacross UAVs. In
today’s mobile networks, a UE hardly moves across dierent
PGWs within the same operator’s network (a single PGW
spans a signicantly large area - hundreds of miles). When
such an event does happen, the connection is terminated
with the existing PGW and re-established with the new PGW
causing service disruption. However, such events are the
norm rather than an exception in our environment. Fig. 9
illustrates the number of potential handoevents that can be
triggered due to appreciable signal strength variations even
during a short UAV ight (less than 20 m) in our experiments.
Each UE-measured Reference Signal Received Power (RSRP)
value in the shaded region by itself can trigger a hando
event in the network if another UAV that delivers suciently
RAN
SkyCore
Agent
Backhaul
Agent
RAN
SkyCore
Agent
Backhaul
Agent
SkyCore control plane protocols
SkyCore data plane protocols
Wireless backhaul
Legacy RAN-EPC protocols
In the
hotspot
mode
RAN
SkyCore
Master
Backhaul
Agent
Fig. 10: The SkyCore network architecture.
higher RSRP is available. Hence, it becomes critical to enable
seamless EPC-EPC communication for handling mobility in
the Edge-EPC architecture. This is needed to also handle
UAV mobility, i.e. when one UAV goes down for a re-charge
and is replaced by another UAV a migration of state from
one UAV (EPC) to another is imperative.
Idle-mode mobility (Tracking/paging):
With the abil-
ity to page idle UEs over large tracking areas (spanning
several BSs), it is fairly straightforward to locate any UE in
the network in the legacy EPC. This is, however, a challenge
for the Edge-EPC architecture, where there is no single PGW
that spans all the UAVs (eNBs). Further, since the notion of
tracking area disappears (due to collapsed EPC), locating a
UE when in idle mode appears to be infeasible, prompting
the need for new or adapted mobility mechanisms.
3 SKYCORE: DESIGN OVERVIEW
SkyCore adopts the Edge-EPC architecture as shown in
Fig. 10. SkyCore collapses the entire EPC and pushes it to
the edge of our network, namely at each of the UAVs them-
selves, where it is co-located with the RAN. While this com-
pletely eliminates wireless from the critical path between
the EPC and RAN, to address the challenges associated with
the Edge-EPC architecture, SkyCore introduces two novel
design components. Briey,
Software refactoring of the EPC functionality:
To re-
duce its compute footprint on the UAV, SkyCore adopts a
software refactoring approach to eliminate distributed EPC
interfaces and collapse all distributed functionalities (Fig. 2)
into a single logical entity. It realizes this by transforming
the distributed data plane functions into a series of switching
ow tables and associated switching actions (correspond-
ing to functions like GTP-U encapsulation/decapsulation,
charging, etc.). It also reduces control plane signaling and
latency by precomputing and storing (in-memory) several
key attributes relating to security keys, QoS prole, etc. for
UEs that can be accessed locally in real time without any
computation.
Ecient inter-EPC communication:
With every UAV
now running its own EPC agent, even a simple eNB-eNB
handoof an active UE across two UAVs now becomes an
inter-MME hando, which needs to be accomplished across
two dierent EPC agents. SkyCore enables a new control/-
data interface that allows agents on dierent UAVs to proac-
tively (in the background) synchronize the state of UEs. This
bypasses the real-time impact of wireless (UAV-UAV links) on
critical control path functions, allowing for seamless hand-
os and tracking of idle-mode UEs right at the edge. The
HSS equivalent in each SkyCore agent maintains the location
(anchoring SkyCore agent) of all UEs in the network. Hence,
when an agent sends a UE location update, the agents in
other UAVs update their HSS accordingly. Thus, whenever
trac needs to be sent from a SkyCore agent to a UE located
at another UAV, the HSS will reveal the destination SkyCore
agent at which the UE is anchored and to whom the trac
has to be routed. The actual routing path to be taken by the
trac on the mesh backhaul is then determined by SkyCore,
with the underlying backhaul topology information made
available by a backhaul agent that resides on the UAV3.
We now explain each of these design components in detail.
4 SOFTWARE REFACTORING OF EPC
4.1 Minimalistic SkyCore Agent
Architecture
Each SkyCore agent has a minimalist and UAV-aware SDN-
based architecture (Fig. 11), consisting of a controller that
executes the control functions to process UEs’ signaling traf-
c and to coordinate with other agents, and a switch that
processes user data trac. In the following, we describe six
high-level steps that we take to refactor and extend the EPC
functionality onto our agent architecture.
Step 1. Decoupling the EPC control and data plane
pipelines
. One of the main reasons behind the high com-
plexity and overhead of the EPC is its nodes performing
mixed control and data plane functions. To make the EPC
functionality suitable for UAVs, we rst decouple the EPC
control and data planes. Among the EPC nodes, the MME,
PCRF, and HSS are pure control nodes. Hence, our decou-
pling does not aect these elements, and only aects the
SGW and PGW. The resulting control components from the
decoupling are the PGW-C, SGW-C, MME, PCRF, and HSS,
and the data elements include the SGW-D and PGW-D (C
stands for control and D for data). While the benets of
decoupling control and data planes have been articulated
before [
28
,
34
], we apply it in the context of UAV networks
and enhance it substantially with the following mechanisms.
Step 2. Categorizing the functionality of the EPC
control plane
. Next, we categorize the EPC control nodes
based on their high-level functionality. In our decoupled EPC,
there are three types of nodes: (1) the SGW-C and PGW-C are
responsible for managing QoS policy enforcement on and
routing of user data trac, (2) the MME exchanges signaling
trac with UEs and eNBs, and (3) the PCRF and HSS dynam-
ically generate network security and QoS policies for the
other nodes. To compress the EPC functionality, we consoli-
date the nodes in each category on top of our agent controller
and remove the EPC distributed protocols as follows.
3
The design of the backhaul agents responsible for maintaining a well-
provisioned, connected wireless mesh topology is outside the scope of this
work.
SGW-D
PCRF
PGW-C
SGW-C
MME
HSS
PGW-D
LTE
Mobility
LTE
Policy
SCTP/
S1AP/NAS
UAV
Control
Consolidation
& optimization
Decoupling the EPC
control & data planes
Refactored Legacy Apps
Other UAVs
New SkyCore Apps
Inter-Agent
Communication
Other
UAVs or
Macrocells
EPC control plane
EPC data plane
UAV engine
Precomputed (In-Memory) Policy Data Store
SkyCore Controller
OpenFow/P4
Step 1 Step 2-6
Charging QoS &
Rate
GTP-U
Processing
Firewall
SkyCore Switch
SkyCore
Overlay
SkyCore Agent
The EPC
RAN
GTP-U
Pub/Sub
Fig. 11: SkyCore refactors the EPC functionality into a lightweight software-driven agent having new interfaces for interaction
with the local UAV and other UAVs.
Step 3. Collapsing the SGW-C, PGW-C, and MME
into light-weight SDN applications
. We extract the inter-
nal functions in the SGW-C and PGW-C and refactor them
into a single SDN application, LTE Policy Application, on
top of the controller. We do the same process for the MME
and transform it into LTE Mobility Application. One notable
aspect of this consolidation is that we naturally eliminate the
complex GTP-C protocol, its six interfaces, and continuous
control messages from the core network (Fig. 2). This makes
the SDN applications extremely lightweight and extensible
without hurting their original functionality. Note that these
applications still exchange information with each other but
through simple local publish-subscribe mechanisms.
Step 4. Eliminating the HSS and PCRF from the core
and replacing them with a precomputed policy data
store.
Next, we focus on the HSS and PCRF that are known
to be the source of today’s signaling storms in cellular net-
works [
10
,
13
]. The HSS stores hundreds of database tables
containing dierent UEs’ states often on disk. Moreover, it
acts as a proxy between the MME and these tables, and per-
forms dierent types of complex security and location track-
ing computations. The PCRF often accesses a logical database
(sometimes implemented in the HSS) and dynamically gener-
ates dierent QoS and charging policies for UEs. In SkyCore,
we completely eliminate these two nodes from our agents
and show that dynamic policy generation can be carefully
replaced with a precomputed in-memory policy data store.
Precomputation combined with in-memory transactions sub-
stantially minimizes the overhead of the core on resource-
challenged UAVs (elaborated in section 4.2). This also re-
moves the complex Diameter protocol (Fig. 2) from the core.
Step 5. Adding UAV-specic SDN applications to the
core
. One of the key dierences between SkyCore and the
traditional EPC is in its continuous interaction with the UAV
hardware and its APIs. In particular, we advocate for two new
applications on top of our agents. Each SkyCore agent runs
UAV Control Application that listens to ight change events
from UAV and remaining battery resources on the UAV. This
is necessary for our agents to properly handoUEs to each
other, e.g., when a UAV needs to immediately leave the net-
work for recharging. Such use cases clearly show the poten-
tial of our SDN-based UAV-aware architecture. In addition,
we design an Inter-Agent (UAV) Communication Application
(section 5) that exchanges control plane messages with its
neighbor agents to synchronize states proactively, thereby
enabling seamless mobility (active and idle). The legacy EPC
applications and new SkyCore core applications that need
to exchange information with each other, do so through our
local publish-subscribe protocols.
Step 6. Replacing the hierarchical data plane gate-
ways with a compact SDN switch
. Since SkyCore is a at
architecture, it eliminates the need for hierarchical gate-
ways on each UAV. To further make our agents compact, we
refactor the SGW-D and PGW-D functionality into a single
software switch. Each data plane function in S/PGW-D is im-
plemented as a separate Match+Action table in this software
switch. Each table performs a lookup on a subset of user’s
data tracelds and applies the actions corresponding to
the rst match. Users’ trac travels through these tables
before leaving or entering the UAV. In particular, our soft-
ware switch performs UL/DL data rates enforcement, stateful
rewall operations, and QoS control by transport-level mech-
anisms (e.g., setting DiServ) based on QoS class identier
(QCI) associated with each UE. While the legacy EPC tunnels
each UE’s trac into two tunnel segments across the RAN,
PGW-D, and SGW-D, SkyCore departs from this approach
and terminates GTP-U tunnels inside our agent switch (de-
capsulates GTP-U header from uplink packets sent by the
eNB and encapsulates downlink packets to the eNB into a
proper GTP-U header) for two reasons. First, per-UE tunnels
do not scale in LTE UAV networks as UEs are mobile and
these tunnels are subject to frequent changes. Second, our
consolidation already eliminates the need for complex GTP-
U tunnels between the SGW-D and PGW-D functionality.
4.2 SkyCore Precomputed Policy Data Store
We now describe how the HSS/PCRF can be replaced with
precomputed network policies on SkyCore agents. As shown
in Fig. 12b, the SkyCore data store associates each UE’s IMSI
(International Mobile Subscriber Identity) information with
its precomputed policies, which can be quickly accessed
by dierent applications on the agent. Our precomputation
approach not only reduces the user-perceived network access
delay but it also makes the core extremely resource-ecient.
00101123456…
00101123456… {Security_profile: {},‘Service_profile’:{‘max_up_bw’: 50mbps, ‘QCI’: 5,…},}
IMSI (Keys) Security and service profiles
{’Security_profile’ :{‘k’: 10, {‘Vec1’: (AV1,EV1) , ‘Vec2’: }, Service_profile: {}}
(b) Skycore’s precomputation of network policies
Precomputed In-Memory Policy Data Store
F1 F2 F3 F4 F5
MAC
SQN
CK 1K AK
KASME
F7
KeNB
F8 F9
KNAS-enc KNAS-int
RAND
K
XRES
F6
AMF
AK
AUTN
Encryption & Integrity Vector (EV)
(KNAS-int, KNAS-enc, KeNB)
Authentication Vector (AV)
(RAND, XRES, CK, 1K, ATUN)
(a) EPC’ heavy security procedures at HSS
Fig. 12: SkyCore’s precomputation of network policies not
only makes the core resource-ecient but also minimizes
network access delay.
Note that our target deployment area spans a few tens of
square miles and each operator knows the last location of
each UE in its network (stored in the HSS). Thus, performing
the policy precomputation for the UEs located in a superset
of the deployment area is sucient. In rare scenarios, where
a valid UE (coming from outside of the superset) tries to
connect to a UAV, we can fall back to online computation
at the local agent and then update the other agents in the
network to avoid future online computation for the same UE.
In section 5, we will discuss how online policy computation
and policy changes by an agent are propagated to other
agents in the network to ensure their state is consistent.
4.2.1 Precomputation of Security policies. Network Ac-
cess Security in LTE networks relies on the shared user-
specic key,
K
, that is stored in the HSS and UE SIM cards.
The LTE security processes assume that cloned UEs and
spoofed networks do not know the correct value of
K
. From
K
, the HSS dynamically computes (shown in Fig. 12a) an
authentication vector (AV) and an encryption vector (EV)
as part of a larger LTE attach process, when UEs switch
on or enter an area with LTE coverage. The EPC and the
UE conrm each other’s identities using the AV. The sig-
naling trac between the UE and the network is encrypted
using the EV to ensure intruders cannot read and modify
them. The computation of these vectors involves resource-
intensive cryptographic operations (F1-F9) on 256-bit long
strings, thereby wasting valuable clock cycles on UAVs.
Oline computation of security vectors.
When there
are a few UEs, the overhead of computing such security vec-
tors on UAVs is manageable. However, when UAVs are provid-
ing on-demand LTE connectivity over a large geographical
region, many UEs are likely to send LTE attach requests to
the network at the same time. Such realistic workloads can
quickly use the available compute resources (for the core) on
UAVs and substantially degrade the QoE of users. To resolve
these issues, our key idea is to depart from real-time security
vector computations on UAVs. We precompute and store a
reasonable number of security vectors for each UE and store
them on the SkyCore agent. All these vectors are computed
with the same
K
, and
RAN D
(a random number generated by
the HSS) but with dierent consecutive sequence numbers
(
SQN
). Dierent
SQN
numbers ensure signaling messages
cannot be replayed by intruders.
Since each pair of vectors is computed with a dierent
SQN
number, it can be used only once by SkyCore during
the LTE attach procedure. If the same pair is reused, the UE
will reject the network assuming it is a spoofed network that
is trying to replay old authentication messages. Thus, each
of our agents locally removes a used pair of security vectors
and invalidates it at other agents through our Inter-Agent
Communication application (see Fig. 11 and section 5). Note
that the number of attach requests generated by a legitimate
UE, when it switches on or comes back into network cover-
age, is limited. Hence, SkyCore precomputes a small number
of such vectors for a UE. In rare cases, when a UE uses all its
precomputed vectors (e.g., due to frequent restarts), SkyCore
agents fall back to computing new vectors for such UEs in
real time and propagate them to other agents in the network.
To nd the best conguration that simultaneously minimizes
the number of online computations and storage needed for
precomputed vectors at UAVs, a SkyCore operator can look
at histograms of the number of LTE attach/detach events [
28
]
for each UE in its network.
4.2.2 Precomputation of Service policies. In LTE networks,
the PCRF dynamically generates quality of service (QoS) and
charging rules for a UE. The PCRF continuously feeds the
PGW and SGW with real-time QoS rules. Rather than gen-
erating these rules in real time by accessing many dierent
tables, we precompute the entire rule set that must be ap-
plied to UEs’ trac, and consolidate and store them onto
our agents. In particular, SkyCore consolidates three types
of rules that deal with (i) QoS (bit rate, loss rate, etc.), (ii)
priority (ow handling during congestion), and (iii) charging
(oine, online and time-dependent). Most of these service
policy rules are almost always static (e.g., max. bit rate/loss
rate) and do not change over time. Thus, we do not need to
synchronize these policies among our agents as UEs move
around in the network. However, some of these rules oper-
ate on some internal states (e.g., remaining bandwidth in an
online charging model). Such dynamic states for dierent
UEs are batched and synchronized periodically among our
agents as they are often not time-critical (section 5).
5 EFFICIENT INTER-AGENT
COMMUNICATION
5.1 Scalable SDN Control and Data Overlays
SkyCore agents seamlessly exchange control and data trac
with each other, a functionality that is lacking among today’s
Fig. 13: (a) SkyCore’s network-wide control and data plane
connectivity for LTE UAV networks. (b) Example of our seg-
ment routing for data trac from UE1 to UE2.
EPC instances. Rather than relying on distributed and multi-
hop wireless routing protocols, we choose to adopt an SDN
approach in the design of SkyCore to support the trac ex-
change between agents. SDN enables us to perform global op-
timization (e.g., multipath trac engineering) and oer ne-
grained programmability (e.g., to eectively support dierent
QoS classes), which are necessary tools to instantly and e-
ciently recongure the core in response to network dynamics
(e.g., UAV departures and arrivals) in our environment. In par-
ticular, we leverage SDN overlays to create two virtualized
network layers (slices) on top of the physical UAV network
(Fig. 13a). One of these network slices is used for control
plane trac between SkyCore agents and the other is for data
trac. Our separation of the control and data trac ensures
time-critical control plane trac is not aected when the net-
work is saturated. To form the overlays, we use trac tunnel-
ing technologies but depart from existing approaches used in
the EPC and SDN-based datacenter (DC) networking [
26
,
27
]
since they require frequent changes to the network cong-
uration (will be discussed shortly). We adopt a novel variant
of segment-based routing in SkyCore and propose a design
for its optimization based on the most advanced capability in
SDN, i.e., P4 language [
20
]. P4 allows us to dene new packet
headers and packet processing actions for the SDN switch
inside our agents to minimize the packet header overhead
on inter-UAV links, which is caused by forming the overlays.
Segment-based overlays equipped with global source
routing. Tunneling:
We inter-connect each pair of neighbor-
ing agents (geographic proximity) using a tunnel dened
with a label. Whenever an agent decides to send control or
data plane trac to any other agent in the network, it pushes
a stack of labels onto the packets. The top-of-the-stack (TOS)
label corresponds to the next tunnel segment that the packet
must traverse. Whenever an agent receives a packet from
its neighbor, it checks the TOS label from the packet and
forwards the packet based on the inner label to its neighbor.
There is a master that is responsible for globally computing
an ecient label stack that each agent must use to com-
municate with the other agents. Instead of adding separate
MPLS packet headers for each label, SkyCore designs a new
packet header based on the P4 language to contain all the
labels in the stack to reduce overhead. It equips switches
with new actions to read the labels at dierent positions.
Routing:
In SkyCore, one of the UAVs is selected (period-
ically and randomly) to double up as a master agent that
is responsible for global route computation. It periodically
collects information from other agents, related to average
loss rate and bandwidth on wireless links between dier-
ent agents (UAVs), remaining battery capacity on UAVs, and
the amount of trac demand between dierent UAVs. The
master agent uses this information to compute and dissemi-
nate forwarding rules for routing trac over the UAV mesh
backhaul in the sky. For eciency, a simplied version of
the leader selection algorithm introduced in the LEACH pro-
tocol [
25
] can be used for the master selection in SkyCore.
Proximity-based segments enable scalability:
Note that UAV
and UE mobility is common in our environment. Hence, a
conventional EPC approach to establishing per-UE tunnels
(GTP-U tunnels–see section 2) will require frequent tunnel
updates (tearing down, modication, or setting up). Simi-
larly, employing a tunnel between every pair of UAV agents
(akin to remote DC-DC tunnels in Google’s B4 backbone
network [
27
]) will require updates to a large fraction of the
tunnels, even when only a small number of UAVs move. In
contrast, most of SkyCore’s tunnel segments do not change
in such scenarios as they are designed to carry aggregate
trac only between nearby pairs of UAVs.
Example:
A simpli-
ed example of our tunneling scheme is depicted in Fig. 13b,
transporting trac from UE1 to UE2 across three UAVs. Note
that the source and sink agents perform an additional GTP-U
processing, introduced in section 3, to seamlessly eliminate
GTP-U from the core.
5.2 Proactive Stateless Mobility Support
SkyCore replaces the notion of centralized HSS and PCRF
with a precomputed policy data store replicated at dierent
agents. Hence, it is essential that the UE states and policies
be consistent across dierent agents, particularly during UE
mobility. Reactive approaches to consistency management
e.g., Distributed hash table (DHT) [
46
], put wireless (inter-
UAV links) on the critical path of control functions. SkyCore
avoids this real-time dependence by adopting a proactive
synchronization of state between agents each agent proac-
tively broadcasts its changes to UE policies and states to
other agents in the network. Such an approach (i) minimizes
the control plane delay between agents, particularly in mo-
bility scenarios as the destination agent already knows the
latest information about the mobile UE; (ii) enables seam-
less handoof active UEs to a neighboring UAV, when the
current UAV goes down for a recharge; and (iii) is scalable
because the amount of control plane trac that is broad-
casted on inter-agent backhaul links is negligible compared
to user data plane trac among agents (section 7).
!
eNodeB-1!
LTE!UAV!2!
LTE!UAV!1!
Backhaul!
SkyCore!
Agent-1!
!
SkyCore!
Agent-2!
!
eNodeB-2!
!
Fig. 14: Multi-UAV SkyCore prototype.
A SkyCore agent needs to send only three types of broad-
cast update messages in the network to build up a consistent
network-wide view: (i) security update to notify other agents
that it has used one of the security vectors precomputed for
a UE and to request other agents to invalidate the vectors, (ii)
location update to inform other agents that a particular UE
has attached to its UAV, and (iii) policy update to communi-
cate its local changes to the precomputed QoS and charging
prole of a UE.
5.2.1 Idle/Connected-mode Mobility. SkyCore’s proactive
state synchronization scheme accelerates the handling of
increased mobility events in multi-UAV LTE networks.
Idle-mode mobility (Paging).
The underlying Edge-EPC
design in SkyCore limits each UAV (BS) to its own tracking
area (TA). Hence, when an idle-mode UE moves from one
UAV to another, it realizes a change in its TA on waking up
(prompted by a periodic timer) and sends a TA update re-
quest to the SkyCore agent on the new UAV. Since the agent
at the new UAV already has the UE’s latest policies and states
from SkyCore’s proactive updates, it knows which security
vectors to use for communication with the UE. Hence, it
immediately sends a TA update response back to the UE,
which can then quickly switch back to its idle mode to con-
tinue saving power. In the background, it also broadcasts the
updated location of the UE to all other SkyCore agents in the
network, eliminating the need for explicit UE paging. This
also ensures that the other agents can push the correct label
stack on the packets destined for this UE.
Active-mode mobility (Handos).
Now, consider the
UE to instead be in the connected (active) mode during the
move. Based on the LTE protocol, it performs continuous
signal strength measurements and sends them to the rst
UAV’s eNB. If the eNB detects the signal strength of the
neighboring second UAV to be stronger, it sends a handore-
quest message to its SkyCore agent. This agent then noties
the agent on the second UAV of the incoming UE (without
having to transfer/update any state on the destination agent)
and then conrms the handowith its own eNB, which then
informs the UE. Then, the UE connects to the eNB on the
second UAV, whereupon its SkyCore agent noties all other
agents in the network with a location update for this UE.
Finally, our agent on the rst UAV pushes the updated label
stack corresponding to the UE onto its pending downlink
packets and forwards them to the second UAV.
6 IMPLEMENTATION
SkyCore prototype
. We prototyped a complete version of
SkyCore that involved extensive engineering eort. Our pro-
totype has three notable features: (1) seamlessly works with
commercial LTE RANs and o-the-shelf UEs (SIM cards are
programmed to connect to SkyCore) by exchanging signal-
ing and data trac with them; (2) is fully virtualized and can
manage multiple LTE UAVs out of the box by forming a wire-
less network of SkyCore agents; and (3) fully adheres to our
proposed designs both for a single agent (Figs. 11 and 12) and
across agents (inter-agent communication) (Figs. 10 and 13).
Each SkyCore agent consists of a controller enforcing con-
trol plane policies and a switch processing user data trac.
We developed a high-performance multi-threaded controller
in C++ and built our SkyCore switch on top of OVS [
42
]
software switch in the kernel space. We substantially in-
strumented and optimized OVS as it does not support our
custom ow tables and switch actions (e.g., our P4-enabled
tunneling scheme and GTP-U tunnel encapsulation/decap-
sulation operations). Since our baseline (Edge-EPC based on
OpenEPC [
12
]–will be described shortly) operates in the user
space, we developed another variant of the SkyCore switch
in the user space on top of Lagopus software switch [
9
]. This
ensures that our comparisons are at the architecture level and
independent of a particular packet forwarding technology.
UAV experiments.
We conduct three kinds of experi-
ments. (1) Outdoor Small-scale: 2 UAV, few UEs. We deploy
the SkyCore prototype on two advanced DJI Matrice 600
Pro drones. We securely install two machines on each drone.
One of the machines (platform P1) is a low-end single-board
4-core server with 8 GB of RAMs and 1.9 GHz CPU that
executes SkyCore and Edge-EPC. It is also equipped with a
wireless network card to support our inter-agent communi-
cation. The other machine is a commercial LTE small cell
(ip.access S60 eNB) supporting LTE UEs (50 Mbps downlink
rate per UE) and connects through an Ethernet cable to plat-
form P1. (2) Outdoor Large-scale: 2 UAV, tens of UEs. To stress
test SkyCore’s control and data planes in the presence of a
large number of UEs, we replace the eNB on the drone with
another single-board server that runs a unied RAN/UE em-
ulator (emulates both an eNB and activity of a large number
of UEs). The emulator interacts with the LTE core similar to
real UEs. (3) Emulating Powerful UAV platforms. To under-
stand SkyCore’s performance with more powerful UAVs, we
emulate the latter by replacing platform P1 with a high-end
server (platform P2) an Intel Xeon E5-2687W processor
operating at 3.0 GHz with 12 CPU cores and 128 GB of RAM.
Since it is not possible to y our current drone with such a
server, only these experiments are conducted in the lab.
Baseline.
Through outdoor drone-based experiments, we
have already demonstrated the limitations of the legacy wire-
less EPC on the ground (section 2). Thus, this section is fo-
cused on comparisons between the Edge-EPC architecture
(a standard EPC on each LTE UAV) and SkyCore. We choose
to implement Edge-EPC using OpenEPC [
12
] because it is
the most complete open-source implementation of the 3GPP
EPC architecture that can work with commercial devices
(e.g., LTE eNBs and smartphones). Other non-standard and
more optimized EPC designs (listed in section 9) are not
publicly available, often do not work with the commercial
LTE devices, and inherit most of the standard EPC prob-
lems in our UAV-based LTE network environment (e.g., high
resource usage, lack of support for inter-EPC coordination).
Metrics.
We study four performance metrics under dier-
ent network saturation levels: (1) UE-perceived control delay
in network access (LTE attach/detach), (2) UE-perceived ser-
vice disruption time in LTE active/idle-mode mobility, (3)
CPU usage on our resource-constrained UAVs, and (4) sup-
ported data plane rate for user trac.
7 EVALUATION
We rst show the basic functionality and potential of Sky-
Core in realizing hotspot and standalone LTE UAV networks.
We then demonstrate that SkyCore is more ecient and light-
weight than the Edge-EPC architecture on dierent platforms
both in small and large-scale experimental settings, thanks
to SkyCore’s software refactoring and ecient inter-agent
communication scheme.
eNodeB-1
SkyCore
Agent-1
Backhaul
Agent-1
Moto G
A Terrestrial
LTE/EPC Network
eNodeB-1
SkyCore
Agent-1
eNodeB-2
SkyCore
Agent-2
Backhaul
Agent-1
Backhaul
Agent-2
Moto G Nexus 6
(a) Hotspot—4K streaming & browsing (b) Standalone—UE-to-UE HD video call
Moto G
SkyCore control plane protocol SkyCore data plane protocol Wireless backhaul Legacy EPC protocols
Fig. 15: Our UAV-based setup for the basic functionality ex-
periments. In Edge-EPC (the baseline), each SkyCore agent
is replaced by a 3GPP EPC (OpenEPC).
7.1 Small-Scale On-Drone Evaluation
We form a two-drone LTE network (Fig. 15), each in the par-
tial line of sight (aected by one building) of a single mobile
UE on the ground. Each drone covers a region with the diam-
eter of 650 feet. The drones operate in a small overlapping
area for our mobility experiments.
Fig. 16: Basic functionality–hotspot UAV-based LTE network
use case: exchanged data trac and control events over time.
7.1.1 Basic Functionality–LTE Hotspots Use Case. Form-
ing on-demand hotspots is an important use case for LTE
UAV as well as 5G networks (section 2). In a single-drone
experiment, we show this functionality by connecting one of
our drones to the Internet through a terrestrial LTE network
not accessible to our UEs on the ground (see Fig. 15a). Next,
we turn on a Moto G phone on the ground, which sends an
LTE attach request to the SkyCore agent through the on-
drone eNB. SkyCore agent successfully completes the LTE
attach process by quickly accessing its precomputed policy
data store. Then, we visit CNN.com and watch a 4K Youtube
video on the phone. Finally, we take the Moto G into the
airplane mode, causing the UE to properly detach from our
agent. Fig. 16 shows this basic functionality by depicting the
data trac exchanged between the UE and the Internet.
755 5
Proactive
Update
850
1. LTE Attach
Process
3. LTE Attach
Process
Proactive
Update
2100
2100 300
7. LTE Idle-mode
Process
310
P4-based
Segment Routing
6. User Traffic
Processing
5
5. User traffic
Processing
SkyCore Agent—Switch
SkyCore Agent— Controller
4. Policy Store
Update Process
2. Policy Store
Update Process
8. LTE Idle-mode
Process
Time
(ms)
SkyCore Agent-1
SkyCore Agent-2
Time
(ms)
Moto G
Nexus 6
Fig. 17: Basic functionality-standalone UAV-based LTE net-
work use case: UE-to-UE HD video call enabled by SkyCore’s
ecient inter-agent communication scheme. Control and
data plane processing times and trac exchanges inside and
between the SkyCore agents on the two drones.
7.1.2 Basic Functionality–Standalone LTE Use Case. Next,
we demonstrate SkyCore’s ability to create standalone LTE
networks (e.g., between rst responders across an impass-
able mountain as discussed in section 2). To emulate such a
scenario, we establish a direct video call between our two
UEs across a building, each connected to a separate drone,
through our inter-agent data plane overlay (see Fig. 15b).
Fig. 17 shows the timeline of control and data plane trac
exchanges between the two SkyCore agents. We again turn
on a Moto G phone in the area covered by the rst drone. Its
SkyCore agent handles the LTE attach process and sends a
background SkyCore update message to the other drone’s
agent. The message consists of location, policy and security
updates as described in section 3. After the second agent pro-
cesses this update, we turn on a Nexus 6 phone in the area
covered by the second drone, triggering a similar SkyCore
update message to the rst agent in the background. Finally,
we establish a 35-sec HD video call from the Nexus 6 to the
Moto G. Owing to SkyCore’s proactive background updates,
the agent corresponding to the Nexus 6 does not have to
wait to discover the location of the other UE. Based on our
segment-based tunneling scheme, it immediately pushes the
correct label stacks on its egress user data trac and for-
wards it to the other agent. A similar process manifests in the
reverse direction. In this two-UAV enabled video call, 7.5K
video packets were successfully exchanged between the UEs.
7.1.3 Performance Benefits of Refactoring. Using the same
setting, we demonstrate that SkyCore is signicantly more
lightweight than Edge-EPC. For a fair comparison with Edge-
EPC, we employ SkyCore’s user-space version here. We sam-
ple and average the LTE attach/detach delay and uplink/-
downlink bandwidth for the Moto G in the area covered by
the rst drone at 40 locations. As Fig. 18 and Table 1 show,
SkyCore on average reduces the network control plane de-
lay (spent in the core) by 69%-90% and the UE-perceived
control plane delay by 40%-60%. In addition, it doubles the
uplink/downlink rates measured for the UE. Further, Sky-
Core lowers the avg. CPU usage on the machine running
the core network by 25% in the LTE attach/detach events.
These savings come from our precomputation of network
policies and consolidation of the EPC functionality onto our
compact SDN-driven agents.
Table 1: Benets of SkyCore’s software refactoring of the
EPC functionality on UE-perceived QoS.
Avg. Data plane
Bandwidth (Mbps)
Avg. UE-perceived
Control delay (ms)
Downlink Uplink Attach Detach
SkyCore 48.2 17.8 921 300
Edge-EPC 21.7 10.9 1545 750
Fig. 18: Breakdown of the network access delay in the core.
Fig. 19: Benets of SkyCore’s inter-agent communication
scheme: SkyCore provides seamless active-mode mobility
support while Edge-EPC causing severe connection drops.
7.1.4 Eicient Inter-Agent Communication–Hando.Un-
like Edge-EPC, SkyCore supports seamless UE mobility, ow-
ing to its ecient inter-agent communication scheme. In
this experiment, we measure the service disruption experi-
enced by a mobile UE moving between the regions covered
by our two drones and triggering a handoevent. Fig. 19
depicts the signal strength received from the two drones on
the UE and its continuous bandwidth measurements using
iPerf3
. The RAN on the rst drone collects UE-measured
RSRP values and sends a HandoRequired message to its
local SkyCore agent when the RSRP values from the second
drone become higher. Since SkyCore agents on the drones
are already synced, the UE gets migrated to the second drone
within a minimal 140 ms (incurred in the inter-agent coor-
dination). In contrast, Edge-EPC does not support mobility
of the UE and thus forces the UE to go through the detach
process with the EPC on the rst drone, followed by the
heavy attach process with the EPC on the second drone. The
entire process results in 2 seconds of disconnection time,
signicantly impacting mobile application performance.
(a) Platform P1 (b) Platform P2
Fig. 20: Benets of SkyCore’s software refactoring at scale.
SkyCore substantially reduces network access time in LTE
UAV networks within the limits of their resources.
(a) Platform 1 (b) Platform 2
Fig. 21: Benets of SkyCore’s software refactoring of the
EPC at scale: SkyCore uses minimal CPU resources to han-
dle large-scale network access requests.
7.2 Large-Scale On-Drone Evaluation
Using the same two-drone experimental setting, we replace
the ip.access eNB with a RAN/UE simulator on each drone
to test SkyCore and Edge-EPC under large-scale network
access and mobility workloads.
7.2.1 Performance Benefits of Refactoring at Scale– At-
tach/detach Storm. This experiment demonstrates SkyCore’s
operating potential in highly resource-constrained UAV envi-
ronments. Our RAN/UE emulator on the rst drone emulates
aash crowd event with a large number of users entering the
region covered by a drone. Similarly, the emulator creates
LTE detach storms having many users gracefully discon-
necting from the drone. During this process, we sample the
CPU utilization of the LTE core machine and measure the
average control plane delay perceived by the UEs. In Fig. 20a,
we observe that the UEs experience exponentially larger de-
lays when the attach/detach load on Edge-EPC increases.
In particular, when the number of attach requests per sec.
reaches 100, the UEs must wait by up to 6 seconds before
connecting to the network, thereby degrading QoE. In con-
trast, we notice that the network access delay is below 1s
when the drone employs SkyCore because of our software
refactoring of the EPC functionality. To better understand
the reason, we look at Fig. 21a showing the CPU utilization
of the LTE core machine. Since the EPC is a complex system,
we observe that Edge-EPC quickly uses available CPU re-
sources on the drone and thus faces performance bottlenecks.
Although user-perceived control plane delay in the detach
process is usually less critical in practice, the same trend can
be observed for both SkyCore and Edge-EPC.
(a) Platform P1 (b) Platform P2
Fig. 22: Benets of SkyCore’s ecient inter-agent communi-
cation scheme at scale: SkyCore seamlessly supports large-
scale idle/active-mode UE mobility between UAVs.
(a) Platform 1 (b) Platform 2
Fig. 23: Benets of SkyCore’s ecient inter-agent commu-
nication at scale: SkyCore supports large-scale idle/active-
mode UE mobility in a resource-ecient manner.
7.2.2 Eicient Inter-Agent Communication at Scale– UE
Mobility Storm. This experiment demonstrates SkyCore’s
capability in handling increased mobility events in LTE UAV
networks. We add the second drone to our experiment. Our
RAN/UE simulator on the rst drone and second drone sim-
ulate scenarios where a large number of connected and idle
UEs move between the areas covered by the two drones. It
sends a variable number of LTE HandoRequired Messages
and TA Update Requests to the core network to trigger active-
mode and idle-mode mobility events. We increase the num-
ber of mobility events until either Edge-EPC or SkyCore face
performance bottlenecks. We measure the service disruption
experienced by the UEs when Edge-EPC and SkyCore are in
place as well as the CPU utilization of the LTE core machine.
Fig. 22a shows that the UEs experience a large control plane
delay and service disruption in the Edge-EPC deployment.
Due to lack of control plane communication between the
Edge-EPC instances, the UEs have to undergo a complete
LTE detach process (with the rst drone) and attach process
(with the second drone) both during connected-mode and
Table 2: SkyCore’s inter-agent communication scheme
(broadcast proactive updates) has a negligible overhead on
inter-UAV backhaul links and thus is scalable.
LTE attach/detach & mobility events per sec.
Platform P1 Platform P2
20 40 80 100 100 200 300 400
Bandwidth overhead on
Inter-drone Link (Mpbs) 1.2 2.2 3.8 5.5 5.5 8.5
14.2
18
idle-mode mobility. In Edge-EPC, when 100 mobility events
occur per second, users on average experience by up to 10
second of disruption, which is very signicant. More impor-
tantly, by transforming each mobility event to a pair of LTE
attach-detach events, we observe in Fig. 23a that Edge-EPC
creates severe bottlenecks on the drone platform. In contrast,
the SkyCore agents sitting on the two drones quickly and
seamlessly execute the handoand TA update operation, ow-
ing to our proactive synchronization of network policies and
states associated with dierent UEs in the background. Thus,
they incur minimal computation for mobility workloads.
7.2.3 Eicient Inter-Agent Communication at Scale– Over-
head of Proactive Updates. Next, we demonstrate the scala-
bility of SkyCore’s proactive synchronization scheme in our
multi-UAV environment. Our metric is the overhead of proac-
tive updates on inter-drone backhaul links. For the above
two experiments and various number of LTE attach/detach
and mobility events per second, Table 2 shows the maximum
rate of the broadcast trac on the backhaul link connecting
the SkyCore agents. We observe that the bandwidth over-
head does not increase linearly with respect to the number
of LTE events in the system as the updates corresponding to
the dierent UEs are periodically (every 100 ms in our setup)
batched together. Our second observation is that when the
load is 100 events per sec (representing tens of thousands
of UEs in today’s 4G/LTE networks [
28
]), the overhead on
the backhaul link is only 5.5 Mbps. Compared to potential
user data trac in the system, this overhead is negligible.
Using a back-of-the-envelope calculation, it is straightfor-
ward to show that this overhead remains small in our target
environment consisting of a few to tens of UAVs.
(a) Platform P1 (b) Platform P2
Fig. 24: SkyCore’s refactoring of the EPC increases the data
rate support on resource-challenged UAVs.
7.2.4 Performance Benefits of Refactoring at Scale– Stress-
testing Data Plane. In a single-drone experiment, we instruct
our RAN/UE simulator to generate data trac for a variable
number of UEs in the network in parallel. It encapsulates the
trac of each UE into a separate GTP tunnel similar to real
RANs. We run
iPerf3
bandwidth tests for the simulated UEs
(a) Platform P1 (b) Platform P2
Fig. 25: SkyCore’s refactoring of the EPC minimizes the CPU
resource needed on UAVs for a specic forwarding rate.
in parallel. Fig. 24a shows the aggregate, steady forwarding
rate supported by SkyCore and Edge-EPC. When using the
same packet forwarding technology, we observe that Sky-
Core (user space) supports 2
more packet forwarding rate
compared to Edge-EPC on the drone. Our software refac-
toring and data plane consolidation substantially removes
the I/O costs and processing delays from the LTE core data
plane. We were able to further improve the throughput by 2
(close to a Gbps) by moving our software switch to the kernel
space. Maximum is the ideal version of our OVS switch that
processes data trac without applying any network policies.
7.3 Scaling to Powerful UAV Platforms.
We replace the core machine (platform P1) with a high-end
server (platform P2) to emulate more power UAV platforms
in the future. Figs. 20b, 21b, 22b, 23b, and Table 2 show our
results with platform P2 for the previous four experiments.
We observe that SkyCore is substantially more resource-
ecient than Edge-EPC even on high-end servers. SkyCore
is able to scale and provide almost line-rate forwarding rate
while using a fraction of the drone’s CPU resources. We plan
to move SkyCore’s implementation to OVS-DPDK to more
eciently leverage the available CPU cores.
8 RELATED WORK
SDN/NFV-based EPC.
Recently, the wireless networking
community has proposed several software-dened EPC solu-
tions. SoftCell [
28
], SoftMoW [
34
,
35
,
38
] and others [
11
,
47
]
enhance the programmability of the EPC by decoupling its
control and data planes. KLEIN [
43
] and SCALE [
18
] opti-
mize the placement of the EPC nodes on geo-distributed
DCs. ECHO [
40
] deals with EPC-node failure in unreliable
public clouds. PEPC [
44
] and SoftBox [
36
] scale the EPC data
plane by creating a per-UE EPC-in-box. While there are some
similarities between SkyCore and these proposals, the dier-
ences are signicant. These prior designs are customized for
highly-reliable, often hierarchical DC infrastructure, where
over provisioning and reactive network updates are inex-
pensive. In contrast, SkyCore operates in an unreliable and
resource-constrained wireless environment, where such ap-
proaches scale poorly.
SDN control and data planes.
There is a rich literature
in distributed SDN control planes designs with hierarchical
and at structures (e.g., ONOS [
19
], [
24
,
29
,
39
,
50
]). Most of
the schemes are designed for DC networks and operate based
on a centralized data store or complex consensus algorithms,
which are ill-suited for our unreliable multi-UAV environ-
ment. There are some memory-ecient switch designs for
SDN [
33
,
37
,
49
] that can be leveraged to improve packet
forwarding between SkyCore agents.
RAN optimization for LTE UAVs.
DroneNet [
22
] ex-
tends the coverage of existing LTE cells by creating WiFi
on-drone hotspots. Some recent works [
31
,
32
,
48
] investigate
the theoretical optimization of a UAV trajectory for certain
mobile users on the ground (e.g., maximize the minimum
average rate among all user). These RAN eorts are predomi-
nantly for a single UAV and complementary to SkyCore that
focuses on the EPC design for multi-UAV LTE networks.
9 DISCUSSIONS AND REMARKS
We presented the design and implementation of a novel
Edge-EPC architecture- SkyCore, supporting the untethered
and reliable operation of multi-UAV LTE networks. Several
aspects of SkyCore are worthy of further investigation.
Relevance to high-altitude platforms:
While today’s
high-altitude UAV networks [
6
,
8
] might aim to provide In-
ternet connectivity through a legacy EPC on the ground, we
believe the benets of a SkyCore design for an edge EPC
signicantly outweigh its drawbacks and apply to such de-
ployments as well.
Impact of wireless inter-UAV backhaul design.
We
did not discuss the design of the backhaul agents forming
the physical wireless mesh network among UAVs. The design
of an ecient backhaul needs to be jointly optimized with
the RAN as the position of the UAV simultaneously aects
the performance of the backhaul as well as the access to UEs.
Scalability implication for large UAV-based LTE net-
works.
SkyCore’s design focuses on the challenges unique
to multi-UAV LTE networks that typically span from a few to
at most tens of UAVs (city-scale). While our design decisions
(e.g., inter-agent proactive updates, policy precomputation)
are ecient and scalable for our target environment, they
are not designed to scale in nation-wide LTE networks with
hundreds of millions of UEs.
Applicability to terrestrial 5G networks:
While EPC-
RAN communication is often reliable in terrestrial networks,
operators can leverage Edge-EPC designs (e.g.,SkyCore) to
move the EPC functionality to their edge clouds or cell tow-
ers and realize ultra-low latency required by many 5G use
cases. Such deployments are motivated by operators’ push
towards mobile edge computing (MEC) [
21
,
41
] and their
eorts in deploying white box switches at cell towers [2].
ACKNOWLEDGMENT
We would like to thank the reviewers, our shepherd, Ayon
Chakraborty and Amir Khojastepour at NEC Labs America,
and Harsha V. Madhyastha and Mosharaf Chowdhury at the
University of Michigan for providing valuable feedback on
our work. This work was conducted when Mehrdad Moradi
was an intern at and funded by NEC Labs America.
REFERENCES
[1]
Amazon patents fulllment center towers to increase drone delivery
eciency. https://goo.gl/x66KXF.
[2]
AT&T is deploying white box hardware in cell towers to power mobile
5G era. https://goo.gl/snRW6M.
[3]
AT&T prepares for hurricane season with disaster response exercise.
https://goo.gl/8zLvLn.
[4] CBRS Spectrum. https://goo.gl/3zbYyo.
[5] Drones can provide cell phone service . https://goo.gl/GmR93Y.
[6] Facebook Project Aquila. https://goo.gl/gHYVa7.
[7] Flying COW connects Puerto Rico. https://goo.gl/NEq1HA.
[8] Google X: Project Loon. https://goo.gl/skSz1z.
[9] Lagopus: SDN switch. http://www.lagopus.org/.
[10] LTE signaling storm. https://goo.gl/ZuwYfe.
[11] M-Cord: Mobile Cord. http://goo.gl/pZgSvg.
[12] OpenEPC. http://www.openepc.com/.
[13]
Oracle communications LTE diameter signaling index. https://goo.gl/
6BZ8Fo.
[14] Verizon Continues tests using drones. https://goo.gl/ba2Wz8.
[15] Verizon trials drones as ying cell towers. https://goo.gl/q9YjNv.
[16]
When COWs y: AT&T sending LTE signals from drones. https:
//goo.gl/9u33qC.
[17]
B. Aoun, R. Boutaba, Y. Iraqi, and G. Kenward. Gateway placement
optimization in wireless mesh networks with QoS constraints. IEEE
JSAC, 2006.
[18]
A. Banerjee, R. Mahindra, K. Sundaresan, S. Kasera, K. Van der Merwe,
and S. Rangarajan. Scaling the lte control-plane for future mobile
access. In Proc. ACM CoNEXT, 2015.
[19]
P. Berde, M. Gerola, J. Hart, Y. Higuchi, M. Kobayashi, et al. ONOS:
towards an open, distributed sdn os. In Proc. ACM SIGCOMM Workshop
on HotSDN, 2014.
[20]
P. Bosshart et al. P4: Programming protocol-independent packet pro-
cessors. ACM CCR, 2014.
[21]
J. Cho et al. ACACIA: Context-aware edge computing for continuous
interactive applications over mobile networks. In Proc. ACM CoNEXT,
2016.
[22]
A. Dhekne et al. Extending cell tower coverage through drones. In
Proc. ACM HotMobile, 2017.
[23]
B. Han, V. Gopalakrishnan, et al. Network function virtualization:
Challenges and opportunities for innovations. IEEE Communications
Magazine, 2015.
[24]
S. Hassas Yeganeh and Y. Ganjali. Kandoo: a framework for ecient
and scalable ooading of control applications. In Proc. ACM SIGCOMM
Workshop on HotSDN, 2012.
[25]
W. R. Heinzelman, A. Chandrakasan, and H. Balakrishnan. Energy-
ecient communication protocol for wireless microsensor networks.
In Proc. IEEE HICSS, 2000.
[26]
C.-Y. Hong, S. Kandula, R. Mahajan, et al. Achieving high utilization
with software-driven WAN. ACM CCR, 2013.
[27]
S. Jain, A. Kumar, S. Mandal, et al. B4: Experience with a globally-
deployed software dened WAN. ACM CCR, 2013.
[28]
X. Jin, L. E. Li, et al. Softcell: Scalable and exible cellular core network
architecture. In Proc. ACM CoNEXT, 2013.
[29]
T. Koponen, M. Casado, N. Gude, et al. Onix: A distributed control
platform for large-scale production networks. In Proc. USENIX OSDI,
2010.
[30]
Y. Li, Z. Yuan, and C. Peng. A control-plane perspective on reducing
data access latency in lte networks. In Proc. ACM MobiCom, 2017.
[31]
X. Lin, R. Wiren, S. Euler, A. Sadam, et al. Mobile networks connected
drones: Field trials, simulations, and design insights. arXiv preprint
arXiv:1801.10508, 2018.
[32]
X. Lin, V. Yajnanarayana, S. D. Muruganathan, et al. The sky is
not the limit: LTE for unmanned aerial vehicles. arXiv preprint
arXiv:1707.07534, 2017.
[33]
Y. Lin, U. C. Kozat, J. Kaippallimalil, M. Moradi, A. C. Soong, and
Z. M. Mao. Pausing and resuming network ows using programmable
buers. In Proc. ACM SOSR, 2018.
[34]
M. Moradi, L. E. Li, and Z. M. Mao. SoftMoW: a dynamic and scal-
able software dened architecture for cellular wans. In Proc. ACM
SIGCOMM Workshop on HotSDN, 2014.
[35]
M. Moradi, L. E. Li, and Z. M. Mao. Softmow: A dynamic and scalable
software dened architecture for cellular WANs. Presented as part of
the Open Networking Summit (ONS), 2014.
[36]
M. Moradi, Y. Lin, Z. M. Mao, S. Sen, and O. Spatscheck. Softbox: A
customizable, low-latency, and scalable 5G core network architecture.
IEEE JSAC, 2018.
[37]
M. Moradi, F. Qian, Q. Xu, Z. M. Mao, D. Bethea, and M. K. Reiter.
Caesar: High-speed and memory-ecient forwarding engine for future
internet architecture. In Proc. ACM/IEEE ANCS, 2015.
[38]
M. Moradi, W. Wu, L. E. Li, and Z. M. Mao. SoftMoW: Recursive and
recongurable cellular wan architecture. In Proc. ACM CoNEXT, 2014.
[39]
M. Moradi, Y. Zhang, Z. M. Mao, and R. Manghirmalani. Dragon: A
scalable, exible and ecient trac engineering framework for isp
networks. IEEE JSAC, 2018.
[40]
Nguyen, Binh and Zhang, Tian and Radunovic, Bozidar and others. A
reliable distributed cellular core network for hyper-scale public clouds.
MSR Technical Report, 2018.
[41]
M. Patel et al. Mobile-edge computing introductory technical white
paper. White Paper, Mobile-edge Computing (MEC) industry initiative,
2014.
[42]
B. Pfa, J. Pettit, T. Koponen, et al. The design and implementation of
open vswitch. In Proc. USENIX NSDI, 2015.
[43]
Z. A. Qazi et al. Klein: A minimally disruptive design for an elastic
cellular core. In Proc. ACM SOSR, 2016.
[44]
Z. A. Qazi, M. Walls, A. Panda, et al. A high performance packet core
for next generation cellular networks. In Proc. ACM SIGCOMM, 2017.
[45]
A. S. Rajan et al. Understanding the bottlenecks in virtualizing cellular
core network functions. In Proc. IEEE LANMAN, 2015.
[46]
I. Stoica, R. Morris, D. Liben-Nowell, D. R. Karger, M. F. Kaashoek,
F. Dabek, and H. Balakrishnan. Chord: a scalable peer-to-peer lookup
protocol for internet applications. IEEE/ACM TON, 2003.
[47]
A. Syed and J. Van der Merwe. Proteus: a network service control plat-
form for service evolution in a mobile software dened infrastructure.
In Proc. ACM MobiCom, 2016.
[48]
Q. Wu, Y. Zeng, and R. Zhang. Joint trajectory and communication
design for multi-UAV enabled wireless networks. IEEE/ACM TON,
2018.
[49]
Y. Yu, D. Belazzougui, C. Qian, and Q. Zhang. Memory-ecient
and ultra-fast network lookup and forwarding using othello hash-
ing. IEEE/ACM TON, 2018.
[50]
Y. Zhang and M. Moradi. Sdn based interdomain and intradomain
trac engineering, July 4 2017. US Patent 9,699,116.
... Researchers proposed various SDN-enabled A-T network architectures for mobile and WiFi connectivity applications [190]- [192], [197], [198], [200]. The authors of [191] consider the scenario that UAVs could serve as access points or handover links for ground UEs moving between macro cells. ...
... They propose an SDN-UAV architecture to deploy shifting policies and network management, where the macro BS on the ground acts as a controller and UAVs operate as SDN switches. In [192], the proposed architecture SkyCore relocates the BS's Evolved Packet Core (EPC) entity to run on the UAVs. The EPC functionalities are defined as lightweight SDN applications to eliminate distributed interfaces and reduce function complexity. ...
... Design an SDN-enabled architecture using network state prediction based on UAVs physical position and trajectory [191] Ground station Mobile networks Mininet, OpenFlow Employ SDN to solve the problem of UEs handover in UAV-assisted mobile network [192] UAVs LTE mobile networks P4 w/OpenFlow, Lagopus, Open vSwitch ...
Article
Full-text available
6G networks are envisioned to deliver a large diversity of applications and meet stringent Quality of Service (QoS) requirements. Hence, integrated Terrestrial and Non-Terrestrial Networks (TN-NTNs) are anticipated to be key enabling technologies. However, the integration of TN-NTNs faces a number of challenges that could be addressed through network virtualization technologies, such as Software-Defined Networking (SDN), Network Function Virtualization (NFV), and Network Slicing (NS). In this survey, we provide a comprehensive review of the adaptation of these networking paradigms in 6G networks. We begin with a brief overview of Non-Terrestrial Networks (NTNs) and virtualization techniques. Then, we highlight the integral role of Artificial Intelligence (AI) in improving network virtualization by summarizing major research areas where AI models are applied. Building on this foundation, we identify the main issues arising from the use of SDN, NFV, and NS in integrated TN-NTNs, and propose a taxonomy of integrated TN-NTNs virtualization offering a thorough review of relevant contributions. The taxonomy is built on a four-level classification that indicates — for each study — the level of TN-NTNs integration, the virtualization technology used, the problem addressed, the type of the study, and the proposed solution, which can be based on conventional or AI-enabled methods. Finally, we discuss open issues and give insights on future research directions for the advancement of integrated TN-NTNs virtualization in the 6G era.
... More relevant to dynamic networks are works [51,16,34,55] that explore enhancing UAV networks for flexibility and adaptability, while also providing LTE support. Specifically, SkyCore offers an Edge-EPC solution tailored for multi-UAV LTE networks, bypassing traditional EPC constraints in UAV settings. ...
Preprint
Full-text available
In this work, we present Dyna-5G, a dynamic, self-organizing 5G New Radio (5G-NR) network designed for massive Machine-to-Machine (M2M) networks. Traditional 5G NR networks, characterized by their centralized architecture, face challenges in supporting applications that require dynamic, decentralized communication, such as autonomous vehicles and drone swarms for emergency responses. These scenarios often suffer from the centralized model's single point of failure, undermining the reliability required in critical and fully autonomous applications. Dyna-5G addresses these challenges by allowing each device in the network to function as either part of the Radio Access Network (RAN) and Core Network, or as User Equipment (UE), thus maintaining network functionality even when conventional infrastructure components are compromised. Dyna-5G has built-in mechanisms carefully designed specifically for M2M networks, such as failure-recovery and ad-hoc entry and exit. We demonstrate the performance and feasibility of Dyna-5G using a custom-built testbed that simulates real-world missions, demonstrating our network's robustness, adaptability, and failure recovery capabilities. The results indicate that our entire 5G network model can fully re-organize in 6 seconds at maximum, without compromising the mission.
... Despite the benefits of cellular networks, the literature has highlighted various network architectures that did not achieve optimal results in this integrated technology. To tackle these challenges, Moradi et al. [80] proposed a SkyCore-based framework called the Evolved Packet Core (EPC) scheme. In this scheme, the authors addressed the network architectural issues in the context of inter-networking of IoTs, UAVs, and cellular networks, but they did not manage the computation complexities effectively while integrating these technologies, which as a whole affected the communication process of these networks in terms of latency and throughput in the real deployment. ...
Article
Unmanned Aerial Vehicle (UAV)-assisted Internet of Things application communication is an emerging concept that effectuate the foreknowledge of innovative technologies. With the accelerated advancements in IoT applications, the importance of this technology became more impactful and persistent. Moreover, this technology have demonstrated useful contributions across various domains, ranging from general to specific applications. Examples include wildfire monitoring, coastal area monitoring, deforestation monitoring, and sensitive military operations, where human access is limited or not feasible. These examples underscore the technology’s importance in scenarios where direct human involvement is challenging or impossible. Although this technology offers numerous benefits, it is essential to note that it also faces several challenges. Among these, Quality of Service (QoS) is a key concern, which limits its useability in various applications. Unfortunately, most researchers in the present literature have overlooked this important factor without giving it considerable attention. To fill this gap, we are presenting a systematic review of the present literature associated with the QoS metrics of this emerging technology from 2015 to 2023 to highlight their contributions and limitations. Based on the systematic review, we highlight the open challenges of this technology to set a roadmap for futuristic research. Finally, we compared each portion of this work with the previously published review articles to confirm the essence of this work, along with an explanation of why this survey is needed and in-time.
Article
As appropriate deployment of unmanned aerial vehicles (UAVs) in UAV-assisted wireless networks is critical for the next-generation wireless networks, we in this paper propose centralized and decentralized UAV deployment approaches that can be applied to any UAV-assisted wireless networks for any performance metrics. The proposed centralized deployment combines the deep neural network (DNN)-based surrogate model with the zeroth-order optimization (ZOO) such that the deployment can be optimized via using the predicted network performance of the surrogate model. Since the accurate prediction of the DNN surrogate model is critical, we discuss its design and update approaches. To let UAVs update their locations for better network performance by exchanging local information with neighboring UAVs, the proposed decentralized deployment approaches combine distributed optimization frameworks with ZOO and DNN surrogate model. We conduct realistic simulations in different network scenarios with different performance metrics to evaluate our proposed approaches. Results show that our proposed can outperform all the reference schemes in all scenarios considering different performance metrics.
Article
In this paper we study the deployment of an Unmanned Aerial Vehicle (UAV) network that consists of multiple UAVs to provide emergent communication service for people who are trapped in a disaster area, where each UAV is equipped with a base station that has limited computing capacity and power supply, and thus can only serve a limited number of people. Unlike most existing studies that focused on homogeneous UAVs, we consider the deployment of heterogeneous UAVs where different UAVs have different computing capacities. We study a problem of deploying K heterogeneous UAVs in the air to form a temporarily connected UAV network such that the network throughput – the number of users served by the UAVs, is maximized, subject to the constraint that the number of people served by each UAV is no greater than its service capacity. We then propose a novel O(sK)O(\sqrt{\frac{s}{K}}) -approximation algorithm for the problem, where s is a given positive integer with 1sK1 \le s\le K , e.g., s=3 . We also devise an improved heuristic, based on the approximation algorithm. We finally evaluate the performance of the proposed algorithms. Experimental results show that the numbers of users served by UAVs in the solutions delivered by the proposed algorithms are increased by 25% than state-of-the-arts.
Article
Full-text available
Multi-Agent Reinforcement Learning (MARL) is widely used to solve various problems in real life. In the multi-agent reinforcement learning tasks, there are multiple agents in the environment, the existing Proximal Policy Optimization (PPO) algorithm can be applied to multi-agent reinforcement learning. However, it cannot deal with the communication problem between agents. In order to resolve this issue, we propose a Graph-based PPO algorithm, this approach can solve the communication problem between agents and it can enhance the exploration efficiency of agents in the environment and speed up the learning process. We apply our algorithms to the task of multi-UAV navigation for communication coverage to verify the functionality and performance of our proposed algorithms.
Article
Full-text available
To optimize network cost, performance, and reliabil- ity, SDN advocates for centralized traffic engineering (TE) that enables more efficient path and egress point selection as well as bandwidth allocation. In this work, we argue SDN-based TE for ISP networks can be very challenging. First, ISP networks often are very large in size, imposing significant scalability challenges to the centralized TE. Second, ISP networks usually have diverse types of links, switches, and cost models, leading to a complex combination of optimizations. Third, ISP networks not only have many choices of internal paths but also include rich selections of egress points and interdomain routes, unlike cloud/enterprise net- works. To overcome these challenges, we present a novel TE appli- cation framework, called Dragon, for existing SDN control planes. To address the scalability challenge, Dragon consists of hierarchi- cal and recursive TE algorithms and mechanisms that divide flow optimization problems into subtasks and execute them in parallel. Further, Dragon allows ISPs to express diverse objectives for different parts of their network. Finally, we extend Dragon to jointly optimize selection of intradomain and interdomain paths. Using extensive evaluation on real topologies and prototyping with SDN controller and switches, we demonstrate that Dragon outperforms existing TE methods both in speed and in optimality.
Conference Paper
Full-text available
The emerging 5G networks and applications require network traffic to be buffered at different points in a wide area network with different policies based on user mobility, usage patterns, device and application types. Existing Software Defined Network (SDN) solutions fail to provide sufficient decoupling between the data-plane and control-plane to enable efficient control of where, when, and how a network flow is buffered without causing excessive control-plane traffic. Alternative approaches either cater to application-specific use cases or follow completely new paradigms that are not compatible with 5G evolution. In this paper, we present (1) a new programming abstraction for network buffering, (2) a new set of northbound APIs for network applications to pause and resume traffic flows in the network, (3) a new set of southbound APIs to scalably control where, when, and how a network flow is buffered, and (4) evaluations based on the prototype implementation of data-plane and control-plane functions. Our evaluations indicate significant performance and scalability potentials, and show near-optimal results in mobility-management and connectionless communication use cases.
Article
Full-text available
We propose a novel cellular core network architecture, SoftBox, combining software-defined networking (SDN) and network function virtualization (NFV) to achieve greater flexibility, efficiency, and scalability compared to today’s cellular core. Aligned with 5G use cases, SoftBox enables the creation of customized, low latency, and signaling-efficient services on a per user equipment (UE) basis. SoftBox consolidates network policies needed for processing each UE’s data and signaling traffic into a light-weight, in-network, per-UE agent. We design a number of mobility-aware techniques to further optimize: (i) resource usage of agents, (ii) forwarding rules and updates needed for steering a UE’s traffic through its agent, (ii) migration costs of agents needed to ensure their proximity to mobile UEs, and (iv) complexity of distributing the LTE mobility function on agents. Extensive evaluations demonstrate the scalability, performance, and flexibility of the SoftBox design. For example, basic SoftBox has by 86%, 51%, and 87% lower signaling overheads, data plane delay, and CPU core usage, respectively, than two open source EPC systems. Moreover, our optimizations efficiently cut different types of data and control plane loads in the basic SoftBox by 51%-98%.
Conference Paper
Full-text available
Control-plane operations are indispensable to providing data access to mobile devices in the 4G LTE networks. They provision necessary control states at the device and network nodes to enable data access. However, the current design may suffer from long data access latency even under good radio conditions. The fundamental problem is that, data-plane packet delivery cannot start or resume until all control-plane procedures are completed, and these control procedures run sequentially by design. We show both are more than necessary under popular use cases. We design DPCM, which reduces data access latency through parallel processing approaches and exploiting device-side state replica. We implement DPCM and validate its effectiveness with extensive evaluations.
Article
We present SWAN, a system that boosts the utilization of inter-datacenter networks by centrally controlling when and how much traffic each service sends and frequently re-configuring the network's data plane to match current traffic demand. But done simplistically, these re-configurations can also cause severe, transient congestion because different switches may apply updates at different times. We develop a novel technique that leverages a small amount of scratch capacity on links to apply updates in a provably congestion-free manner, without making any assumptions about the order and timing of updates at individual switches. Further, to scale to large networks in the face of limited forwarding table capacity, SWAN greedily selects a small set of entries that can best satisfy current demand. It updates this set without disrupting traffic by leveraging a small amount of scratch capacity in forwarding tables. Experiments using a testbed prototype and data-driven simulations of two production networks show that SWAN carries 60% more traffic than the current practice.
Conference Paper
Economies of scale associated with hyper-scale public cloud platforms offer flexibility and cost-effectiveness, resulting in various services and businesses moving to the cloud. One area with little progress in this direction is cellular core networks. A cellular core network manages the state of cellular clients; it is essentially a large distributed state machine with very different virtualization challenges compared to typical cloud services. In this paper we present a novel cellular core network architecture, called ECHO, particularly suited to public cloud deployments, where the availability guarantees might be an order of magnitude worse compared to existing (redundant) hardware platforms. We present the design and implementation of our approach and evaluate its functionality on a public cloud platform. Analysis shows ECHO promises higher availability than existing telco solutions.
Article
Network algorithms always prefer low memory cost and fast packet processing speed. Forwarding information base (FIB), as a typical network processing component, requires a scalable and memory-efficient algorithm to support fast lookups. In this paper, we present a new network algorithm, Othello hashing, and its application of a FIB design called concise, which uses very little memory to support ultra-fast lookups of network names. Othello hashing and concise make use of minimal perfect hashing and relies on the programmable network framework to support dynamic updates. Our conceptual contribution of concise is to optimize the memory efficiency and query speed in the data plane and move the relatively complex construction and update components to the resource-rich control plane. We implemented concise on three platforms. Experimental results show that concise uses significantly smaller memory to achieve much faster query speed compared to existing solutions of network name lookups.
Conference Paper
Cellular traffic continues to grow rapidly making the scalability of the cellular infrastructure a critical issue. However, there is mounting evidence that the current Evolved Packet Core (EPC) is ill-suited to meet these scaling demands: EPC solutions based on specialized appliances are expensive to scale and recent software EPCs perform poorly, particularly with increasing numbers of devices or signaling traffic. In this paper, we design and evaluate a new system architecture for a software EPC that achieves high and scalable performance. We postulate that the poor scaling of existing EPC systems stems from the manner in which the system is decomposed which leads to device state being duplicated across multiple components which in turn results in frequent interactions between the different components. We propose an alternate approach in which state for a single device is consolidated in one location and EPC functions are (re)organized for efficient access to this consolidated state. In effect, our design "slices" the EPC by user. We prototype and evaluate PEPC, a software EPC that implements the key components of our design. We show that PEPC achieves 3-7x higher throughput than comparable software EPCs that have been implemented in industry and over 10x higher throughput than a popular open-source implementation (OpenAirInterface). Compared to the industrial EPC implementations, PEPC sustains high data throughput for 10-100x more users devices per core, and a 10x higher ratio of signaling-to-data traffic. In addition to high performance, PEPC's by-user organization enables efficient state migration and customization of processing pipelines. We implement user migration in PEPC and show that state can be migrated with little disruption, e.g., migration adds only up to 4μs of latency to median per packet latencies.
Article
Many use cases of unmanned aerial vehicles (UAVs) require beyond visual line-of-sight (LOS) communications. Mobile networks offer wide area, high speed, and secure wireless connectivity, which can enhance control and safety of UAV operations and enable beyond visual LOS use cases. In this article, we share some of our experience in Long-Term Evolution (LTE) connectivity for low altitude small UAVs. We first identify the typical airborne connectivity requirements and characteristics, highlight the different propagation conditions for UAVs and mobiles on the ground with measurement and ray tracing results, and present simulation results to shed light on the feasibility of providing LTE connectivity for UAVs. We also present several ideas on potential enhancements for improving LTE connectivity performance and identify fruitful avenues for future research.