ArticlePDF Available

Abstract and Figures

The novel network slicing paradigm represents an effective turning point to operate future wireless networks. Available networking and computational resources may be shared across different (instantiations of) services tailored onto specific vertical needs, envisioned as the main infrastructure tenants. While such customization enables meeting advanced key performance indicators (KPIs) introduced by upcoming 5G networks, advanced multi-tenancy approaches help to abate the cost of deploying and operating the network. However, the network slicing implementation requires a number of non-trivial practical considerations, including how resource sharing operations are actually implemented, how involved parties establish the corresponding agreement to instantiate, operate, and terminate such a sharing or the design of functional modules and interfaces supporting these operations. In this paper, we present a novel framework that unveils proper answers to the above design challenges. While existing initiatives are typically limited to single-domain and single-owner scenarios, our framework overcomes these limitations by enlarging the administrative scope of the network deployments fostering different providers to collaborate so as to facilitate a larger set of resources even spread across multiple domains. Numerical evaluations confirm the effectiveness and efficiency of the presented solution.
Content may be subject to copyright.
Received May 17, 2019, accepted June 12, 2019, date of publication June 17, 2019, date of current version July 1, 2019.
Digital Object Identifier 10.1109/ACCESS.2019.2923364
A Future-Proof Architecture for Management
and Orchestration of Multi-Domain
NextGen Networks
VINCENZO SCIANCALEPORE 1, (Member, IEEE), CHRISTIAN MANNWEILER 2,
FAQIR ZARRAR YOUSAF 1, PABLO SERRANO 3, (Senior Member, IEEE),
MARCO GRAMAGLIA 3, JULIE BRADFORD4, AND IGNACIO LABRADOR PAVÓN5
1NEC Laboratories Europe GmbH, 69115 Heidelberg, Germany
2Nokia Bell Labs, 81541 Munich, Germany
3Department of Telematics Engineering, University Carlos III of Madrid, 28911 Leganés, Spain
4Real Wireless Team, London RH20 4XB, U.K.
5ATOS, 28037 Madrid, Spain
Corresponding author: Marco Gramaglia (mgramagl@ it.uc3m.es)
This work was supported in part by the 5G-MoNArch Project, in part by the Phase II of the 5th Generation Public Private Partnership
(5G-PPP) Program, in part by the European Commission within the Horizon 2020 Framework Program under Grant 761445, in part by the
5G-MoNArch Project builds on the results of the 5G-PPP Phase I Project 5G-NORMA, and in part by the European Union Horizon
2020 Project 5G-CARMEN under Grant 825012. The work of UC3M has also received funding from the Horizon 2020 Programme under
Grant 815074 - 5G EVE.
ABSTRACT The novel network slicing paradigm represents an effective turning point to operate future
wireless networks. Available networking and computational resources may be shared across different
(instantiations of) services tailored onto specific vertical needs, envisioned as the main infrastructure tenants.
While such customization enables meeting advanced key performance indicators (KPIs) introduced by
upcoming 5G networks, advanced multi-tenancy approaches help to abate the cost of deploying and operating
the network. However, the network slicing implementation requires a number of non-trivial practical
considerations, including how resource sharing operations are actually implemented, how involved parties
establish the corresponding agreement to instantiate, operate, and terminate such a sharing or the design of
functional modules and interfaces supporting these operations. In this paper, we present a novel framework
that unveils proper answers to the above design challenges. While existing initiatives are typically limited
to single-domain and single-owner scenarios, our framework overcomes these limitations by enlarging
the administrative scope of the network deployments fostering different providers to collaborate so as to
facilitate a larger set of resources even spread across multiple domains. Numerical evaluations confirm the
effectiveness and efficiency of the presented solution.
INDEX TERMS 5G mobile communication, computer network management, network architecture, network
function virtualization.
I. INTRODUCTION
The fifth generation (5G) of wireless and mobile communi-
cation networks needs to natively support multiple advanced
services. This requirement significantly deviates from the
legacy approach, which is built on the one-fits-all paradigm:
the same telecommunication services (both voice and data)
are provided to any kind of customer, regardless of the mobile
applications that are actually carried over the network. This
The associate editor coordinating the review of this manuscript and
approving it for publication was Moayad Aloqaily.
monolithic view is no longer sufficient when dealing with a
large ecosystem of use cases as the one currently envisioned
for 5G (that could be extended with novel services).
5G networks should fulfill a set of requirements that are
identified by Key Performance Indicators (KPIs) such as,
e.g., high data rates, extremely low latency, robustness, and
reliability. To efficiently address such a stringent set of KPIs,
a key-technology has been proposed, namely network slicing.
A network slice is a set of physical and logical resources
(e.g., radio, transport and cloud/edge resources) gathered
from a common pool and assigned to an infrastructure tenant
79216 This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/ VOLUME 7, 2019
V. Sciancalepore et al.: Future-Proof Architecture for Management and Orchestration of Multi-Domain NextGen Networks
FIGURE 1. U.K. data, cost, and revenue trends [9]. (a) U.K. monthly mobile data volume and revenue per subscriber. (b) U.K. MNO revenue
per GB trends.
to deliver some specific services [1]. Such services could
be, e.g., enhanced/extreme Mobile Broadband (e/xMBB) for
‘‘regular’’ end-users, or high-reliable and ultra-low latency
services for autonomous driving [2]. The potential advan-
tages brought by network slicing are evident [3], given its
ability to provide highly customizable services over the
same shared infrastructure and, in turn, creating new revenue
opportunities for the involved stakeholders.
This ability to customize services is attracting a lot of
attention from researchers (both academic and industrial).
In particular, one key challenge is the design of mechanisms
to efficiently chain different network functions at different
locations while fulfilling the target KPIs [4], [5]. But there
is (at least) another key aspect that has been overlooked
so far: the analysis of diverse agreements among multiple
stakeholders that need to collaborate to bring network slicing
into practice. Some of these stakeholders are network opera-
tors (from one or multiple domains), infrastructure providers,
service providers, and tenants, which shall agree in terms of
control and management of the resources involved to instan-
tiate a service.
In this paper, we present a novel network management and
orchestration architecture that fully supports the customiza-
tion of network slices across services and tenants. While
our framework is backward compatible with current stan-
dardization efforts (namely, ETSI NFV MANO [6]), it fur-
ther introduces a number of key features that enable the
development of a series of use cases of economic relevance,
as we discuss next. These main features our architecture
are:
A formal definition of the different actors of the ecosys-
tem, along with the various roles they can play as well as
different agreements that can be reached to support and
deliver intended services.
Efficient support for multi-domain environments.
Instead of complex peer-to-peer interactions, our archi-
tecture enables clean coordination between systems,
thus providing a versatile framework to e.g. deploy new
services over multiple domains, or extend traditional
services using the novel infrastructure.
Design of a novel element, namely Inter-Slice Resource
Broker, to enable an efficient sharing of resources across
slices while guaranteeing the reservation of resources as
specified in the Service Agreements across actors.
The remainder of the article is organized as follows.
Section II motivates the need for network slicing from an
economic standpoint. Section III formally introduces the dif-
ferent actors and roles of the ecosystem while illustrating the
flexibility of the envisioned solution via two representative
and timely use cases. Section IV provides a brief overview
of the current solution for network slicing and identifies its
limitations for the case of multi-domain scenario. Section V
presents the proposed architecture, which is composed of
both novel modules and extensions over the ETSI NFV
MANO framework thereby describing the lifecycle of a net-
work slice. Section VI empirically assesses the improvements
introduced by our proposal, and Section VIII provides con-
cluding remarks.
II. ECONOMIC MOTIVATION FOR NETWORK SLICING
As discussed before, there are clear technical advantages that
network slicing will bring into next generation networks.
Here we discuss both qualitative and quantitatively another
dimension in which network slicing can provide significant
benefits, namely, the economic axis. Indeed, network slicing
can contribute to addressing some of the revenue/cost issues
that current network deployments face, which are discussed
next.
We first summarize the main economic motivations for
the development of the proposed architecture and associated
technologies. Fig. 1presents an analysis of historical mobile
data volumes and revenues for the UK (based on [7]). The
figure shows a slight downward trend in prices paid by mobile
subscribers, driven by competition in the UK mobile market
and limits on consumers’ willingness to pay. This is despite
subscribers receiving much higher data volumes through their
subscriptions over time, and has created a steep downward
trend in the mobile revenue received per Gigabyte (GB)
of data delivered. Alongside this, subscriber volumes have
grown only marginally with headline retail mobile revenues
in the UK falling by 8%, i.e., from GBP 17 billion to GBP
15.6 billion between 2012 and 2017.
This difficulty in growing or maintaining revenues
from consumer-focused services puts pressure on reducing
VOLUME 7, 2019 79217
V. Sciancalepore et al.: Future-Proof Architecture for Management and Orchestration of Multi-Domain NextGen Networks
TABLE 1. V2I cost and revenue results combined (2020–2030), middle
scenario, £million, and central London.
TABLE 2. V2I cumulative discounted cash flow, middle scenario, £ million,
and central london.
network costs to maintain margins. However, with increas-
ing volumes of data being consumed on mobile networks,
the opportunity to reduce costs has been limited. In fact,
according to GSMA [8], for the period 2010–2017 the Capital
Expenditure (CAPEX) investment from European MNOs has
been at a rate between 12% and 18% of revenues.
The adoption of the architecture proposed in this paper
has the potential to both improve margins and de-risk the
long-term business case for providers:
Increasing the revenue per GB, by developing rela-
tionships with end users and tenants who place a higher
value than regular consumers on tailored mobile services
with a guaranteed quality of service level.
Reducing the cost per GB, compared with many dis-
parate and dedicated private networks, due to economies
of scale and scope of combining multiple services on the
same network.
The two above effects have been investigated in [9], where
we consider a multi-service virtualized network delivering
a range of services in a central London study area. This
analysis considers initially the costs and revenues associated
with a ‘‘business as usual’’ scenario over a 2020 to 2030 time
period. The network costs are based on making use of existing
sites in the area (representative of a typical existing MNO),
and then either (i) adding antennas and/or frequency bands
to these, or (ii) choosing to build new sites, depending on
which approach is most cost-effective to serve the increas-
ing. Tables 1and 2present an extract of the reported rev-
enues, costs and resulting discounted cash flow for the years
2020, 2025 and 2030 with the accumulated total over the
2020-2030 time period given in the final column. Compar-
ing the total income against costs for the entire 2020 to
2030 period gives the Net Present Value (NPV), which is
reported in Table 3and can subsequently be translated into
a Return on Investment (ROI) for the period.
TABLE 3. V2I summary of financial measures, middle scenario, £ million,
and central london.
The analysis in [9] also aims at understanding the busi-
ness impact of delivering more bespoke services alongside
existing consumer mobile broadband services, using the same
network infrastructure set (via network slicing). To this aim,
the analysis considers the provision of vehicle to infras-
tructure (V2I) services in combination with eMBB services
over a virtualized multi-service 5G network. This analysis
includes:
Deriving revenue forecasts for the V2I services consid-
ered, based on the benefits derived by end users and
hence their willingness to pay. For example, the impact
on insurance premiums were considered for some of the
V2I services in this context.
Deriving the additional network CAPEX and opera-
tional expenditure (OPEX) beyond the planned ‘‘busi-
ness as usual’’ eMBB network, to accommodate the
envisaged V2I services in terms of their capacity and
coverage requirements. This included developing ser-
vice definitions and demand forecasts for the three
candidate V2I services considered: (i) infotainment,
(ii) assisted driving (non-critical traffic, navigation,
maintenance etc. information updates to drivers), and
(iii) semi-automated driving (critical real time updates
on upcoming hazards).
The resulting impact on revenue and costs of offering one
of the three V2I services considered in combination with the
eMBB baseline case is shown on Table 1, with the resulting
discounted cash flows on Table 2. Finally, the impact on total
revenues against costs over the 2020 to 2030 is reported in
the NPV and ROI values given in Table 3. These show up to
a 10% improvement in ROI compared to the baseline eMBB
case. While only a limited set of services were considered
in this example, this is indicative that extending existing
mobile networks to deliver a wider range of more bespoke
mobile services via network slicing could help to improve
the long-term business case challenges of reducing margins
that the mobile industry currently faces. This is a benefit that
will likely increase with the number of services combined on
the same network. The same study also showed significant
socio-economic benefits from delivering a wider range of ser-
vices than currently, such as smart metering and V2I services,
from mobile networks.
III. ROLE, BUSINESS MODELS, AND USE CASES OF THE
5G ECOSYSTEM
In this section, we describe the potential use cases that are
enabled by the architecture proposed in this paper. First,
we describe the actors in our architecture and their main
interactions.
79218 VOLUME 7, 2019
V. Sciancalepore et al.: Future-Proof Architecture for Management and Orchestration of Multi-Domain NextGen Networks
FIGURE 2. Actors and roles in a network slicing scenario.
A. ACTORS AND ROLES
An actor is an individual or institution playing a role in our
ecosystem. Actors could, therefore, be e.g. a telecommuni-
cations company, a (medium or large scale) HW and/or SW
provider or an enterprise from a vertical industry. The role is
the function assumed by the actor in the corresponding sce-
nario, e.g., providing the infrastructure, operating the network
slice or consuming the service. As we discuss next, an actor
could play different roles.
We illustrate the above concepts in Fig. 2, which represents
a scenario where a network operator is providing different
communication services to a number of tenants via a number
of slices. We use red/italics to identify the different actors,
while black/bold is used to identify the different roles (we
formally introduce the different roles next). The ‘‘telecommu-
nications company’’ acting as ‘‘Network Operator’’ could be,
in the current ecosystem of enterprises, a large telco corpora-
tion (e.g., the one operating at national scale). This operator
would run a number of slices over a heterogeneous infrastruc-
ture of HW and SW elements, which could be provided by
HW/SW vendors. Individuals could play as ‘‘end users’’ (e.g.,
using a mobile phone), but also certain companies could act
as the final consumers of a service (e.g., to monitor a fleet of
cars or robots). Mobile Virtual Network Operators (MVNOs)
or companies from vertical industries could leverage on slices
provided by the network operator to create their services,
where each slice would contain HW/SW infrastructure from
the different available infrastructure providers, depending on
the needs for each one.
Based on the above description, we can identify the follow-
ing roles:
Infrastructure Providers (InPs): They own and man-
age part or all of the network infrastructure (physical
and/or logical). It can be further distinguished between
Radio Access Network (RAN) infrastructure provider
(owning the physical infrastructure such as the antenna
sites and the HW equipment for the antennas), and data-
center/cloud infrastructure provider (managing local and
central datacenters). The former provides access to radio
resources (i.e., spectrum), while the latter provides virtu-
alized resources such as virtualized computing, storage,
FIGURE 3. The high level roles as defined by 3GPP in [10].
and networking, by deploying a virtualization environ-
ment to logically abstract the physical infrastructure.
Network Operators (NOs): They operate the phys-
ical and virtualized network functions and the com-
munications links to realize a mobile network, and
provide mobile connectivity towards mobile end-users.
They also sell dedicated mobile network instances
(network slices as a service), each one realizing spec-
ified telecommunication services such as, e.g., mas-
sive machine-type communication (mMTC), to tenants.
A network operator could lease the needed physical or
logical resources from one or more specialized InPs;
also, a InP could additionally act as a Mobile Service
Provider (MSP) using its own infrastructure, and prob-
ably, leasing certain resources from other specialized
InPs.
Tenants: They are business entities that rent and
leverage a network slice provided by the Network Oper-
ator. They could be MVNO, other enterprises (e.g.
vertical industries), or any other organization requir-
ing a telecommunications service for their business
operations.
End Users: They are the ultimate consumers of the ser-
vice provided by the tenant, which could be individuals
subscribed to a ‘‘classical’’ communication service (e.g.,
voice, video), employees accessing a company-private
VPN service, or sensors requiring some means to trans-
port for their gathered data (e.g. for an IoT scenario).
One important aspect of the actor/role distinction, already
hinted above, is that a specific actor can play different roles.
For instance, the telecommunications company acting as Net-
work Operator can also act as Infrastructure Provider (which
is what usually happens today as well: they provide their
own infrastructure to build-up services, and they also can hire
certain resources to third parties). Also, a vertical industry
company could play the role of Tenant, but also the role
of Infrastructure Provider. Another important feature is that
network slices could be sold to the companies acting as
tenants, but also the telecommunication company acting as
Operator could have its own slices to provide certain services
(as operators do now).
These roles are fully aligned with the ones currently stud-
ied in 3GPP [10], which are depicted in Fig. 3. The utmost
VOLUME 7, 2019 79219
V. Sciancalepore et al.: Future-Proof Architecture for Management and Orchestration of Multi-Domain NextGen Networks
level in the 3GPP management architecture is represented by
Communication Service Customers (CSC), who are enjoying
a service (e.g., industry 4.0) provided by Communication Ser-
vice Providers (CSPs). Therefore, CSCs are totally aligned
with the End Users concept defined above, while Tenants,
instead, perfectly match with the definition of CSP provided
by 3GPP. CSPs organize and structure a communication ser-
vice on top of the Network Operator (NOP) by, for instance,
requesting for a network slice template from the NOP port-
folio (e.g., the Industry 4.0 service may require mMTC and
eMBB slice instances). NOPs, thus, have a direct mapping
with the roles described above (i.e. the Network Operators).
To build the network slice instances they use VNFs from
one or more Network Equipment Providers (NEPs), which
are finally executed in the underlying cloud infrastructure.
In 3GPP terminology, the roles that agglomerate the lower
layers of the architecture are the Virtualization Infrastructure
Service Provider (VISPs), who provide virtualized connec-
tivity by using the hardware from one or more NFVI supplier
(offering e.g., transport network functionality) and the Data
Centre Service Provider (DCSP), who finally offers the bare
metal (acquired from an HW supplier) to run the service
requested by CSC. From the above discussion, it is clear that
our proposed actors and roles setup is fully compliant with
the one standardized by 3GPP.
B. NETWORK SLICE SERVICE P ORTFOLIO
There are different ways, from the managing and control per-
spective, in which a Network Operator can provide a network
slice to a given tenant. These range from the provision of a
‘‘bearer service’’ for data traffic, with no associated control,
to the provision of a set of control primitives that enable
tailoring the composition of the functions to transport this
data. Of course, these offer types have different requirements
in terms of the potential impact on efficiency, the required
interfaces and security considerations [11]. We envision the
following portfolio of slice services:
Offer type N (No control): in this case, the Network
Operator operates the slice and provides communica-
tion services on behalf of the tenant. A tenant requests
the commissioning of a network slice by providing the
high-level requirements of the telecommunication ser-
vice to be provided. Operation of the network slice is
completely handled by the NO, and the tenant might
only receive coarse-grained performance reports. This
allows for a flexible Network Slice market that is ulti-
mately enabled by the algorithms running in a module
dedicated to this aim.
Offer type L (Limited control): here the Network
Operator allows the tenant to have some partial con-
figuration and control options over the slice. Apart
from the coarse-grained specification of the previous
case, such as e.g. bandwidth, in this case, the tenant
can specify more fine-grained configuration options
for the requested network slice. Moreover, selected
network operations (e.g., subscriber data management,
QoS control) are performed by the tenant. Still, the major
part of network operation is handled by the NO, although
the network slice could integrate a set of tenant-owned
functions that are customized (and certified) for its
needs. This interaction of these onboarded functions
with the NO’s systems would be strictly monitored and
controlled by the NO.
Offer type F (Full control): finally, with this case,
the Network Operator allows extended slice configura-
tion and control options for the tenant. In addition to
the control provided in the previous case, in this one
the tenant has rather wide control over the deployed
network functions. This can go as far as, e.g., the tenant
onboarding its own network functions for selected areas
(mobility, session management, etc.), contributing with
its own infrastructure, or operating a part of the network
slice independent of the NO.
The above represents a service portfolio that a Network
Operator could define to provide different types of slices to
tenants. We illustrate the advantages of this flexibility in the
next section, describing two different use cases of relevance
for future networking scenarios.
One major difference between our proposal and the ones
currently defined in the state of the art [12] is the flexibility.
Indeed, [12] envisions the type N operation, but type L and
type F are out of the scope. As a matter of fact, [12] explicitly
states in clause 5.1.1.2 that the Network Operator may create
network slices on top of the Infrastructure Provider resources,
but the management system shall be extended to provide more
or different models, that necessarily rely on novel manage-
ment system such as the one presented here. Analogously,
the proposal in [13] distinguishes between two types of slices,
Internal or External, depending on their intended use. For
the External slices, only two offers are supported, Provider
managed or Tenant managed, which would correspond to
Offers N and F, respectively –no limited control is envisioned.
C. USE CASE: ON-DEMAND VIDEO SLICES
Nowadays, content providers, such as e.g. Youtube or Netflix
use the underlying mobile network as a ‘data pipe’ to the end
user, relying on the standard best effort configuration. This
has several drawbacks, including (i) no KPI is guaranteed,
which could result in a poor service, (ii) the content provider
cannot tune the network configuration, e.g., asking for priori-
tized handling of traffic; and (iii) the NO cannot monetize this
traffic, as it is not offering any added value to the over the top
service (OTT). The possibility of dynamically instantiating
network slices can solve these problems, allowing thus for a
flexible Network Slice Market.
While much mobile content is delivered today as OTT
services based on best effort quality, the provision of content
over mobile networks continues to be an evolving area. Some
mobile operators, in an effort to differentiate themselves, have
offered data plans which include video from particular appli-
cations as so-called ‘‘zero-rated services’’ (i.e., they do not
count towards a user’s data allowance [14]). One well-known
79220 VOLUME 7, 2019
V. Sciancalepore et al.: Future-Proof Architecture for Management and Orchestration of Multi-Domain NextGen Networks
example of this service in the US is T-Mobile’s Binge On.
Furthermore, AT&T and Verizon have also launched unlim-
ited data plans with managed video traffic, which is offered
at different video quality depending on the price of the data
plan. Additionally, venues are looking for ways to differen-
tiate their offered experience, with novel services such as
360 degree or virtual reality content, developed by groups
such as Intel [15]. Other examples of the on-demand mul-
timedia service delivery are vehicular environments [16],
mobile edge computing (MEC) [16] and Smart Cities [18].
Finally, marketing and advertising are also evolving to make
use of mobile technologies, mixed reality, and geo-location
data [19].
The introduction of dynamic network slicing will help to
support quickly trialing these new products, and potentially
drive the need for a flexible Network Slice Market. Following
these lines, we envision the future network slicing landscape
as follows. We assume a scenario in which the InP is a NO,
owning the RAN infrastructure along with datacenters at the
edge, while it leases cloud resources from another InP for the
central cloud. The NO supports deploying customized net-
work slices to sustain dynamic requests from various tenants,
for instance:
Event-tailored: The tenant is an event organizer who
requires a ‘‘video upload’’ slice for a localized area (e.g.
a stadium and its surroundings) during a given period of
time corresponding to the event (e.g. a football match).
The requested service requires a high throughput in
the uplink, a relatively average latency, and support for
relatively little mobility in a localized coverage.
Nationwide delivery-tailored: The tenant requires a
‘‘congestion free’’ video slice for wide (national) cov-
erage, supporting HD video. To fulfill this demand,
the MSP needs to set up a slice providing high through-
put and reliability, and medium latency and mobility,
with nationwide coverage. These are requirements dif-
ferent from a ‘‘common’’ eMMB slice, which might
also include optimizations to better support transmission
protocols like e.g. DASH.
If the tenant has no networking expertise or does not want
to manage the network slicing mechanics, the NO will provi-
sion and operate the corresponding network slice(s) tailored
for this communication service (i.e., Offer type N). In this
case, the KPIs of interest to the tenant could be the desired
coverage area, target throughput, the number of devices,
etc. The use of this offer enables better optimization of the
resources required to implement the slices, as the MSP keeps
the full control on those resources, and therefore algorithms
can leverage multiplexing gains of traffic among slices.
On the other hand, in case the tenant wants to partially
or completely control the actual resources implementing the
slice (i.e., offer types L or F), the potential gains due to
multiplexing are reduced. In this case, the MSP needs to
carefully review the allocation of resources before accepting
new slice requests, as the tenant might not accept any change
in the agreed resource allocations (cf. Sec. III-B regarding
resource commitment models) in case a new slice instance
shall be commissioned, or another tenant is experiencing a
lack of resources.
D. USE CASE: INDUSTRY 4.0
Here we consider the case where the tenant is a vertical
enterprise that owns some Industry 4.0 factory sites.1More
specifically, we assume that the tenant wants a secure network
for highly sensitive traffic from monitoring sensors in the
product line of the factory floor. The requirements for this
indoor-only service are: low latency, average throughput, and
very-high reliability, with no support for mobility.
The driver for this scenario comes from the vertical’s
requirement for a secure private network within the factories
fully isolated for its critical monitoring sensors network. Full
isolation of the vertical’s traffic against any network provider
or network user is only possible by running a private network
on the infrastructure owned by the vertical. The vertical relies
on the resources provided by its own private network infras-
tructure inside the factory premises. In that case, the vertical
combines the role of tenant, NO, and infrastructure provider.
More precisely, the various organizations of the vertical (pro-
duction line, delivery line entities) could be considered as
‘‘inner tenants’’ of the vertical. In this ecosystem, the MSP
becomes a possible business partner, selling to the vertical its
expertise into designing and rolling out IoT networks onto
the vertical’s private infrastructure, or some software assets
for realizing the non-critical slices.
In addition to critical IoT communications, the vertical
uses network slicing for optimizing its own network for its
various organization entities (e.g., product line, delivery line,
commercial service). An example of three slices (critical
IoT, non-critical IoT, and corporate eMBB) deployed by
the Industry 4.0 vertical on its own private infrastructure is
depicted in Fig. 4. These slices are:
Private slice for critical IoT, indoor coverage. This slice
may require customization down to the PHY layer, with
only transmission (and reception) functionality shared
across other network slices. It will implement its own
specific radio scheduler, which will result in higher com-
plexity when managing radio resources for multiples
slices, and will simplify Network Access Stratum NAS)
signaling, as sensors inside the factory do not need any
mobility management.
Private slice for non-critical IoT, factory campus cov-
erage. This slice serves the forklift sensors inside the
factory campus. The non-critical IoT slice may not need
a customized PHY or MAC layer, which increases the
deployment flexibility and reduces their costs, given that
no proprietary technology is needed.
Private slice for eMBB, factory campus coverage. This
slice provides employees with access to its private
1Industry 4.0 refers to a fourth industrial revolution combining pro-
duction methods with state-of-the-art information and communication
technology [20]
VOLUME 7, 2019 79221
V. Sciancalepore et al.: Future-Proof Architecture for Management and Orchestration of Multi-Domain NextGen Networks
FIGURE 4. Architecture for deploying industry 4.0 slices onto a vertical private network.
corporate data network, which may be connected to
the Internet and thus allow also Internet access. The
vertical’s employees have mobile devices and subscrip-
tions to access the corporate network, with both of
them (devices and subscriptions) being managed by the
vertical.
IV. ETSI NFV MANO AND ITS LIMITATIONS
In this section, we first provide a quick overview of the ETSI
NFV MANO Framework, describing its most relevant ele-
ments (some of which will be extended by our architecture),
and summarizing the different strategies that could be used
to assign resources to network slices. Then, we discuss the
limitations of the framework when dealing with multi-domain
operation.
A. SUMMARY OF THE ETSI NFV MANO FRAM EWORK
Fig. 5depicts the NFV MANO system architecture proposed
by the ETSI ISG NFV [6], which is composed of the follow-
ing three main functional blocks:
Virtualized Infrastructure Manager (VIM) for the man-
agement of NFV Infrastructure (NFVI) resources like
computing, networking, and storage.
Virtualized Network Function Manager (VNFM) for the
lifecycle management (LCM) of Virtualized Network
Functions (VNFs) that are deployed and instantiated
over the NFVI.
NFV Orchestrator (NFVO) for the service and resource
management of the network services (NS) that are
formed by chaining multiple VNFs over virtual
links (VL) and characterized by the VNF Forwarding
Graph (VNF-FG).
These three functional blocks interact with each other
using standard interfaces, which are specified for the relevant
reference points and serve to provide lifecycle management
of virtualized resources belonging to different realms.
In addition to these functional blocks, there are var-
ious catalogs that contain relevant descriptor files such
as, e.g., the VNF Descriptor (VNFD) file and the NS
Descriptor (NSD) file, specifying the operational, functional,
resource, performance, and policy requirements of the VNFs
and NSs, respectively. The NFV MANO system allocates
FIGURE 5. The NFV Management and Orchestration (MANO) framework
as specified by ETSI (cf. [6]).
resources, deploys, and instantiates VNFs/NSs over the NFVI
according to the request and requirements specified in the
respective VNFD/NSD files.
The multiple NS instances (that could belong to differ-
ent tenants or to the Network Operator itself) are managed
by the NFV MANO system. As part of the LCM tasks,
the system has the capability to instantiate, migrate, scale-
in/out/down/up, update/upgrade, and terminate VNF/NS
instances. Also, the system can orchestrate network resources
and NSs based on a set of policy rules.
The architecture presented in this section is the current
state of the art for NFV Orchestration, and it has several
real-life implementations coming from different working
groups such as OSM [21].
B. RESOURCE COMMITM ENT MODELS
ETSI directives on resource management procedures [22]
define three so-called ‘‘resource commitment models’’: reser-
vation model, quota model, and on-demand model: with the
quota model, the NFVI limits the resources that a slice
can obtain from a particular NFVI-PoP (Point of Presence);
with the reservation model, a specified amount of resources
is statically allocated to a particular tenant or slice, even
if this results in the resources remaining idle for most of
79222 VOLUME 7, 2019
V. Sciancalepore et al.: Future-Proof Architecture for Management and Orchestration of Multi-Domain NextGen Networks
the time; finally, with the on-demand resource commitment
model, no reservation or preemptive allocation of resources
is made: NFVI resources are assigned only once they are
requested.
Regarding the co-existence of the quota model and the
reservation model, a VIM will, as the default behavior, also
apply the slice quota to the slice reservation being made.
However, further rules will determine the behavior of the
VIM if a reservation exceeds the specified slice quota. In our
architecture, these rules are determined from the policies
implemented in a novel element, the Inter-slice Resource
Broker.
C. ISSUES IN MULTI-DOMAIN SCENARIOS
The multi-domain issue is being duly addressed in ETSI ISG
NFV that has published two reports. Report [23] deals with
managing the connectivity of an NS deployed over multiple
NFVI-PoPs, and assumes a single MANO system that man-
ages interconnectivity issues over the WAN links connect-
ing these NFVI-PoPs. The second report [24] highlights the
different architectural options and recommendations to sup-
port MANO operations in multiple administrative domains.
According to [24], two distinct administrative domains are
specified: one domain is characterized by the NFVIaaS
provider and the other by the NFVIaaS consumer. These
two domains may either be owned by the same or different
actors, and the NFVIaaS provider domain can support mul-
tiple NFVIaaS consumers. For the sake of clarity and expla-
nation, this work re-uses a few ETSI concepts and therefore
refers to the NFVIaaS provider’s administrative domain as
‘‘Infrastructure Domain’’ (IDo) and the NFVIaaS consumer’s
administrative domain as ‘‘Orchestration Domain’’ (ODo).
The IDo consists of NFVI resources and has at the min-
imum one VIM. The ODo, on the other hand, consists of
the VNFM and the NFVO functional blocks of the NFV
MANO system. It may also be that the IDo consists of the
VIM and VNFM in which case the ODo will be composed of
NFVO only. A single IDo may support one or more isolated
ODo(s), and/or a single ODo may consume the resources
of multiple IDos. The latter is the case when the ODo has
to take care of the LCM of NS(s) deployed across multi-
ple IDos, and this also is one of the focus issues of this
article.
There are several deployment options for IDo/ODo, and
each one will have performance implications on the ODo
performance when providing MANO services. We focus
on the scenario where a NS is deployed across multi-
ple IDos and analyze the implications on the MANO per-
formance. It should be noted that [25] and [26] introduce
the notion and definition of Quality of Decision (QoD),
which can be used to quantify the performance of a
MANO system. QoD is measured in terms of the following
criteria:
How resource efficient the management action is. The
resource efficiency is in turn measured in terms of:
Whether both the long-term and short-term
resource requirements of the managed VNF will be
fulfilled in the selected compute node.
How non-intrusive a management action has been
for other VNFs that are already provisioned in the
selected compute node. That is, to what extent will
the managed VNF VM affect the performance of
other VNFs in the selected compute node in terms
of resource availability.
Number of times the management action has to be exe-
cuted before the most-suitable compute node is deter-
mined to migrate/scale the managed VNF to.
The timeliness of the computation and execution of
MANO LCM actions. The latter criterion is more rel-
evant in the management of a multi-site NS scenario as
described below.
Fig. 6shows two main deployment options of the MANO
in a multi-site environment. Fig. 6(a) shows a scenario where
a central ODo is used to manage multiple IDos in same
and different NFVI-PoPs. In case the IDos and the ODo are
co-located within the same NFVI-PoP (e.g., NFVI-PoP-3 in
the figure), then the LCM operations on the NS instance(s)
can be imparted without much impact on MANO execution
time.
However, such collocated management of NS instances is
no longer suitable when the IDos are deployed in geograph-
ically dispersed NFVI-PoPs and interconnected via WAN
infrastructure. In such a scenario, i.e., when managing an NS
that is deployed across multiple NFVI-PoPs, issues in the
WAN infrastructure can impact the performance and hence
the QoD of the NFVO/VNFM in the ODo. For instance,
WAN delays will impact the timely delivery of performance
monitored data/KPIs from the different IDos towards the
centralized ODo. This, in turn, will delay the NFVO/VNFM
to analyze and derive appropriate LCM decision and will also
result in the delay of the application of the LCM actions.
It could also render the monitored data, and hence the corre-
sponding LCM decision, as stale by the time it gets processed.
Moreover, a central ODo also introduces a single-point-of-
failure.
To overcome the above issues of managing a multi-site
NS from a central ODo, the [24] proposes to distribute the
ODo such that each NFVI-PoP has its own MANO stack.
In other words, each IDo domain will have its own ODo (i.e.,
NFVO and VNFM) as illustrated in Fig. 6(b). The MANO
operations on the multi-site NS instance(s) are then coordi-
nated in a peer-to-peer manner between the respective NFVO
instances over a newly proposed Or-Or reference point [24].
However, this approach not only brings more complexity, but
is sub-optimal in view of the delay-sensitive nature of MANO
operations. Considering the above challenges, we propose
an architectural option that is based on a distributed MANO
system (as in as in Fig.6(b)), but instead of a peer-to-peer
interaction, our architecture proposes that the coordination
between the different MANO systems is carried out by an
VOLUME 7, 2019 79223
V. Sciancalepore et al.: Future-Proof Architecture for Management and Orchestration of Multi-Domain NextGen Networks
FIGURE 6. MANO Deployment options in multi-domain scenario.
(a) Centralized Management of Infrastructure domains. (b) Distributed
Management of Infrastructure domains.
FIGURE 7. ISRB and tenant-specific ETSI NFV MANO stack instances.
over-arching entity. Our proposed architecture extends the
standard ETSI NFV MANO system (see Fig.5) and offers a
versatile solution. The details of our proposed multi-domain
MANO system architecture is presented in the next section.
V. A NOVEL MULTI-DOMAIN MANAGEMENT AND
ORCHESTRATION ARCHITECTURE
Multi-domain orchestration allows network operators (NOs)
to deploy network slices in multiple administrative domains
to support, e.g., some specific performance requirements
or particular tenant requests. In general, ETSI assumes
that in a typical NFV scenario, there will not be a sin-
gle organization controlling and maintaining a whole NFV
system [27]. It therefore provides multiple options for the
FIGURE 8. Tenant-specific ETSI NFV MANO stack instances.
specific mapping of NFV MANO functions to administrative
domains. While a subset of options assumes a rather horizon-
tal split into IDos and ODos (cf. Sec. IV-C), this section first
depicts a solution where a single IDo-0 hosts multiple ODos.
In a second step, IDo-0 integrates NFVI resources of another
IDo-1 but maintains the single IDo perspective as seen by the
ODos.
If the NFV Infrastructure (NFVI) spans across several
locations (i.e., multiple NFVI-PoPs), the network providing
connectivity between these locations can be regarded to be
part of the NFVI [27]. NFVI resource management across
an NO’s IDos can be performed using one or more VIMs as
needed. Usually, however, VIMs are intended to manage the
NFVI resources within one NO’s IDo. For the multi-domain
orchestration framework proposed in this article, required
resources from these different domains are integrated into a
single NFVI block that is orchestrated from just one adminis-
trative domain. From the business perspective, specific agree-
ment among the different administrative domains should be
reached to make this possible. From the technical perspective,
the domain designated for orchestration is granted access to
the resources in the ‘‘external’’ domain(s).
The proposed Management and Orchestration architec-
tural framework, therefore, includes the novel function of an
Inter-Slice Resource Broker (ISRB) which enables orches-
tration per ODo by managing a set of complete ETSI NFV
MANO stacks, i.e., with their corresponding NFVO, VNFM,
VIM and catalog sets as depicted in Fig 7. The NO as the
owner of the IDo operates a dedicated, so-called Operator-
MANO (o-MANO) stack (this and further management func-
tions of the NO are depicted as solid rectangles). Both the NO
(or specific departments thereof) and tenants can define their
own ODo which is associated with the dedicated so-called
tenants-MANO (t-MANO) stack. An ODo-specific t-MANO
stack orchestrates a tenant’s network slice(s). In particular,
the VIM(s) of each ODo manage(s) the set of NFVI resources
assigned to the tenant (as outlined next, NFVI resources
can come from multiple infrastructure domains). This archi-
tecture has the additional flexibility for enabling per tenant
t-MANO stacks based on the paradigm proposed in [28].
As a second major extension beyond ETSI NFV MANO,
Fig. 8shows the framework for cross-IDo orchestration:
NFVI-0, owned by InP-0 (i.e., the NO), virtually extends
its IDo by integrating resources from NFVI-1, owned by
InP-1. Each InP could have instantiated different t-MANO
stacks for different tenants, beyond the InP’s o-MANO stack.
The framework for combined multi-ODo and multi-IDo
79224 VOLUME 7, 2019
V. Sciancalepore et al.: Future-Proof Architecture for Management and Orchestration of Multi-Domain NextGen Networks
FIGURE 9. Multi-domain networking in the infrastructure domain.
orchestration is depicted in Fig. 9. For the sake of simplic-
ity, no t-MANO stacks are depicted in the NFVI-1 domain.
The following subsections describe the conceptual details
and the new components of the proposed framework. For
a detailed description of a first implementation and oper-
ation of the multi-domain management and orchestration
system, the reader is referred to [29], which describes a 5G
testbed setup in a commercial seaport environment. The setup
leverages on the presented technology to orchestrate several
network slice instances across the administrative domains of
the port authority and the mobile network operator. Another
implementation is provided in [30], which describes an open
source implementation of a multi-slice radio access network
and the orchestration framework.
A. NEW COMPONENT S AND CONCEPTS
1) INTER-SLICE RESOURCE BROKER (ISRB)
Sharing the same infrastructure across tenants and network
slices entails the following trade-off: resource reservation
versus flexible resource sharing. Specific service-level agree-
ments (SLAs) define the concrete embodiment of reservation
and on-demand allocation rules. For example, a tenant may
request a fixed amount of NFVI resources (in such a case,
the t-MANO stack assigned to a tenant could exclusively
manage the allocated quota of resources). In another case,
a tenant may agree that a percentage of the associated, but
unused resources, may be dynamically allocated to other
tenants or slices, thus realizing cost savings and allowing
multiplexing gains. The role of the ISRB is to have a general
view of the whole infrastructure that can be offered within a
single administrative domain, as well as monitoring the usage
of the resource subsets allocated to a tenant’s t-MANO stack.
It controls the dimensioning of resources that are assigned
to each tenant and their status, including those resources not
yet assigned. Upon instantiation of a tenant’s network slice,
initial quotas are assigned to the responsible t-MANO stack
instance. Despite this initial assignment, resource allocation
can be reshaped at runtime if tenants request for that. This
also implies that in case a tenant’s slices do not utilize all
allocated resources, the reserved but idle resources will not
automatically be made available to slices of other tenants.
Moreover, special terms in SLAs can allow a tenant to exceed
its assigned quota for a certain time and at a certain cost.
In case a t-MANO stack is permanently decommissioned,
the associated resources are released to NFVI-0 again.
Except for the NO (or InP, respectively), tenants are neither
aware of the existence nor the resource utilization level of
other tenants. They only have an SLA specifying their right
to use certain resources in a certain manner.
The rules for the resource commitment model utilized by
individual tenants as well as for resource sharing and priori-
tization are kept in a policy catalog maintained by the ISRB.
This catalog also contains an inventory map of currently
available infrastructure resources and their allocation to each
tenant. The tenant-specific NFVO reports resource utilization
to the ISRB that may use this information to reshape the
current association patterns according to new external trig-
gers, such as a new slice creation request or a re-orchestration
request from an already hosted slice.
In the case that NFVI-0 resources are not sufficient to
accommodate all slice instances and their requirements,
NFVO-0 triggers a request for additional resources with
ISRB-0. As one option, ISRB-0 can subsequently request
resources from another IDo, e.g., ISRB-1, cf. Fig. 9.
If ISRB-1 can accommodate the request, the o-MANO stack
of InP-1 will then partition an according subset of NFVI-1
resources and expose them to InP-0’s o-MANO stack as one
or multiple separate NFVI-PoP(s). For this purpose, VIM-1 of
InP-1 will integrate towards VNFM-0 and NFVO-0 using the
respective reference points. After the successful integration
of the NFVI-1 resource subset into the NFVO-0 IDo, ISRB-
0 can henceforth re-allocate them to other t-MANO stacks
within InP-0.
2) NFV INFRASTRUCTURE CONCEPT
The IDo NFVI-0 constitutes a conceptual re-design of the
ETSI NFV IDo. This comprises two major characteristics:
(1) Disposition of NFVI-0 resources into tenant-specific sub-
sets according to the resource commitment model requested
by the tenant and (2) integration of NFVI resources from other
domains into NFVI-0 domain forming a virtually integrated,
single domain, cf. Fig. 9. For the depicted setup, InP-0 and
InP-1 have to reach an agreement on:
The amount and type of InP-1 NFVI resources to be allo-
cated into the InP-0 Infrastructure Domain; this includes
the typical NFVI resources (compute, storage and net-
working).
The commitment model to allocate them (static quota,
dynamically scalable quota, allowed/supported proto-
cols, etc.).
The agreement could also include the VIM(s) in case
the rented resources would require a specific VIM
VOLUME 7, 2019 79225
V. Sciancalepore et al.: Future-Proof Architecture for Management and Orchestration of Multi-Domain NextGen Networks
(VIM-1 in Fig. 9). From the technical point of view,
the only condition for integration is that such ‘‘external’
VIMs shall provide the ETSI NFV Or-Vi and Vi-Vnfm
reference points in order to connect with the VNFM(s)
and the NFVO.
Additional operational policies (including scaling
rules, security requirements, redundancy, and over-
provisioning levels).
Regarding the fundamental realization of multi-domain
orchestration, this article considers two scenarios:
1) InP-0 extends the NFVI-0 infrastructure in order to
provide a better service to the hosted tenants. E.g.,
Tenant-02 could request more infrastructure resources
from InP-0 than NFVI-0 can satisfy. Then, the InP-0
signs a business contract with another InP (InP-1) to
include resources from that provider into his infras-
tructure domain. The request of Tenant-02 can now
be satisfied and the new resources can be made
available to Tenant-02’s t-MANO stack, no addi-
tional business contract between InP-1 and Tenant-
02 is required. Technically, the fact that some NFVI
resources used by Tenant-02’s t-MANO stack are actu-
ally located in the InP-1 domain is transparent from
the tenant perspective. In other words, for Tenant-
02 it looks like resources from the InP-0 domain
are used and only a single business relationship is
maintained.
2) The tenant explicitly wants to extend its slice using
specific infrastructure from another InP (InP-1). This
could happen when a tenant already has certain infras-
tructure up and running on a different InP and does not
want to take the effort of migrating that infrastructure
into the InP-0 domain. I.e., the tenant already has a
business contract with both, InP-0 and InP-1. In this
case, besides the corresponding update of these both
contracts, a new contract involving both InPs needs to
be agreed.
B. EXTENDED ETSI NFV MANO COM PONENTS
In order to support multi-domain orchestration, the NFV
Orchestrator NFVO-0 (cf. Fig. 9) plays a special role since
it oversees and executes, respectively, the management of
NFVI-0 resources based on the constraints received by ISRB.
Specifically, it needs to support and/or implement the follow-
ing technical aspects:
InP-1 shall isolate and assign the requested NFVI
resources and provide the necessary interfaces/reference
points to InP-0. In particular, this applies to the Nf-Vi
reference point when the InP-0 deploys its own VIMs,
or the Or-Vi and Vi-Vnfm reference points when the
VIMs are supplied by the InP-1.
InP-0 shall take the newly integrated NFVI resources
and associate those to a o-MANO or one (or mul-
tiple) t-MANO stacks, thus forming a ‘‘merged’’,
single-domain set of resources for these stacks.
While Fig. 9does not assume any deployment con-
straints of VIMs or the entire o-/t-MANO stack
instances, for performance reasons (e.g., latency) it
might be necessary to add such constraints. However,
this does not change the functional perspective of the
architecture.
A tenant’s t-MANO stack should have the full informa-
tion about logical node topology and resources, address
space, etc. within the tenant’s dedicated shares of the
two IDos (NFVI-02 and NFVI-12 in Fig. 9). Hence,
NFVI-02 and NFVI-12 can be used, together with the
associated VIMs, to set up the interconnections between
VNFs in the tenant’s ODo accordingly.
Security issues: A tenant may not want to rely on trust
in arbitrary InPs –hence an SLA between tenant and
NO may restrict the choice of InPs the NO can use
to host tenant functions. It can be further noted that
in a multi-domain scenario, inherently the number of
involved parties, data centers, interfaces, and software
routines will be higher, thus increasing the attack sur-
face. There is no specific remedy. Against this, the gen-
eral security rules apply, in particular designing a sound
security architecture and implementing it carefully in
order to minimize the threat of exploitable vulnerabil-
ities.
Adjustment of resources assigned to a specific NFV
MANO stack instance relies on an ‘offline’ procedure
(as for the single-domain use case). I.e., t-MANO stacks
could be assigned with a specific quota of resources
(regardless of whether those resources come from a
single domain or multiple InP domains) that should be
properly dimensioned to dynamically scale during the
slice operation. In both cases (single-domain or multi-
domain), an offline re-negotiation is necessary if the
assigned quota of resources have not been sufficient to
meet the tenant’s necessities.
C. NETWORK SLICE LIFECYCLE
When a network slice shall be commissioned, a network slice
descriptor is provided to the ISRB. Such a descriptor does
not only contain information on control and data layer net-
work functions of a network slice, but also on the associated
MANO stack instance for managing these network functions
(NFs). Hence, the network slice descriptor is comprised of
two major parts that specify the functions, resources, and
policies that are required to, respectively, (i) perform life-
cycle management for a network slice and (ii) realize the
network service requested by the tenant. While the former
point comprises a specification of the NFV MANO stack
instance (NFVO, VNFM, VIM, NFVI instances, catalogs
for network services and functions, etc.) that is dedicated
to the lifecycle management of the network slice, the latter
includes the network service descriptor(s), i.e., the collec-
tion of VNFs and PNFs that, as a whole, form the control
and data layer architecture of the particular network slice
instance.
79226 VOLUME 7, 2019
V. Sciancalepore et al.: Future-Proof Architecture for Management and Orchestration of Multi-Domain NextGen Networks
FIGURE 10. Lifecycle of a network slice [12].
According to [12], the network slices lifecycle manage-
ment is composed of four distinct phases (depicted in Fig. 10):
(i) preparation, (ii) instantiation, configuration and activation,
(iii) run-time, and (iv) decommissioning. The network slice
descriptor contains the necessary information to carry out
phases (ii) to (iv) appropriately.
In a first step, the ISRB uses the network slice descriptor
to commission a new NFV MANO stack. In the second step,
the same network slice descriptor is utilized to generate the
necessary objects and models which the ETSI NFV MANO
stack instance operates on, i.e., NFV service catalogue,
VNF/PNF catalogues, NFV instances, and NFVI resources.
For the allocation of the NFVI resources that are under control
of this MANO stack instance, the ISRB uses a combination of
the resource commitment models as outlined in Section IV-B.
Commissioning of the network slice control and data layer
functions is triggered by the Inter-slice Resource Broker via
the Os-Ma-Nfvo reference point of the NFVO by providing
or referring to the set of network service descriptors to be
instantiated.
The network slice lifecycle management is now del-
egated to the NFV MANO instance and the accord-
ing domain-specific application management functions (see
Fig. 9). This includes several tasks, including the instantiation
and configuration of the network services and associated
network functions, the activation of the network slice, the run-
time supervision and reporting as well as upgrading, reconfig-
uration, scaling, and finally, the deactivation and termination
of the network slice.
VI. PERFORMANCE EVALUATION
Practicability and implementability issues are the main
aspects that must be taken into account when considering
an advanced multi-domain orchestration. Hereafter, we high-
light the main advantages along our novel solution with
respect to traditional and legacy approaches, as well as the
experienced limitations using off-the-shelf equipment. Last,
we show how our solution can be easily integrated to existing
standard interfaces without requiring significant architectural
changes.
A. LCM OPERATIONS TAXONOMY
For our validation analysis, we consider a legacy multi-
domain orchestration wherein a single MANO stack is
deployed across different infrastructure providers. In partic-
ular, each infrastructure domain is provided with its own
Virtualized Infrastructure Manager (VIM) component and
(might be provided) with the VNF Manager (VNFM). The
NFV Orchestrator (NFVO), as the entity in charge of taking
TABLE 4. Scope of LCM operations.
Lifecycle Management (LCM) decisions, is shared among
different infrastructure domains.
We detail in Table 4the relevant LCM operations con-
sidered within our simulation campaign. When an LCM
operation is executed, the VNF Forwarding Graph (VNFFG)
is adjusted accordingly. However, NFVO can trigger LCM
operations locally, i.e., without affecting external domains’
VNF Forwarding Graphs (VNFFGs), or globally, i.e., LCM
operations on other infrastructure domains are required. Note
that in some cases, operations can be executed within local
and global scope. This is described in the network service
descriptor (NSD) following an event-threshold definition:
if the resource increase request exceeds such a threshold,
i.e., it may impact on VNFs chained within the same network
service but running on different infrastructures, the operation
will be executed globally. While some of those operations
might also be monitored to avoid unhandled service degra-
dations, this might further incur in the additional overhead on
the communication means, as shown in the remainder of the
section.
B. VALIDATION RESULTS
We provide our preliminary validation results by studying the
communication overhead as well as the end-to-end service
delay while deploying three relevant multi-domain network
slices.
Our proposal relies on the multi-MANO deployment,
i.e., different independent classical MANO stacks are
deployed on each single infrastructure domain, namely
NFVI-PoP, and all of them are connected to the Inter Slice
Resource Broker (ISRB) component—which can be envi-
sioned as a stand-alone software running our algorithms,
as the one described in [31]—through a dedicated interface.2
The MANO deployment is realized through OpenStack heat
template based on a pre-configured ONAP deployment.3
To clarify the concept, we depict the baseline architectural
solution and our novel framework in Fig. 11. This brings a
two-fold advantage: (i)trustworthiness, as the ISRB is rec-
ognized by all connected infrastructure domains as the only
trusted entity so as to avoid any security threat, and (ii) a clean
master-slave relationship, with the ISRB constantly keeping
track of the status changes of each infrastructure domains, and
taking decisions on LCM operations, playing as a centralized
2Note that, such a new interface can be readily mapped onto the Or-Or
interface as per the standard report [24].
3Advanced changes to existing MANO orchestrator solutions are out of
the scope of this paper. However, they will be addressed in future work.
VOLUME 7, 2019 79227
V. Sciancalepore et al.: Future-Proof Architecture for Management and Orchestration of Multi-Domain NextGen Networks
FIGURE 11. Validation reference scenarios. (a) Legacy multi-domain deployment. (b) ISRB multi-domain deployment.
FIGURE 12. Validation reference scenarios.
entity only when unexpected LCM operations might affect
the overall performance.
We consider three different network services that auto-
matically chain distinct virtualized network functions dis-
tributed over different infrastructure domains, as shown
in Fig. 12. The first considered multi-domain slice delivers
enhanced Mobile BroadBand (eMBB) services by deploy-
ing a virtualized Evolved Packet Core (vEPC), virtualized
switch and transport nodes on the first infrastructure domain
(NFVI-1). A virtualized scheduler and traffic shaper is then
deployed on the second infrastructure domain (NFVI-2)
whereas the virtualized firewall function is executed on the
third infrastructure domain (NFVI-3). The second network
slice provides streaming services with a virtualized video
optimizer function and a virtualized load balancing function
deployed on different infrastructures. We assume such two
network services as delay-tolerant services. Last, we con-
sider an ultra-reliable low latency communication (URLLC)
multi-domain slice running public safety network services.
Specifically, the network service deploys a virtualized imag-
ing processing function on the second infrastructure domain
while the target-matching virtualized function (and its asso-
ciated database) on the third NFVI-PoP.
TABLE 5. Empirical evaluation parameters.
We carry out an exhaustive simulation campaign to eval-
uate the complexity of our solution, namely ISRB, against
aLegacy case in terms of the overhead of different inter-
faces as well as the end-to-end service delay that may
play a fundamental role in case of low-latency applications
(for e.g. URLLC services). Simulation parameters are listed
in Table 5. We implement and emulate the communication
between each deployed virtualized function and we generate
synthetic traffic traces based on a Pareto statistical distribu-
tion.4In the legacy scenario, the communication between the
NFV orchestrator and VNF Managers is stable as the NFVO
continuously collects information on running NFVs (moni-
toring) while quickly reacts in case of unexpected changes.
In our novel multi-domain orchestration solution, our novel
ISRB continuously retrieves the network service descriptor
for each network slice. Once resources have been set up
on different NFV infrastructures, the NFVO locally gathers
monitoring information while transmitting, with a fixed fre-
quency, few packets on the status of the NFVI to the ISRB.
Some LCM operations are taken locally without requiring
the intervention of the ISRB, as specified in Table 4. How-
ever, some of those operations, such as scale-down, scale-in
and migration, might trigger the ISRB reconfiguration only
if the event requires a number of resources that exceeds a
pre-defined threshold, as previously explained.
In Fig. 13, we evaluate the communication overhead
between the centralized entity (NFVO in case of legacy and
ISRB in case of our novel solution) and the distributed NFV
infrastructure. We run our simulations for 20 seconds with a
4We consider this distribution as it exhibits a long-tail in the density func-
tion that helps while evaluating queues of packets and, in turn, experienced
delay.
79228 VOLUME 7, 2019
V. Sciancalepore et al.: Future-Proof Architecture for Management and Orchestration of Multi-Domain NextGen Networks
FIGURE 13. Complexity analysis with increasing traffic at time t=10 seconds. (a) NS 1 with VNF scale-up operation on NFVI-1. (b) NS 2 with VNF
scale-out operation on NFVI-2. (c) NS 3 with VNF migration operation on NFVI-2.
FIGURE 14. Whiskers plot for the overall end-to-end service delay.
plot granularity of 1 ms. In the first 10 seconds, the average
amount of offered load is constant. For the first network
service (please refer to Fig. 12), we assume networking traffic
suddenly increasing at time t=10 seconds. This auto-
matically triggers a scale-up operation of the vEPC function
in NFVI-1, as shown in Fig. 13(a). Several messages are
exchanged to deal with the resource increasing resulting in
a huge overhead that stabilizes after a few seconds. How-
ever, the number of messages exchanged is higher due to
the larger number of used NFV resources. Conversely, ISRB
solution does not show any criticism while dealing with the
scale-up operations. The number of required resources is
within the fixed threshold thereby preventing other NFVIs
to apply local reconfigurations. In Fig. 13(b), when traffic
flows suddenly increases a scale-out operation is triggered on
NFVI-2. In particular, multiple instances of the virtualized
Video Optimizer functions are instantiated to deal with the
unexpected traffic boost. This automatically affects functions
running on NFVI-3 so that both legacy and ISRB exhibit
signaling message exchange. However, due to the limited
number of messages required by the ISRB, even in that case
ISRB results in a short-time and low overhead that turns into
regular frequent messages after less than 1 second. Last, net-
work service 3 has a significant impact on other infrastructure
domains when the traffic increases at time t=10 seconds,
as shown in Fig. 13(c). In particular, a function migration is
required, and both solutions deal with an increasing signaling
overhead that lasts roughly 1 and 3 seconds for ISBR and
legacy, respectively.
While the relative gain of ISRB could appear limited,
this token example unveils the difference in terms of over-
head between the two considered solutions for a short-time
period (20 seconds) and three different NFVI-PoPs. Real
NFV deployments may support up to hundreds of NFVI-PoPs
with a time window that could incur in a huge signaling
burden.
Finally, we depict in Fig. 14 the whiskers plot for experi-
enced delay values over 20 seconds of simulation. Network
service 1 and 2 provide high end-to-end delay as they are
delay-tolerant services. However, ISRB significantly outper-
forms the legacy scheme when the network service 2 is in
place. In case of network service 3, although a centralized
approach (legacy) could slightly benefit the system in terms
of manageable delay, the complexity required does not pay
off. Note that in such case, ISRB still provides affordable
delay performance (below 20 ms).
VII. RELATED WORK
The architecture presented in this paper extends the one
proposed by ETSI NFV to take into account specific char-
acteristics of network slicing, multi tenancy, and service
personalization, as already described above. In this section,
we describe the main difference between our proposed archi-
tecture and one of the most prevalent alternatives to the ETSI
NFV MANO Architecture, namely, the one proposed by the
Open Network Automation Platform (ONAP) [32]. We also
compare our proposal with another relevant architecture for
5G Networks Management and Orchestration, which is pro-
posed by the European 5G-Exchange (5G-Ex) project.
A. ONAP
The ONAP initiative was launched in 2017 with the goal
of providing a common platform to deliver differentiated
network services on a shared infrastructure. As the main
objective of ONAP is generality, in the latest version of their
architecture, the ONAP consortium proposes a clear split
between the general, abstract models that tackle the problem
of service design and the specific modules that control the
lifecycle management of such services. More specifically,
they define the Service Design and Creation (SDC) and
the Runtime Framework realms. In a nutshell, they perform
VOLUME 7, 2019 79229
V. Sciancalepore et al.: Future-Proof Architecture for Management and Orchestration of Multi-Domain NextGen Networks
FIGURE 15. The ONAP architecture.
tasks that are commonly categorized under Network Manage-
ment (SDC) and Orchestration (Run Time).
Therefore, all the tasks related to the abstraction of
resources and the high-level deployment of network ser-
vices are performed within the Design Time Framework,
while all the tasks related to the lifecycle management
and the actual representation of those resources are per-
formed by the run time execution framework. The full
specification of the architecture modules is depicted in the
Fig. 15.
Within ONAP, a network service is thus defined as a collec-
tion of recipes that specify the behavior of a specific service
which are deployed in the ONAP Operation Manager Portal.
Recipes detail, among other things, factors such as the VNF
deployment, the metric that have to be analyzed and the self-
healing of the network.
The ONAP and the architecture presented in this paper
share the same field of operation (i.e., the management,
orchestration, and operation of a multi-service network)
although from a very different standpoint. ONAP is very
much code-oriented and module-driven, while our proposal
builds on top of the ETSI NFV framework and tackles the
same problem with a top-down approach. In the following,
we describe how the different modules of the two architec-
tures relate among them.
Management and Service Orchestration: our architec-
ture perform such tasks at the Service Management and (par-
tially) at the Inter Slice Resource Broker, that defines the
interface towards the NFV-O for the subsequent resource
orchestration procedure. Within ONAP, this functionality
is performed by the SDC framework that then interfaces
towards the run-time modules for lifecycle management.
Resource Orchestration and Lifecycle Management: in
this work we specify procedures for the network slice lifecy-
cle management by leveraging on the ETSI NFV Architecture
modules. ONAP also adopts a similar approach, being the
Virtualized Function Controller a replacement of the ETSI
ENI Orchestration Stack.
From the above discussion, we can recognize one
major difference between the two proposals. While in the
ONAP architecture the concepts of network slicing and
multi-tenancy are left open and possibly enforced through the
ONAP Operations Manager, in our architecture we clearly
define specific roles for the involved stakeholders. We believe
that this tighter definition of the interaction between the roles
of the tenants / service providers and infrastructure providers
as done within our proposed architecture will eventually lead
to a better and clearer interaction of concurrent services
provided on the same infrastructure.
B. 5G-EX
The 5G-Ex project targeted exactly the same problem as
ours: how to provide multi-domain orchestration in a multi-
slice, multi-tenant network [33], [34] Their approach, analo-
gously to ours, define a hierarchy of orchestrators to solve
the multi-domain problem, with a multi-domain orchestrator
(that belong to different network operators) linked to specific
domain orchestrators.
However, the main difference of 5G EX compared to our
approach is the limited flexibility in the type of offers. That is,
the interaction between the tenants and the network operators,
just happens through Business to Customer (B2C) interfaces
that allow only no or limited control to the tenant. Instead,
we believe that a more flexible management API will enable
new and more efficient business models such as the ones
described in Section III.
VIII. CONCLUSIONS
In our work, we have presented a novel 5G management
and orchestration architecture that overcomes the main lim-
itations of the current state-of-the-art frameworks. Namely,
we have designed our architecture with the goals of being
backward compatible while natively taking into account the
novel concepts of multi-tenancy and network slicing across
multiple infrastructure domains. A core-contribution is also
the economic analysis presented in the article that further
motivates the need for such a flexible architecture. In particu-
lar, it comprises (i) a novel Inter-slice Resource Broker entity
and the (ii) NFV infrastructure concept. Validations results
over realistic deployments show how the proposed mod-
ules outperform legacy solutions at affordable costs while
supporting fundamental operations in 5G multi-domain net-
works.
REFERENCES
[1] NGMN Alliance, ‘‘Description of network slicing concept,’’ NGMN
Alliance, Frankfurt, Germany, White Paper, 2016.
[2] Study on New Radio (NR) Access Technology Physical Layer Aspects,
document 3GPP TR 38.802 V15.0.0, 2018.
[3] P. Rost, C. Mannweiler, D. S. Michalopoulos, C. Sartori, V. Sciancalepore,
N. Sastry, O. Holland, S. Tayade, B. Han, D. Bega, D. Aziz, and H. Bakker,
‘‘Network slicing to enable scalability and flexibility in 5G mobile net-
works,’IEEE Commun. Mag., vol. 55, no. 5, pp. 72–79, May 2017.
[4] Y. Xie, Z. Liu, S. Wang, and Y. Wang, ‘‘Service function chaining resource
allocation: A survey,’’ Jul. 2016, arXiv:1608.00095. [Online]. Available:
https://arxiv.org/abs/1608.00095
[5] W. Hahn, B. Gajic, and C. Mannweiler, ‘‘Compound implementation of
chained network functions and virtual resource management performance
evaluation,’’ in Proc. IEEE/IFIP Netw. Oper. Manage. Symp. (NOMS),
Istanbul, Turkey, Apr. 2016, pp. 1301–1304.
[6] Network Functions Virtualisation (NFV); Management and Orchestration,
document ETSI GS NFV-MAN 001, v1.1.1, Dec. 2014.
79230 VOLUME 7, 2019
V. Sciancalepore et al.: Future-Proof Architecture for Management and Orchestration of Multi-Domain NextGen Networks
[7] OFCOM. (Aug. 2018). Communications Market Report 2018. [Online].
Available: http://bit.ly/ofcom_2018
[8] GSMA. (2018). The Mobile Economy—Europe 2018. [Online]. Available:
http://bit.ly/gsma_mobeco
[9] 5G NORMA. (Dec. 2017). Deliverable D2.3—Evaluation Architecture
Design and Socioeconomic Analysis—Final Report. [Online]. Available:
http://bit.ly/5GNORMA-D23
[10] Management and Orchestration; Concepts, Use Cases and Requirements,
document TS 28.530 V15.0.0, 3GPP SA5 WG, 2018.
[11] 5G NORMA. (Dec. 2017). Deliverable D3.3—5G NORMA Network
Architecture—Final Report. [Online]. Available: http://bit.ly/5GNORMA-
D33
[12] Study on Management and Orchestration of Network Slicing for Next
Generation Network, document TR 28.801 V15.1.0, 3GPP SA5 WG,
2018.
[13] L. M. Contreras and D. R. López. (Jan. 2018). A Network Service Provider
Perspective on Network Slicing. IEEE Softwarization. [Online]. Available:
https://sdn.ieee.org/newsletter/january-2018/a-network-service-provider-
perspective-on-network-slicing
[14] G. Yigit. MNOs can Improve Mobile Video Monetisation With New Pric-
ing Models and Intelligent Optimisation. [Online]. Available: http://bit.ly/
video_monetisation
[15] Intel. NFL +Intel True View. [Online]. Available: http://bit.ly/NFL_Intel
[16] I. Al Ridhawi, Y. Kotb, and Y. Al Ridhawi, ‘‘Workflow-net based
service composition using mobile edge nodes,’IEEE Access, vol. 5,
pp. 23719–23735, 2017.
[17] I. Al Ridhawi, M. Aloqaily, Y. Kotb, Y. Al Ridhawi, and Y. Jararweh,
‘‘A collaborative mobile edge computing and user solution for service
composition in 5G systems,’Trans. Emerg. Telecommun. Technol., vol. 29,
no. 11, Jun. 2018.
[18] I. Al Ridhawi, M. Aloqaily, B. Kantarci, Y. Jararweh, and H. T. Mouftah,
‘‘A continuous diversified vehicular cloud service availability framework
for smart cities,’Comput. Netw., vol. 145, pp. 207–218, Nov. 2018.
[19] P. Salter. How Mixed Reality Is RevolutionizingMarketing. [Online]. Avail-
able: http://bit.ly/mixreality_revol
[20] 5G-PPP. (2015). 5G and the Factories of the Future. [Online]. Available:
http://bit.ly/5GPP_FOF
[21] OSM. Open Source MANO. Accessed: Jun. 19, 2019. [Online]. Available:
https://osm.etsi.org/
[22] Network Functions Virtualisation (NFV); Management and Orchestration;
Functional Requirements Specification, document ETSI GS NFV IFA 010,
V2.1.1, 2016.
[23] Network Function Virtualisation (NFV); Management and Orchestra-
tion;Report on Management and Connectivity for Multi-Site Services,
document ETSI GS NFV IFA 022, V3.1.1, 2018.
[24] Network Function Virtualisation (NFV); Management and
Orchestration; Report on Architecture Options to Support Multiple
Administrative Domains, document ETSI GS NFV IFA 028, V3.1.1,
2018.
[25] F. Z. Yousaf, C. Goncalves, L. Moreira-Matias, and X. C. Perez, ‘‘RAVA—
Resource aware VNF agnostic NFV orchestration method for virtual-
ized networks,’’ in Proc. IEEE PIMRC, Valencia, Spain, Sep. 2016,
pp. 1–6.
[26] F. Z. Yousaf and T. Taleb, ‘‘Fine-grained resource-aware virtual network
function management for 5G carrier cloud,’IEEE Netw., vol. 30, no. 2,
pp. 110–115, Mar./Apr. 2016.
[27] Network Functions Virtualisation (NFV); Terminology for Main Concepts
in NFV, document ETSI GS NFV 003, V1.2.1, 2014.
[28] F. Z. Yousaf, V. Sciancalepore, M. Liebsch, and X. Costa-Perez,
‘‘MANOaaS: A multi-tenant NFV MANO for 5G network slices,’’ IEEE
Commun. Mag., vol. 57, no. 5, pp. 103–109, May 2019.
[29] P. Rost, M. Breitbach, H. Roreger, B. Erman, C. Mannweiler, R. Miller,
and I. Viering, ‘‘Customized industrial networks: Network slicing trial at
hamburg seaport,’’ IEEE Wireless Commun., vol. 25, no. 5, pp. 48–55,
Oct. 2018.
[30] G. Garcia-Aviles, M. Gramaglia, P. Serrano, and A. Banchs, ‘‘POSENS:
A practical open source solution for end-to-end network slicing,’IEEE
Wireless Commun., vol. 25, no. 5, pp. 30–37, Oct. 2018.
[31] J. X. Salvat, L. Zanzi, A. Garcia-Saavedra, V. Sciancalepore, and
X. Costa-Perez, ‘‘Overbooking network slices through yield-driven end-
to-end orchestration,’’ in Proc. ACM CoNEXT, Dec. 2018, pp. 353–365.
[32] The Linux Foundation. Open Networking Automation Platform.
Accessed: Jun. 19, 2019. [Online]. Available: https://onap.org/
[33] C. J. Bernardos, B. P. Gerö, M. Di Girolamo, A. Kern, B. Martini, and
I. Vaishnavi, ‘‘5GEx: Realising a Europe-wide multi-domain framework
for software-defined infrastructures,’Trans. Emerg. Telecommun. Tech-
nol., vol. 27, no. 9, pp. 1271–1280, Sep. 2016.
[34] G. Biczok, M. Dramitinos, L. Toka, P. E. Heegaard, and H. Lonsetha-
gen, ‘‘Manufactured by software: SDN-enabled multi-operator composite
services with the 5G exchange,’IEEE Commun. Mag., vol. 55, no. 4,
pp. 80–86, Apr. 2017.
VINCENZO SCIANCALEPORE (S’11–M’15)
received the M.Sc. degree in telecommunications
engineering and the M.Sc. degree in telematics
engineering, in 2011 and 2012, respectively, and
the double Ph.D. degrees, in 2015. He is cur-
rently a 5G Researcher with NEC Laboratories
Europe GmbH, Heidelberg, focusing his activity
on network virtualization and network slicing
challenges. He was a recipient of the National
Award for the best Ph.D. thesis in the areas of
communication technologies (wireless and networking) issued by GTTI,
in 2015.
CHRISTIAN MANNWEILER received the M.Sc.
(Dipl.Wirtsch.Ing.) and Ph.D. (Dr.-Ing.) degrees
from the Technische Universität Kaiserslautern,
Germany, in 2008 and 2014, respectively. Since
2015, he has been a member of the Network
Automation Research Group, Nokia Bell Labs,
where he has been involved in the areas of cog-
nitive network management and SON for 5G sys-
tems. He is the coauthor of numerous articles and
papers on wireless communication technologies
and architectures for future mobile networks. He was involved in several
nationally and EU-funded projects covering the development of cellular and
industrial communication systems.
FAQIR ZARRAR YOUSAF received the Ph.D.
degree from TU Dortmund, Germany. His cur-
rent research interest includes NFV/SDN in the
context of 5G networks. He is currently a
Senior Researcher with NEC Laboratories Europe
GmBH, Germany. He is also an active contrib-
utor to ETSI ISG NFV standards organization,
where he holds Rapporteurship for four work-
items. He has extensive experience in the design,
development, modeling, simulation, and the pro-
totyping of communication systems and protocols for optimizing overall
network performance. He holds one granted patent and 15 filed patents.
His research work has been published in several peer-reviewed journals,
conferences, and book chapters.
PABLO SERRANO (M’09–SM’16) received the
degree in telecommunication engineering and the
Ph.D. degree from the University Carlos III of
Madrid (UC3M), in 2002 and 2006, respectively.
He has been with the Telematics Department,
UC3M, since 2002, where he has been an Asso-
ciate Professor. He has over 80 scientific papers
in peer-reviewed international journals and confer-
ences. He has served as a Guest Editor for Com-
puter Networks. He was on the TPCs of a number
of conferences and workshops including the IEEE INFOCOM and IEEE
WoWMoM.
VOLUME 7, 2019 79231
V. Sciancalepore et al.: Future-Proof Architecture for Management and Orchestration of Multi-Domain NextGen Networks
MARCO GRAMAGLIA received the M.Sc.
degree, in 2009 and the Ph.D. degree in telem-
atics engineering, in 2012. He held a postdoc-
toral research position with Istituto Superiore
Mario Boella, Italy, and the Institute of Elec-
tronics, Computer, and Telecommunications Engi-
neering (IEIIT), National Research Council of
Italy (CNR), Torino, Italy, CNR-IEIIT, Italy, and
IMDEA Networks, Spain. He was involved in EU
projects. He is a Postdoctoral Researcher with the
University Carlos III of Madrid (UC3M).
JULIE BRADFORD received the M.Eng. degree
(Hons.) in electronic engineering in business
management from the University of York, U.K.,
in 2002, and a Postgraduate Certificate (Hons.)
in business management from Bath University,
U.K. She was involved in real wireless of
techno-economic analysis to quantify coverage,
capacity, and user experience in wireless net-
works. Most recently, she has been a part of the
Real Wireless Team providing the commercial and
socio-economic assessment of 5G network architectures within the European
Commission 5G NORMA and 5G MoNArch projects. Previously, she was
a Communications Engineer with QinetiQ, U.K., a Consultant with PA
Consulting, U.K., and a Senior Systems Engineer with Airvana, U.K.
IGNACIO LABRADOR PAVÓN has been in com-
puting and electronics industry for over 30 years,
particularly in the mobile telecommunications
industry developing value-added services for dif-
ferent mobile network operators. During this
time, he was involved in different technical areas
and with different responsibilities, from Software
Developer to a Project Leader. Over the last sev-
eral years, he was a member of the Research
and Innovation Department, Atos, Spain, where he
contributed as the Work Package Leader in different 5G-related research
projects. His main contributions in this area have been in 5G-Norma
(a 5G novel radio multiservice adaptive network architecture) and NGPaaS
(a 5G cloud-based PaaS).
79232 VOLUME 7, 2019
... A prominent difficulty is resource management. Isolation of resources across slices inherently increases network capacity requirements [5], and a dynamic, preemptive and efficient allocation of resources to slices is a key instrument to keep capital expenditure (CAPEX) and operating expenses (OPEX) under control in sliced networks [6]. The rapid fluctuations in service demands, as well as the size and complexity of the slicing ecosystem, make legacy reactive, human-driven approaches to resource management inadequate to the emerging context. ...
... Positive errors yield overprovisioning of x s (t), whereas negative ones have a cost in terms of denied traffic. The former corresponds to the second and third terms 5 in (1), while the latter maps to the monetary fee 6 for SLA violations in (2). The loss function jointly capturing these costs for the extended interval starting at t is t∈T ...
... Under this hypothesis, the second and third terms in (1) reduce to 0 and f 1 x s (t)− i∈Sρ i (t) , respectively. 6 By assuming x s i (t) =ρ i (t), unserviced demands are possible only when x s (t) is insufficient. Then, (2) translates into κs · 1 < i∈Sρi (t) (x s (t)). ...
... The VM interacts with the underlying physical hardware using the hypervisor. Network slicing has been investigated in numerous studies related to network hypervisors, including OpenSlice [179], MobileVisor [180], RadioVisor [181], and HyperFlex [182]. Containers are based on the concept of virtualization at the operating system level and are an alternative to hypervisorbased virtual machines [183]. ...
... The work in [179] has an inter-slice resource broker that allows sharing resources between slices, and reserving resources based on SLAs between multiple InPs. Whereas several studies only focus on multi-domain MANO architectural challenges, they also address the economic benefits of network slicing across multiple domains. ...
Article
The 5th Generation (5G) and beyond networks are expected to offer huge throughputs, connect large number of devices, support low latency and large numbers of business services. To realize this vision, there is a need for a paradigm shift in the way cellular networks are designed, built, and maintained. Network slicing divides the physical network infrastructure into multiple virtual networks to support diverse business services, enterprise applications and use cases. Multiple services and use cases with varying architectures and quality of service requirements on such shared infrastructure complicates the network environment. Moreover, the dynamic and heterogeneous nature of 5G and beyond networks will exacerbate network management and operations complexity. Inspired by the successful application of machine learning tools in solving complex mobile network decision making problems, deep reinforcement learning (Deep RL) methods provide potential solutions to address slice lifecycle management and operation challenges in 5G and beyond networks. This paper aims to bridge the gap between Deep RL and the 5G network slicing research, by presenting a comprehensive survey of their existing research association. First, the basic concepts of Deep RL framework are presented. 5G network slicing and virtualization principles are then discussed. Thirdly, we review challenges in 5G network slicing and the current research efforts to incorporate Deep RL in addressing them. Lastly, we present open research problems and directions for future research.
... In this paper, we examine beyond fifth-generation (B5G) of cellular network NS in a multi-tier multi-tenant multi-slice multidomain (M-TTSD) business model, with SPs and MVNOs being the respective tenants. The multi-domain business model for network slicing enables the aggregation of resources from multiple Infrastructure Providers and also helps in the coverage domain extension of industry players [2]. In an M-TTSD network, to meet the quality of service (QoS) of the respective slice users, the SP would buy virtual resources from more than one MVNO. ...
... In an M-TTSD network, to meet the quality of service (QoS) of the respective slice users, the SP would buy virtual resources from more than one MVNO. Additionally, unlike the single-domain network, in a multi-domain network, the MVNO can buy or lease resources from more than one InP [2]. This is illustrated in Fig. 1. ...
... For better network management, dense networks are divided into several domains. Efficient deployment of service functions knowing that domain-specific information is managed locally, without information exchange is one of the major problems in multidomain networks [10]. Managers and service providers of such networks have internal management strategies for each domain [11 ,12]. ...
... Moreover, a tenant actor in [25] is defined as a business entity that rents and leverages an NSL provided by a network operator. The NO is the actual builder of slices. ...
Article
Full-text available
Identification of ecosystems and Business Models (BM) is an important starting point for new complex system development. The definition of actor (or stakeholder) roles and their interactions (at both business and technical levels), together with target scenarios and use cases, provide essential input information for further system requirement collection and architecture specification. The powerful and flexible Fifth Generation (5G) network slicing technology, which is capable of creating virtually isolated and logically parallel networks, enables a large range of complex services and vertical applications. Although various terminologies and models have been proposed in recent years for BMs in the 5G domain by many studies, projects, standards, and technical specifications from dedicated organizations, they are not always consistent with each other. This study presents an overview and comparison of different BMs for 5G sliced systems, followed by an example on BM definition for a 5G system in a novel ongoing European research project. While a general ecosystem and business model could involve a large range of organizations (including, e.g., regulation and standardization bodies), the scope of this article will be limited to primarily 5G operational BMs, with a focus on those actors or stakeholders who are active and interacting during real-life system operation. Within the project, we perform a selection among some tailored BM configurations, adapted for dedicated slices with different service requirements, aiming to serve Vehicle to Everything (V2X) and Internet of Vehicles (IoV) applications as well as Internet of Things (IoT) for maritime vertical applications. The final part of this article presents our proposed IoT connectivity solutions for various maritime scenarios with and without involving satellite links. Furthermore, we shed light on future challenges and directions for network slicing in beyond 5G systems.
... In [1], a MANagement and Orchestration (MANO) system for physical and virtualized resources is proposed. Exploiting Key Performance Indicators (KPIs) in recent 5G networks and next-generation MANO architectures in multi-domain settings has been studied in [2]. The utilization of mobile (also called multi-access) edge clouds (MEC) close to the road side to improve the reliability and low-latency-aware requires a management platform for virtual network function (VNF) placement and service migration [3]. ...
Chapter
Full-text available
Mobile Edge Computing (MEC) places part of the cloud resources to the edge of the network to increase performance and provide context-aware services. In combination with the expected high performance 5G, many of the limitations for today’s infrastructure could be solved. In the context of the EU 5G-CARMEN project, one of the main challenges is service continuity across organisational and territorial boundaries in a road, motorway and railway settings. The project addresses this challenge by designing and implementing MEC-based services that act as bridges between the different domains, while taking advantage of the high performance and reliability of 5G. Four uses cases have been defined to capture to potential uses of this new paradigm in the mobility domain. In this paper, we present the on going development of service continuity mechanisms within the 5G-CARMEN project.
... In [1], a MANagement and Orchestration (MANO) system for physical and virtualized resources is proposed. Exploiting Key Performance Indicators (KPIs) in recent 5G networks and next-generation MANO architectures in multi-domain settings has been studied in [2]. The utilization of mobile (also called multi-access) edge clouds (MEC) close to the road side to improve the reliability and low-latency-aware requires a management platform for virtual network function (VNF) placement and service migration [3]. ...
Conference Paper
Full-text available
Mobile Edge Computing (MEC) places part of the cloud resources to the edge of the network to increase performance and provide context-aware services. In combination with the expected high performance 5G, many of the limitations for today's infrastructure could be solved. In the context of the EU 5G-CARMEN project, one of the main challenges is service continuity across organisational and territorial boundaries in a road, motorway and railway settings. The project addresses this challenge by designing and implementing MEC-based services that act as bridges between the different domains, while taking advantage of the high performance and reliability of 5G. Four uses cases have been defined to capture to potential uses of this new paradigm in the mobility domain. In this paper, we present the on going development of service continuity mechanisms within the 5G-CARMEN project.
Chapter
One of the core elements for the upcoming generation of wireless cellular networks is the availability of network service access continuity in addition to high-speed internet and low latency. The forthcoming fifth generation (5G) greatly improves users’ demand in terms of faster download rates, exceptional system availability, superb end to end coverage with exceptionally low latency and ultra reliability. One of the solutions to provide end to end low latency is the utilization of Mobile Edge Computing (MEC) in the network. MEC provides cloud advantages to users by setting up a small cloud server in the edge node (i.e. close to the end-user), which decreases the amount of latency in network connections, in this regard, service migration has required as users migrate to the new location. Optimal migration decisions are challenging because they depend on the cloud environment, or edge nodes belong to different orchestrators, and security issues in the migration process must also be resolved in order to prevent unreliable requests. This study provides different approaches to address these challenges by identifying the security implications of migration methods based on the blockchain integration.
Article
Full-text available
Network function virtualization (NFV) technology achieves flexible service deployment by replacing the middleboxes with virtual network functions (VNFs). In NFV, a set of VNFs are chained in a given order, called service function chain (SFC), and accordingly, data flow is steered to traverse all the VNFs in order to offer a service. With a large number of network devices and end users being connected into Internet, there is a growing demand for large‐scale multi‐domain networks to dynamically deploy the SFC across multiple network domains, in order to support efficient service provisioning. To this end, in this paper, we first investigate the state of the art of multi‐domain SFC deployment, and then propose an intelligent multi‐domain SFC deployment (IMSD) architecture by leveraging software‐defined networking (SDN), NFV, and deep learning technologies. Furthermore, we discuss the potential challenges to realize the IMSD and provide some promising solutions. We first survey the state of the art of multi‐domain SFC deployment and then integrate software‐defined networking (SDN), NFV, and deep learning to propose an intelligent multi‐domain SFC deployment (IMSD) architecture. Furthermore, we discuss the potential challenges to realize the IMSD and provide some promising solutions.
Article
Network Function Virtualization enables network slicing as a novel paradigm for service provisioning. With network slicing, Virtual Network Functions (VNFs) can be instantiated at different locations of the infrastructure, choosing their optimal placement based on parameters such as the requirements of the service or the resources available. One limitation of state-of-the-art technology for network slicing is the inability to re-evaluate orchestration decisions once the slice has been deployed, in case of changing service demands or network conditions. In this paper, we present ACHO, a novel software framework that enables seamless re-orchestration of VNFs of any kind, including RAN and Core. With ACHO, VNFs and resources can be easily re-assigned to match, e.g., varying user demands or changes in the nodes’ load. ACHO uses lightweight mechanisms, such as splitting the engine of a VNF from the data it requires to perform its operation, in such a way that, when re-allocating a VNF, only the data is moved (a new engine is instantiated in the new location). We demonstrate the use of ACHO in a small scale testbed, showing that (i) the proposed re-orchestration is feasible, (ii) it results much faster than existing alternatives (especially for relocation), and (iii) the framework can be readily applied to existing VNFs after minimal changes to their implementation.
Article
Full-text available
The dramatic densification of connected mobile devices and the expected use cases from the vertical industry demand an innovative network design that meets upcoming stringent requirements. The adoption and harmonized integration of novel concepts, such as network functions virtualization and network programmability, enables the system to master the high expectation -- from the fifth generation communication network in support of flexibility -- to provide tailored and mutually isolated network slices, high performance, agility, and automation. This effectively involves a number of technical challenges for managing and orchestrating physical and virtualized slice resources by means of an advanced management and orchestration (MANO) system. This article sheds light on potential benefits and implementation aspects when the MANO framework is abstracted into customized and distributed MANO instances, thereby empowering the MANO-as-a-service (MANOaaS) paradigm. In particular, such distributed instances are provided to different network tenants for a greater level of control on requested network slice(s). The notion of management level agreements in the context of MANOaaS is introduced as well as differentiated per tenant while being embedded into the proposed architecture. We also position the proposed MANOaaS concept and associated extensions to the MANO reference architecture from the viewpoint of standardization bodies and ongoing open source projects.
Conference Paper
Full-text available
Network slicing allows mobile operators to offer, via proper abstractions, mobile infrastructure (radio, networking, computing) to vertical sectors traditionally alien to the telco industry (e.g., automotive, health, construction). Owning to similar business nature, in this paper we adopt yield management models successful in other sectors (e.g. airlines, hotels, etc.) and so we explore the concept of slice overbooking to maximize the revenue of mobile operators. The main contribution of this paper is threefold. First, we design a hierarchical control plane to manage the orchestration of slices end-to-end, including radio access, transport network, and distributed computing infrastructure. Second, we cast the orchestration problem as a stochastic yield management problem and propose two algorithms to solve it: an optimal Benders decomposition method and a suboptimal heuristic that expedites solutions. Third, we implement an experimental proof-of-concept and assess our approach both experimentally and via simulations with topologies from three real operators and a wide set of realistic scenarios. Our performance evaluation shows that slice overbooking can provide up to 3x revenue gains in realistic scenarios with minimal footprint on service-level agreements (SLAs).
Article
Full-text available
The intelligent and connected transportation system (ICTS) is a significant and mandatory component of the smart city architecture. Multimedia content sharing, vehicle power management, and road navigation are all examples of ICTS services. As smart cities continue to deploy different technologies to improve the performance and diversity of vehicular cloud services, one of the main issues that prevails is efficient and reliable service discovery and selection for smart vehicles. Furthermore, cloud service providers (SPs) are limited to the availability, variety and quality of services made available to vehicular cloud subscribers. Smart vehicles rely on a number of SPs to acquire the required services while moving. It therefore becomes challenging for vehicular cloud subscribers to acquire services that meet their quality of experience (QoE) preferences. This paper introduces a new service provision scheme to provide continuous availability of diversified cloud services targeting vehicular cloud users through a cluster-based trusted third party (TTP) framework. TTPs act as cloud service mediators between cloud service subscribers and providers. Vehicles that are considered to have similar patterns of movement and service acquisition characteristics are grouped into service-specific clusters. TTPs communicate with service providers and cluster heads to negotiate for services with high QoE characteristics. A location prediction method is adopted to determine a vehicle's future location and allow services to be negotiated for before the vehicle's arrival. We provide simulation results to show that our approach can adequately discover and deliver cloud services with increased QoE results, minimal overhead burden and reduced end-to-end latency.
Article
Full-text available
Mobile edge computing (MEC) is an emergent technology that has revolutionized traditional cloud service solutions. Mobile edge computing extends cloud computing by providing processing, storage, and networking capabilities at the edge of the mobile network. Delay‐sensitive and context‐aware applications are able to execute within close proximity of mobile users. Additionally, today's cloud services are not tailored to user specifications, but rather diversified toward a group of users. To guarantee delivery of user‐specific services in 5G networks, service composition techniques should be incorporated. This article envisions a real‐time, context‐aware, service‐composition collaborative framework that lies at the edge of the network, comprising MEC and user devices for fast composite service delivery. The proposed solution decomposes cloud data into a set of files and services, which are then replicated to MEC nodes. Frequently requested files and services are further cached onto user mobile devices for faster access. Both MEC nodes and mobile users advertise their services onto the collaborative edge/user space, where services are delivered either composite or unrendered according to users' requests. Service composition is achieved through a learning‐based workflow‐net approach that relies on previous composition results to build service composition models to be used for new compositions. The presented solution provides guaranteed and fast delivery of the requested cloud composite services to end users while sustaining QoS requirements and load balancing among edge and mobile nodes.
Article
Full-text available
Content delivery through cloud networks has gained popularity due to the cloud’s ability to provide on-demand services. However, composite services such as customized multimedia content introduce both delays and resource limitations if traditional cloud solutions are used. With recent advances in Mobile Edge Computing (MEC), customized media delivery can be achieved through compositions of Service Specific Overlays (SSOs). This paper presents a workflow-net based mechanism for mobile edge node cooperation in fog-cloud networks to form guaranteed SSOs. The proposed solution uses a mathematical cooperation operator to turn the SSO composition problem expressed as workflow-nets into algebraic representations. In turn, the minimal cost cooperative path from the workflow-net is determined such that it guarantees the delivery of the requested composite media services to clients. Experimental results show that the composition process can be adequately established and carried out in a timely manner.
Article
Full-text available
Foreseen 5G verticals hold the promise of being true value-added services, hence bringing significant income to their respective providers. However, the nature of these verticals are very demanding in terms of both economic and technical requirements, such as multi-operator cooperation, end-to-end quality assurance, and the unified orchestration of network and cloud resources. Existing systems fall short of satisfying these requirements, but emerging network softwarization and resource virtualization technologies, such as SDN and NFV, show promise for being key enablers in this context. In this article, we introduce the 5G Exchange (5GEx) concept that builds on SDN and NFV, and facilitates the provisioning of multi-operator 5G services by means of inter-operator management and orchestration of virtualized network, compute, and storage resources. We present potential 5GEx use cases, conceptual architecture, and value proposition. We also outline open research questions on how to exchange information in such a coopetitive environment, and provide an outlook on the impact of 5GEx on a network service provider's business and operation.
Article
Full-text available
We argue for network slicing as an efficient solution that addresses the diverse requirements of 5G mobile networks, thus providing the necessary flexibility and scalability associated with future network implementations. We elaborate on the challenges that emerge when we design 5G networks based on network slicing. We focus on the architectural aspects associated with the coexistence of dedicated as well as shared slices in the network. In particular, we analyze the realization options of a flexible radio access network with focus on network slicing and their impact on the design of 5G mobile networks. In addition to the technical study, this paper provides an investigation of the revenue potential of network slicing, where the applications that originate from such concept and the profit capabilities from the network operator’s perspective are put forward.
Article
Network slicing represents a new paradigm to operate mobile networks. With network slicing, the underlying infrastructure is "sliced" into logically separate networks that can be customized to the specific needs of their tenant. Hands-on experiments on this technology are essential to understand its benefits and limits, and to validate the design and deployment choices. While some network slicing prototypes have been built for RANs, leveraging on the wide availability of radio hardware and open source software, there is currently no open source suite for end-to-end network slicing available to the research community. In this article we fill this gap by developing an end-to-end network slicing protocol stack, POSENS, which relies on a slice-aware shared RAN solution. We design the required algorithms and protocols, and provide a full implementation leveraging on state-of-the-art software components. We validate the effectiveness of POSENS in achieving tenant isolation and network slices customization, showing that no price in performance is paid toward this end. We believe that our tool will prove very useful to researchers and practitioners working on this novel architectural paradigm.
Article
Driven by a massive surge in digitization and customization, so-called vertical industries are expected to be a major beneficiary of the 5G of mobile networks. The use cases of such vertical industries define qualitative and quantitative requirements unprecedented in the history of mobile network development. Autonomous vehicles, traffic light control, video surveillance, and IIoT, to only name a few, introduce challenging requirements regarding both conventional performance metrics, such as throughput and coverage, as well as formerly rather subordinate system metrics, such as deterministic latency, ultra-high reliability and resilience, high number of devices, multi-tenant networks, and demanding security mechanisms. Nokia, Deutsche Telekom, and Hamburg Port Authority have deployed a largescale 5G trial testbed in the Hamburg port area. The testbed proves in a real, large-scale industrial environment that basic features of network slicing, namely slice isolation, flexible slice customization, and multitenancy, are already technically feasible. Three exemplary communication services have been selected and are demonstrated in the testbed. Multi-connectivity is implemented as a key component to achieve high reliability throughout the testbed area. The testbed shows that all network domains must be involved in the setup of network slices, that is, user terminals, radio access, core network, and enterprise networks, in order to efficiently operate and manage network slices. Therefore, the discussed life cycle management is key for the interaction between mobile service provider and tenants of the network.