ONE: Online Energy-efficient User Association, VNF Placement and Traffic Routing in 6G HetNets

Abstract

In this paper, we study the joint problem of real-time user association, traffic routing and Virtual Network Function (VNF) placement towards maximizing the mobile network's energy efficiency and user acceptance ratio. The studied problem is formulated as a Mixed Integer Linear Program (MILP), under various computational, capacity, and flow conservation constraints, while also meeting the Service Function Chain (SFC), throughput and delay requirements of each service request. In addition to the optimal solution of the MILP problem, an efficient low computational complexity heuristic (named as ONE) is also developed and evaluated via simulations. The proposed solutions are shown to significantly outperform the State-of-Art (SoA) in terms of energy efficiency, with ONE achieving up to 89% of the optimal energy efficiency value with up to 90% lower computational time even under heavy user traffic load scenarios.

Full text

ONE: Online Energy-efficient User Association,
VNF Placement and Traffic Routing in 6G HetNets
A. Mesodiakaki⋆,†, M. Gatzianas⋆,†, G. Kalfas⋆,†, C. Vagionas⋆,†, R. Maximidis⋆,†and N. Pleros⋆,†
⋆Department of Informatics, Aristotle University of Thessaloniki, Thessaloniki, Greece
†Center for Interdisciplinary Research and Innovation, Thessaloniki, Greece
Emails: {amesodia, mgkatzia, gkalfas, chvagion, maximidis, npleros}@csd.auth.gr
Abstract—In this paper, we study the joint problem of real-time
user association, traffic routing and Virtual Network Function
(VNF) placement towards maximizing the mobile network’s
energy efficiency and user acceptance ratio. The studied problem
is formulated as a Mixed Integer Linear Program (MILP),
under various computational, capacity, and flow conservation
constraints, while also meeting the Service Function Chain (SFC),
throughput and delay requirements of each service request. In
addition to the optimal solution of the MILP problem, an efficient
low computational complexity heuristic (named as ONE) is also
developed and evaluated via simulations. The proposed solutions
are shown to significantly outperform the State-of-Art (SoA)
in terms of energy efficiency, with ONE achieving up to 89%
of the optimal energy efficiency value with up to 90% lower
computational time even under heavy user traffic load scenarios.
Index Terms—6G, Green Networks, Heuristics, Multi-Access
Edge Computing (MEC), Resource Allocation, Service Function
Chain (SFC), Virtual Network Function (VNF).
I. INTRODUCTION
To support the ever-increasing traffic demands, next genera-
tion (i.e., 6G), networks will be highly heterogeneous in terms
of employed technologies, operating frequencies, and types of
resources (network, compute and storage), thus comprising
a “network of networks” [1]. The Radio Access Networks
(RAN) architecture is also transformed through disaggregation
and virtualization of RAN functions to support service het-
erogeneity, coordination of multi-Radio Access Technologies
(multi-RATs), and on-demand service deployment.
In this setting, service deployment and optimal operation
becomes challenging. This is due to the fact that each Service
Request (SR) corresponds to a Service Function Chain (SFC)
consisting of an ordered set of Virtual Network Functions
(VNFs) that should be efficiently deployed in the compu-
tational nodes of the network, ensuring that the VNFs are
deployed in the exact order specified by the SFC and the
capacity constraints of the nodes and involved links are not
violated [2]. These VNFs can be also deployed in different
forms, i.e., in Virtual Machines (VMs), containers or uniker-
nels, and different network locations, i.e., in Multi-Access
Edge Computing (MEC) units for reduced latency, in data
centers for reduced cost or in intermediate locations, thus
presenting a sophisticated delay vs. cost trade-off to exploit.
The 6G transport network interconnecting the computation
and communication nodes will consist of front/mid/back-
haul (X-haul) links of different technologies and capabilities
(i.e., both fiber and wireless links), calling for joint considera-
tion of the Access Network (AN) and transport in the modeling
process. For the wireless X-haul links, an attractive solution
lies in the use of millimeter wave (mmWave) frequencies,
due to their wide spectrum bands and high antenna gains to
compensate for the increased path loss in these bands [3].
In the AN, a combination of gNodeBs (gNBs) and a dense
overlay of Small Cells (SCs) employing 5G-New Radio (5G-
NR) frequencies, including mmWave, is anticipated.
To that end, real-time resource allocation becomes chal-
lenging due to the high network and resource heterogeneity,
resulting in a large number of strongly coupled decision
variables. In this context, efficient online resource allocation
strategies should: i) jointly consider all different types of
resources, i.e., communication, computational and storage, and
technologies, e.g., 5G-NR, mmWave, fiber, as well as their
constraints, ii) take into account the End-to-End (E2E) network
path from the traffic’s source to the destined User Equipment
(UE) to guarantee E2E optimality, iii) induce low computa-
tional complexity to enable real-time decisions, while meeting
the E2E delay target, and iv) achieve high energy efficiency.
Developing energy-efficient solutions serves a twofold goal: a)
reducing the Operational Expenditure (OPEX) of the involved
stakeholders (e.g., Mobile Network Operators, Infrastructure
Providers etc.), and b) leading to environmentally friendly
solutions by limiting the associated carbon footprint. In this
context, energy-efficient online resource allocation solutions
are needed that will jointly solve the user association, traffic
routing and VNF placement problems, while ensuring SFC
chaining and guaranteeing the QoS of the SRs in 6G networks.
A. Related work and Contribution
The joint study of online VNF placement (including SFC
chaining) and routing has recently attracted significant re-
search interest. However, the majority of the works in this field
focus either on cloud nodes located at the core network [2],
[4]–[8] or include MEC nodes [9] interconnected with wired
links, thus overlooking the wireless segments of 6G networks.
Given that the formulated problems are computationally in-
tractable [2], [4]–[9], they also propose heuristic algorithms
and numerically evaluate their performance. The main differ-
ences among these works are in the targeted objective and
considered constraints i.e., minimization of implementation
cost (expressed by bandwidth, CPU and flow table resource

consumption as well as delay, packet loss and jitter) [2],
profit maximization [4], minimization of traffic load balancing
ratio subject to delay, bandwidth and capacity constraints [5],
minimization of operating cost and network congestion [6],
minimization of link utilization subject to delay constraints
[9] and total network power consumption minimization [7],
[8]. However, these works do not study mobile networks, thus
overlooking the AN part and any wireless X-haul solutions,
which are expected to play a key role in 6G networks.
Our paper substantially extends the State-of-the-Art (SoA)
by jointly studying user association, traffic routing and VNF
placement in a Heterogeneous Network (HetNet), consisting
of gNBs and SCs in a distributed computing (i.e., MEC,
fog, cloud) network infrastructure, and incorporating AN and
wireless X-haul links in the problem formulation via their
respective power and capacity constraints. Detailed channel
and power models accounting for the inherent wireless channel
fluctuations are also employed. Due to the high complexity
of the developed optimal solution, a heuristic algorithm is
also proposed and numerically evaluated. The challenging
aspect of online sequential decision making is captured in the
problem formulation and developed heuristics, while taking
into account the duration of each SR, thus complementing
previous works focusing exclusively on offline solutions [3],
[10]. Our results demonstrate that the proposed heuristic can
achieve up to 89% of the optimal performance with up to 90%
lower complexity, while significantly outperforming the SoA.
The rest of the paper is structured as follows: Section II
presents the system model and problem statement, while Sec-
tion III describes the proposed heuristic (ONE). The proposed
solutions are compared to other SoA algorithms in Section IV,
while Section V concludes the paper. We denote sets as V,
indicator functions as I[·]and equality by definition as △
=.
II. SY ST EM M OD EL A ND P ROB LE M STATE ME NT
We extend the model of [3] by incorporating the tempo-
ral variation aspect of Service Requests (SRs) along with
constraints due to previously allocated resources for still
running services. The mobile network is modeled as a directed
graph G(V,E), with Vthe set of non-UE nodes and Ethe
set of links among them, as shown in Fig. 1. Nodes in V
constitute gNBs/SCs (jointly referred to as Base Stations,
BSs), switches/routers and middlebox devices. Each link
e= (u, v)∈ E has capacity ceand imposes delay (sum
of transmission, propagation and processing delay) δeto all
transit packets. Let Efi ,Ewl be the sets of wired/fiber and
wireless links, respectively, with Vsw ⊆ V the set of nodes
with at least one incident fiber link. We denote with h(e) = u,
t(e) = vthe head, tail, respectively, of directed link (u, v). Let
Jbe the set of UEs, where UE j∈ J can connect to a BS in
set A(j)⊆ V. We focus on downlink and define sets EAN
△
=
(a, j) : a∈ A(j), j ∈ J and ˜
E△
=E ∪EAN . We also assume
that each BS a∈ A △
=∪j∈J A(j)can allocate at most ¯
N(RB)
a
Resource Blocks (RBs) to UEs.
Let Vc⊆ V be the set of nodes with computational
resources for hosting VNFs, where node y∈ Vchas an amount
Aggreg.
layer 1 Aggreg.
layer 2
: gNB, SC
: fog resources
: MEC resources : cloud resources : switch
: wireless X-haul link : access link : VNFs
: fiber link
: network node
Fig. 1. Heterogeneous communication and computation mobile network
infrastructure capable of supporting VNF deployment.
cyof computational resources (measured in GFLOPS). Let F
be the set of available VNFs, where multiple instances of
a given VNF in the same or different nodes (depending on
network traffic) are allowed. Each VNF f∈ F is described as
f△
= (If, πf, wf, Df), where Ifidentifies VNF functionality,
πf>0is the data processing capacity of the VNF (in Mbps),
wf>0is the amount of CPU resources (in GFLOPS) required
for the VNF’s operation, and Df>0is the delay incurred on
an individual data packet when processed by the VNF.
We define an SFC as an ordered tuple ρ△
=⟨f(ρ)
1, . . . , f(ρ)
Nρ⟩
of VNFs and denote with Cthe set of all SFCs. SFC ρ∈ C
is served only if its traffic passes through the VNFs in ρin
exactly the specified order. We write f⇝ρto denote that
ρcontains VNF fand define Rρ
△
={f∈ F :f⇝ρ}.ρ
is equivalently described by a directed graph G(ρ)(V(ρ),E(ρ)),
where V(ρ)contains the VNFs in Rρas virtual nodes and
E(ρ)describes the relative order of the VNFs (e.g., Fig. 2 of
[9]). For any virtual edge e′∈ E(ρ), we denote with h(e′),
t(e′)∈ F the VNFs at the head, tail of e′.
At a random time τj, each UE jrequests an SR qj
△
=
(sqj, rqj, δqj, ρqj, ζqj), with sqj∈ V the “source” node of qj
(“destination” node of qjis UE j), rqj>0the E2E required
throughput, δqj>0the E2E latency target, ρqj∈ C the
requested SFC and ζqj>0the amount of time for which
the service must be offered to the UE (i.e., neglecting, for
simplicity, the service setup time, qjmust be served in the
time interval [τjτj+ζqj]). We allow for arbitrary continuous
distributions of the SR inter-arrival intervals and sharing of
VNF instances among multiple SFCs. For each UE j, we
denote with Ojthe temporal order in which j’s request arrives
(i.e., Oj= 1 indicates jarrives first) and with N(RB)
a,j the
number of RBs needed to achieve rate rqjon link (a, j).
Under frequency-flat slow fading [3], we consider the simple
case where constant power p(RB)
ais assigned to each RB. Let
N(t)△
={j∈ J :t−ζqj< τj< t}be the set of SRs
that arrived before tand are still served at t. Defining ˙τ(k)
as the k-th earliest SR epoch (i.e., ˙τ(k) = τjiff Oj=k)
and ˙γ(k)△
={j:Oj=k}as the respective UE, it follows
that all SRs in N( ˙τ(k)) are still served at ˙τ(k). We denote
Nk
△
=N( ˙τ(k)) and impose the following rule of operation:

Rule 1: The computational and network resources allocated
on specific network nodes and paths towards serving UE j’s
SR arriving at τjremain “reserved” for this UE until the
service expires (at τj+ζqj), i.e., no “migration” of allocated
VNFs and resources is allowed and resources are released only
when all UE services using them have expired.
Rule 1, motivated by the assumption that active services
can tolerate no disruption due to migration-induced downtime
during SR reconfiguration, is the crucial difference from
[3] since, in its absence, each new SR would trigger a re-
configuration of all currently running SRs and lead to the
formulation, optimal solution and heuristics of [3]. We can
now provide our problem statement: “for randomly generated
UE SRs, we seek to jointly determine the location of deployed
VNFs and E2E routing paths to serve the SRs so that total
network power consumption is minimized under Rule 1”.
A. Power consumption model and problem formulation
Unless otherwise stated, we use indices as follows: j∈ J ,
a∈ A,y∈ Vc,˜y∈ Vc∪ J ,f∈ F,u, v, w ∈ V,m, n ∈ Vsw.
Each SR instance kgenerates a new optimization problem
with decision variables xj,a
△
=I[UE jconnects to BS a],
ϕ˜y,f,qj
△
=I[VNF fin SFC ρqjis deployed on node ˜y]and
θe′,qj
e=θe′,qj
u,v =I[ebelongs to the physical path in Gonto
which virtual link e′∈ E(ρqj)is mapped for SFC ρqj]for
physical link e= (u, v)∈ E. Also, Nf, ˜yis the number of VNF
finstances deployed on ˜y. We denote with ˆxj,a ,ˆ
ϕ˜y,f,qj,ˆ
θe′,qj
e
the corresponding variables for the previous SR instance k−1
(for k= 1, it holds ˆxj,a =ˆ
ϕ˜y,f,qj=ˆ
θe′,qj
e= 0). Note that,
when solving for SR k, variables ˆxj,a ,ˆ
ϕ˜y,f,qj,ˆ
θe′,qj
ebecome
parameters and not decision variables. Although all variables
implicitly depend on k, we omit this fact for cleaner notation.
Denoting with ξythe Boolean variable indicating whether
node yactually runs deployed VNFs, the power consumed
by node y’s CPU is P(CP U)
y=P(CP U,i)
yξy+ (P(CP U,m)
y−
P(CP U,i)
y)·Uy, with P(CP U,m)
y,P(CP U,i)
ythe maximum and
idle CPU power and Uy
△
=Pf∈F
Nf,y wf
cvthe CPU load factor
[8]. For a fiber link e= (n, m)∈ Efi and switch n∈ Vsw , we
define decision variables ψn
△
=I[switch nhas active incident
fiber links], zn,m
△
=I[eactually carries traffic, in either link
direction]and wn,m
△
=I[ecarries traffic from nto m], so that
it holds zm,n ≥wm,n,zm,n ≥wn,m . The conditions in (1),
for a sufficiently large constant C1>0, should also hold.
X
e′∈E(ρq˙γ(k))
θe′,q ˙γ(k)
e+X
j∈Nk
X
e′∈E(ρqj)
ˆ
θe′,qj
e≤C1we,∀e∈ Efi,
X
e∈Efi :h(e)=n
we+X
e∈Efi :t(e)=n
we≤C1ψn,∀n∈ Vsw.(1)
Switch nconsumes power P(sw)
n=P(sw)
idle ψn+Pport·
Pm∈Vsw:(n,m)∈Ef i zn,m, where P(sw)
idle is the switch’s idle
power and the sum accounts for the switch’s active fiber links,
with Pport being the consumed power of each active port [8].
Following the power model for mmWave links in [3],
link e∈ Ewl consumes power P(mmW )
e=N(mmW )
RF ·
χeP(mmW,i)
e+∆(mmW )
eF(ℓe), where N(mmW )
RF is the
number of RF chains, P(mmW,i)
eis the transmitter idle
power, χe
△
=I[eactually carries traffic],ℓeis the
load-dependent link utilization, ∆(mmW )
eis a slope pa-
rameter depending on the link’s transmitter electronics,
and F(·)is a piecewise-linear function capturing the
nonlinear throughput/power relation. For the AN, BS a
spends power P(BS)
a=N(BS)
RF µaP(BS,i)
a+ ∆(BS)
ap(RB)
a·
x˙γ(i),aN(RB)
a, ˙γ(i)+Pj∈Niˆxj,a N(RB)
a,j , with µa
△
=I[at least
one UE attaches to a]and N(BS)
RF ,P(BS,i)
a,∆(BS)
ahaving the
same semantics as for the mmWave links. We apply the trick of
[11] and employ auxiliary Boolean variables σe, for e∈ Ewl,
and µa, νa, for a∈ A, to convert the activation constraints
into the form of (2), for a sufficiently large C2>0.
χe+C2σe≥1,∀e∈ Ewl, µa+C1νa≥1,∀a,
1−C2χe≤X
e′∈E(ρq˙γ(k))
θe′,q ˙γ(k)
e+X
j∈Nk
X
e′∈E(ρqj)
ˆ
θe′,qj
e
≤C2(1 −σe),∀e∈ Ewl,
(2)
1−C1νa≤x˙γ(k),a +X
j∈Nk
ˆxj,a ≤C2(1 −νa),∀a.
Apart from basic self-consistency conditions (omitted due to
space limits), we also introduce the following constraints:
X
y∈Vc
ϕy,f,q ˙γ(k)= 1,∀f∈ Rq˙γ(k),
X
a∈A( ˙γ(k))
x˙γ(k),a = 1, x ˙γ(k),b = 0,∀b∈ A( ˙γ(k)) ,
(3)
ϕy,f,q ˙γ(k)rq˙γ(k)+X
j∈Nk:f⇝ρqj
ˆ
ϕy,f,qjrqj≤Nf ,yπf,∀f , ∀y,
X
f∈F
Nf,y wf≤cy,X
f∈F
Nf,y ≤C1ξy,∀y,
(4)
X
e′∈E(ρq˙γ(k))
θe′,q ˙γ(k)
erq˙γ(k)+X
j∈Nk, e′∈E(ρqj)
ˆ
θe′,qj
erqj≤ce,∀e,
x˙γ(k),aN(RB)
a, ˙γ(k)+X
j∈Nk
ˆxj,aN(RB )
a,j ≤¯
N(RB)
a,∀a,
(5)
X
v:(u,v)∈˜
E
θe′,qj
u,v −X
w:(w,u)∈˜
E
θe′,qj
w,u =ϕu,h(e′),qj−ϕu,t(e′),qj,
j= ˙γ(k),∀e′∈ E(ρqj),∀u∈˜
V,
(6)
X
y∈Vc
X
f∈Rq˙γ(k)
ϕy,f,q ˙γ(k)τf+X
e∈E X
e′∈E(ρq˙γ(k))
θe′,q ˙γ(k)
eδe
+X
a∈A( ˙γ(k))
x˙γ(k),aδ(a, ˙γ(k)) ≤δq˙γ(k),
(7)
where (3) ensures that the VNFs of the arriving SR are
deployed and the arriving UE attaches to exactly one BS.
Eq. (4) ensures that the deployed VNF instances on a node
meet the data processing requirements for the current traffic
(including previous SRs still running) without exceeding the

available CPU resources, while computational nodes are de-
ployed only when needed. Eq. (5) captures the link and BS
capacity constraints, while (6) is a flow conservation/routing
condition which forces the packets of the new SR’s SFC to
pass through the corresponding VNFs in the correct order.
Eq. (7) captures the E2E delay requirement for the new SR.
Hence, for each SR instance k= 1,...,|J |, the net-
work’s total power consumption is Ptotal
△
=Pn∈Vsw P(sw)
n+
Py∈VcP(CP U )
y+Pe∈Ewl P(mmW )
e+Pa∈A P(BS)
aand we
formulate our problem as a sequence of MILPs.
For each instance k:minimize Ptotal,
s.t. (1)–(7) ,(8)
Due to the high complexity of solving (8), we next propose
a low-complexity heuristic algorithm and use the optimal
solution of (8) only as a benchmark tool for evaluating the
heuristic’s performance.
III. ONE: PROP OS ED ONLINE HEURISTIC FOR
ENERGY-EFFI CI EN T USER ASSOCIATION,TR AFFI C ROU TI NG
AN D VNF PL ACE ME NT
An ONline heuristic, named as ONE, is proposed, which
performs jointly the user association, the traffic routing and
the VNF placement with the aim of maximizing the network
energy efficiency and the UE acceptance ratio. As shown in
Fig. 2, ONE follows a continuous process which consists of
2 steps. In the 1st step, user association and traffic routing
is decided, while in the 2nd, the VNFs of the UE’s SFC are
placed in the exact order defined in the SFC.
Upon a new UE SR arrival, ONE constructs a weighted
graph with all feasible paths from the source of the requested
SFC to the destination (i.e., the UE), setting as weight their
respective power consumption. These links include: i) any X-
haul transport links (both wireless and fiber ones) from the
source of the traffic to the serving BS of the UE and ii) the
AN link, from the serving BS to the UE. ONE calculates
the Kshortest-weighted paths and selects the one with the
lowest power consumption, as long as there is no delay and
capacity constraint violation, while taking into account the
already associated UEs. Otherwise, proceeds with the next
path until either a path with no constraint violation is found
or no other paths remain to examine. In the second case, ONE
blocks the user and checks for new UE SR arrivals.
Once a path satisfying all constraints is determined for
the new SR, ONE moves to the 2nd step, where the VNF
placement takes place. Specifically, for each VNF in the order
specified by the SR’s SFC, ONE sorts the computational nodes
by Ω, i.e., the normalized sum of closeness (node centrality
metric), maximum computational capacity and CPU load of
the node, in descending order. The CPU load is set to: a)
1 (high priority), when there is already an instance of this
VNF in the examined node, and no new instance initiation is
needed to host the VNF under study, b) 0.1 (low priority),
when there are enough computational resources to host the
VNF in the studied node, but a new VNF instance is needed,
and c) 0 otherwise (no priority). Then, ONE selects the
Start Construct a weighted graph including:
Feasible AN links, X-haul transport links (both wireless and fiber ones)
Weights: Link power consumption taking into account the already associated UEs
Calculate the
K
shortest-weighted paths
First step
Move to the next
path
Satisfies
delay &
capacity
constraints?
Other
VNFs?
YES NO
Put UE to unsatisfied users YES
NO
Other
paths
available?
User association & traffic routing
We select the (next) node with the highest Ω to place the
VNF ensuring correct VNF ordering
YES
YES
NO
Change
selected path
For each VNF of the user's SR we sort the computing
nodes of the path by parameter Ω that considers:
•Node centrality-closeness
•Maximum computational capacity
•CPU load
Other nodes
available in
the path?
NO
Comp.
capacity
constraint
satisfied?
Place the
VNF
Second step
VNF placement
Select the (next) shortest-weighted pat h
NO
YES
NO
YES
NO NO
UE
arrival
?
Update the network
state
Fig. 2. Operation flowchart of ONE: Proposed Online Heuristic for Energy-
efficient User association, Traffic Routing and VNF placement.
first computational node of the sorted set that has sufficient
computational resources to host the VNF. If there is no such
node, ONE goes back to step 1 and the next path (out of the K)
is studied. For the new path, the same procedure takes place
until either all VNFs of the SR’s SFC are placed or there is no
other path to examine and the user is being blocked. In case
all VNFs in the SFC are placed, ONE updates the network
state and checks for new UE SRs. Upon a new SR arrival, the
aforementioned procedure is repeated.
IV. PER FO RM AN CE E VALUATION
We consider 2 SC clusters overlaying a gNB sector of 500 m
radius and assume hotspot UE traffic distribution [3], as shown
in Fig. 3. In each cluster, 4 SCs are uniformly placed within
100 m from the cluster centers, with the minimum allowable
distances among BSs from [3]. We assume that the gNB and
a randomly selected SC in each cluster are fiber-connected to
the 1st and 2nd Aggregation Layers (ALs), while mmWave X-
haul links also exist among BSs separated by a distance ≤200
m. In each aggregation layer, 4 randomly selected nodes have
fiber connectivity among them and with the fiber-connected
BSs so that no node is disconnected. The optimal solution is
computed by IBM CPLEX, whereas the heuristics (both the
proposed and the SoA ones) are developed in MATLAB.
Without loss of generality, 5 SFCs built from the following
VNFs are considered: NAT (Network Address Translation),
FW (Firewall), TM (Traffic Monitor), WOC (WAN Opti-
mization Controller), IDPS (Intrusion Detection/Prevention
System), VOC (Video Optimization Controller). For each SFC,
its constituent VNFs, requested rate (uniformly distributed
in the given interval) and E2E latency, share of the total
requests and service duration are shown in Table I, along
with the VNFs’ data processing capacities and requirements
in GFLOPS. For a given UE value, 10 deployment scenarios,
each of them with 10 different UE traffic distributions, are run.

Fig. 3. Simulation setup example.
The simulation parameters are presented in Table II. Or-
thogonal channels are assumed between the gNB and the SCs.
However, SCs of different clusters may interfere to each other.
The bandwidth of the mmWave X-haul links is set to 200
MHz, and the fiber link capacity to 10 Gbps. The link budget
formulas and respective parameter values of [3] are used for
the AN and mmWave X-haul links. Each MEC computational
node has 8 CPU cores while nodes in the 1st and 2nd AL have
24 and 48 cores, respectively. P(CP U,m)
ytakes values in the
set {55, 70}for the MEC nodes, {150, 220}for the 1st AL
and {200, 278}for the 2nd AL nodes, while the node’s CPU
idle power is set to 10% of the respective maximum power.
The Optimal CPLEX-based solution and proposed heuristic
(ONE) are compared with the following SoA schemes [8],
which are adapted to the studied setup for a fair comparison:
• Holu: It first places the VNFs on the computational node with
the highest closeness centrality and CPU utilization and then
it chooses the traffic route with the lowest power consumption
that satisfies the E2E latency requirement of the SR.
• BCSP: It places the VNFs on the computational node with
the highest betweeness centrality and selects the least delay
route that satisfies the E2E latency requirement of the SR.
Provided that the SoA algorithms do not account for user
association, we use the default selection criterion in both,
i.e., the users connect to the BS that provides the highest SINR.
A. Simulation results
Fig. 4, 5 demonstrate the energy efficiency ( i.e., number of
successfully sent bits per 1 Joule of energy consumption) and
the computational time (sec), respectively, of all algorithms for
different traffic loads, i.e., number of users and arrival rates
λ. As can be seen, under high traffic load, higher λleads
to lower energy efficiency for all algorithms, since new UEs
are likely to arrive while previously associated UEs are still
served. This increases, on average, the number of concurrently
served UEs, leading to increased power consumption. As for
the computational time, it increases with higher λfor all
algorithms, except for Optimal. This is because, although
TABLE I
SFC AN D VNF DETA ILS [8]
SFC specifications
Type Rate Latency Share Duration
(VNF order) (Mbps) (ms) (%) (sec)
Web [0.6-1] 500 20 20
(NAT-FW-TM-WOC-IDPS)
VoIP [0.404-0.64] 100 20 100
(NAT-FW-TM-FW-NAT)
Streaming [5-24] 100 39 300
(NAT-FW-TM-VOC-IDPS)
Gaming [0.24-0.5] 60 6 300
(NAT-FW-VOC-WOC-IDPS)
Ultra RT AI/ML [15-25] 1 15 40
(NAT-NAT)
VNF specifications
Type NAT FW TM VOC WOC IDPS
Process Capacity (Mbps) 500 400 200 578 300 600
GFLOPS Requirement 110 440 55 110 110 440
TABLE II
SIMULATION PARAMETERS [3], [8], [12]
Parameter Value Parameter Value Parameter Value
P(sw)
idle 315 W Pport 7 W Packet length 1.5 KB
N(gN B)
RF 8N(SC)
RF 4N(mmW )
RF 64
∆(gN B)
a4.7 ∆(SC)
a4∆(mmW )
e100
P(gN B,i)
a6.8 W P(SC,i)
a130 W P(mmW,i)
e3.9 W
fAN 2 GHz fX−haul 60 GHz ¯
N(RB)
a100 RBs
SRs always arrive in batches of 1 (i.e., 1 UE per SR), the
branch-and-cut search performed by Optimal is more likely
to quickly reject some choices, when the network already
serves a higher number of UEs (for higher λ), which would
otherwise be examined (for lower λ), thus leading to decreased
computational time. Clearly, ONE achieves an excellent trade-
off between energy efficiency and complexity, achieving up
to 89% of the optimal energy efficiency, with up to 90%
less computation. All algorithms accept all SRs in all cases,
except for BCSP which achieves a little lower than 100% UE
acceptance ratio (99.5% in the worst case, i.e, for increasing
number of UEs and λvalues). This stems from the fact that
BCSP does not consider the nodes’ CPU load, leading to less
efficient VNF placement, which under high traffic load can
block few UEs. We also note that these results only illustrate
the minimum achievable computational gains of ONE, due to
the assumed 1 UE per SR and, in case an SR involves more
UEs, higher computational gains compared to the optimal
solution are expected.
ONE offers up to 90% and 60% higher energy efficiency
than Holu and BCSP, respectively, with low computational
time, as shown in Fig. 5. This is because, contrary to the SoA,
ONE jointly considers the user association problem, resulting
in higher flexibility, albeit with slightly higher complexity
than BCSP. Nonetheless, compared to Holu, ONE achieves
up to 70% lower computational time, as the CPU load and
node centrality criteria in the case of Holu combined with the
predefined serving BSs, can result in more alternative paths
to be computed under high traffic load. In particular, in Holu
and BCSP, the serving BSs are already determined (based on
the best SINR) when the optimal VNF placement and traffic

Fig. 4. Energy efficiency (bits/Joule) of all algorithms for different traffic
load conditions (number of UEs per gNB area) and arrival rates (λ).
Fig. 5. Execution time (sec) for different traffic load conditions (number of
UEs per gNB area) and arrival rates (λ).
routing are performed. This is supported by Fig. 6, which
provides the power break-down of all algorithms under low
(N= 10) and high (N= 40) traffic for λ= 10, where
Optimal and ONE always select SCs as serving BSs and not
the gNB (contrary to Holu and BCSP), thus achieving much
lower power consumption. We also note that the Optimal
and ONE power consumption scales better than the SoA
with increasing load (ONE still achieves 79% of the Optimal
energy efficiency when N= 40 and λ= 10), which further
supports our approach of jointly studying user association,
traffic routing and VNF placement to guarantee truly optimal
E2E real-time network performance.
V. CONCLUSION
This paper investigated real-time joint user association,
traffic routing and VNF placement in 6G HetNets towards
minimizing network energy consumption while maintaining a
high UE acceptance ratio. The proposed model, leading to a
MILP formulation, captures all characteristics of the employed
technologies, resource and service types as well as their
constraints, while making minimal assumptions. To address
Fig. 6. Power break-down in W of all algorithms for low and high traffic
(N=10 and N=40, respectively) as well as high arrival rate (λ=10).
the high computational complexity of the optimal solution, we
proposed ONE, an energy-efficient real-time heuristic, which
was shown to significantly outperform the SoA, by jointly
considering the user association problem, achieving up to 89%
of the optimal value with up to 90% lower complexity.
ACKNOWLEDGMENT
This work is supported by H2020 5GPPP 5G-COMPLETE
(GA 871900) and H2020 5GPPP INT5GENT (GA 957403).
REFERENCES
[1] “European Vision for the 6G Network Ecosystem,” The 5G Infrastruc-
ture Association, 2021, v. 1.0.
[2] L. Liu et al., “Joint Dynamical VNF Placement and SFC Routing
in NFV-Enabled SDNs,” IEEE Transactions on Network and Service
Management, vol. 18, no. 4, pp. 4263–4276, Dec. 2021.
[3] M. Gatzianas et al., “Energy-efficient Joint Computational and Network
Resource Planning in Beyond 5G Networks,” in 2021 IEEE Global
Communications Conference (GLOBECOM), Dec. 2021, pp. 1–6.
[4] N. Promwongsa et al., “Joint VNF Placement and Scheduling for
Latency-sensitive Services,” IEEE Transactions on Network Science and
Engineering, Mar. 2022, early access.
[5] X. Fan et al., “Real-Time Update of Joint SFC and Routing in Software
Defined Networks,” IEEE/ACM Transactions on Networking, vol. 29,
no. 6, pp. 2664–2677, Dec. 2021.
[6] X. Shang, Z. Liu, and Y. Yang, “Online Service Function Chain
Placement for Cost-Effectiveness and Network Congestion Control,”
IEEE Transactions on Computers, vol. 71, no. 1, pp. 27–39, 2022.
[7] X. Zhang et al., “Near-Optimal Energy-Efficient Algorithm for Virtual
Network Function Placement,” IEEE Transactions on Cloud Computing,
vol. 10, no. 1, pp. 553–567, Mar. 2022.
[8] A. Varasteh et al., “Holu: Power-aware and delay-constrained vnf
placement and chaining,” IEEE Transactions on Network and Service
Management, vol. 18, no. 2, pp. 1524–1539, 2021.
[9] S. Yang et al., “Delay-aware Virtual Network Function Placement and
Routing in Edge Clouds,” IEEE Trans. Mobile Comput., vol. 20, no. 12,
pp. 445–459, Feb. 2021.
[10] M. Gatzianas et al., “Offline Joint Network and Computational Resource
Allocation for Energy-Efficient 5G and beyond Networks,” Applied
Sciences, vol. 11, no. 22, Nov. 2021.
[11] A. Mesodiakaki et al., “Optimal User Association, Backhaul Routing
and Switching off in 5G HetNets with Mesh Millimeter Wave Backhaul
Links,” Ad Hoc Networks, vol. 78, pp. 99–114, Sep. 2018.
[12] W. V. Heddeghem et al., “Power Consumption Modeling in Optical
Multilayer Networks,” Photon. Netw. Commun., vol. 24, pp. 86–102,
Jan. 2012.