Quality of Experience for Streaming Services: Measurements, Challenges and Insights

Abstract

Over the last few years, the evolution of network and user handsets’ technologies, have challenged the telecom industry and the Internet ecosystem. Especially, the unprecedented progress of multimedia streaming services like YouTube, Vimeo and DailyMotion resulted in an impressive demand growth and a significant need of Quality of Service (QoS) (e.g., high data rate, low latency/jitter, etc.). Mainly, numerous difficulties are to be considered while delivering a specific service, such as a strict QoS, human-centric features, massive number of devices, heterogeneous devices and networks, and uncontrollable environments. Thenceforth, the concept of Quality of Experience (QoE) is gaining visibility, and tremendous research efforts have been spent on improving and/or delivering reliable and added-value services, at a high user experience. In this paper, we present the importance of QoE in wireless and mobile networks (4G, 5G, and beyond), by providing standard definitions and the most important measurement methods developed. Moreover, we exhibit notable enhancements and controlling approaches proposed by researchers to meet the user expectation in terms of service experience.

Full text

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2965099, IEEE Access
Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
Digital Object Identifier 10.1109/ACCESS. 2019. DOI
Quality of Experience for Streaming
Services: Measurements, Challenges
and Insights
KHADIJA BOURAQIA1, ESSAID SABIR1, MOHAMED SADIK1, AND LATIF LADID2
1NEST Research Group, ENSEM, Hassan II University of Casablanca, Morocco
2University of Luxembourg, Luxembourg
E-mail: khadija.bouraqia@ensem.ac.ma, e.sabir@ensem.ac.ma, m.sadik@ensem.ac.ma, latif.ladid@uni.lu
Corresponding author: Essaid Sabir (e-mail: e.sabir@ensem.ac.ma).
ABSTRACT Over the last few years, the evolution of network and user handsets’ technologies, have
challenged the telecom industry and the Internet ecosystem. Especially, the unprecedented progress of
multimedia streaming services like YouTube, Vimeo and DailyMotion resulted in an impressive demand
growth and a significant need of Quality of Service (QoS) (e.g., high data rate, low latency/jitter, etc.).
Mainly, numerous difficulties are to be considered while delivering a specific service, such as a strict
QoS, human-centric features, massive number of devices, heterogeneous devices and networks, and
uncontrollable environments. Thenceforth, the concept of Quality of Experience (QoE) is gaining visibility,
and tremendous research efforts have been spent on improving and/or delivering reliable and added-value
services, at a high user experience. In this paper, we present the importance of QoE in wireless and mobile
networks (4G, 5G, and beyond), by providing standard definitions and the most important measurement
methods developed. Moreover, we exhibit notable enhancements and controlling approaches proposed by
researchers to meet the user expectation in terms of service experience.
INDEX TERMS Quality of Experience(QoE); Quality of Service (QoS); QoE Measurements; QoE
Enhancements; 4G/5G/B5G; D2D; M2M; EDGE Computing; Content Caching.
I. INTRODUCTION
UNTIL recently the quality of service (QoS) [1] provided
has been evaluated from a technical perspective to
determine network performance, through measuring several
factors (i.e., throughput, available bandwidth, delay, error
probability, jitter, packet loss, etc.). Nonetheless, for many
services like video streaming, QoS cannot capture the influ-
ence of the network fluctuation on the user experience [2]. In
1994, and according to the International Telecommunication
Union (ITU) recommendation, ITU-T Rec.E.800, [3] the
quality of service was defined as:
“Collective effect of service performance which determines
the degree of satisfaction of a user of the service”
Markaki redefined it [4] as the,
“Capability of a network to provide better service to
selected network traffic ... described by the following
parameters: delay and jitter, loss probability, reliability,
throughput and delivery time”
As we notice in the second definition, user satisfaction is
not considered anymore. Giving that many service providers
are competing for more costumers; a new notion has emerged
Quality of Experience (QoE), used instead of QoS to enhance
the service and get the consumer’s feedback on a specific ser-
vice (e.g., network). The QoE is related to both objective QoS
(i.e., objective metrics depict the influence of the network and
application performance on the user) and subjective [5] (i.e.,
the individual user experience obtained from expectation,
emotional state, feeling, preference, etc.). In other words, it
is an evaluation of individuals experience when interacting
with technology and business entities in a particular context
[6] to provide satisfaction to the end-user.
Here we introduce some definitions of this new concept by
starting with the most used definition for QoE:
“Overall acceptability of an application or service as
perceived subjectively by the end-user ... includes the
complete end-to-end system effects ... maybe influenced by
user expectations and context.”
VOLUME 4, 2016 1

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2965099, IEEE Access
Bouraqia et al.
by ITU-T SG 12 in 2007 [7] but it does not clarify what the
QoE is about and how it could be measured. Based on ITU-T
SG 12 2007 and Dagstuhl seminar 2009 [8] a new influencing
factor , context, was added as follows :
“Degree of delight or annoyance of the user of an
application or service as perceived subjectively includes the
complete end-to-end system effects ... maybe influenced by
user state, content and context.”
Afterward, a better definition was also proposed in the
Dagstuhl Seminar [8]:
“Describes the degree of delight of the user of a service,
influenced by content, network, device, application, user
expectations, and goals, and context of use.”
The last one as far as we know, is considered as a working
definition of QoE is [9]:
“QoE is the degree of delight or annoyance of the user of an
application or service. It results from the fulfillment of his or
her expectations with respect to the utility and/or enjoyment
of the application or service in the light of the user’s
personality and current state.”
We conclude that from all the definitions mentioned above,
there is no practical or exact definition to explain the sub-
stance of the QoE, how to measure it, or what it impacts
on the users’ expectations. However, these definitions give
a broad understanding of the QoE, which offer an excellent
opportunity to research and explore it in depth.
The rest of this paper is structured as follows. We provide
an overview of the influencing factors on the users’ ex-
perience in section II. We introduce different models and
approaches used to measure the QoE in section III. Then,
in section IV, we discuss controlling methods proposed by
various researchers to improve the QoE, section V exhibits
the challenges and enhancements aiming to bring the content
closer to the end user. In section VI, we discuss some recent
technologies and hot problems related to QoE. Finally, a few
concluding observations are drawn in Section VII.
II. FACTORS INFLUENCING THE QUALITY OF
EXPERIENCE
Since the QoE is still a new concept, content providers,
service and network providers, in addition to researchers are
facing new challenges related to delivering, measuring, and
controlling QoE. Then, investigating and analyzing the QoE
influencing parameters (IFs) [10] is a first step to go. It is
hard to predict the QoE because of its subjective nature, see
Figure 1. Therefore, in order to evaluate the overall service
quality, factors that influence the users’ perception should be
determined beforehand [11]. Qualinet [9] has defined IFs of
the QoE as follows:
“Any characteristic of a user, system, service, application,
or context whose actual state or setting may have influence
on the Quality of Experience for the user.”
FIGURE 1. Challenging subjective evaluation from different perspectives.
The IFs could interrelate, thus they should not be classified
as isolated entities. From this perspective, they are classified
into three categories:
•Human-related Influencing Factor: any variant or
invariant property or characteristic of a human user.
The characteristic can describe the demographic and
socio-economic background, the physical and mental
constitution, or the user’s emotional state.
•System-related Influencing Factors: properties and
characteristics that define the technically generated
quality of a service or an application. They are asso-
ciated to media capture, transmission, coding, storage,
rendering, and reproduction/display, also to the commu-
nication of information itself from content production to
the user.
•Context-related Influencing Factors: are factors that
embrace any situation property to describe the user’s
environment, in terms of physical (location and space,
including movements within and transitions), temporal,
social (people present or involved in the experience),
economic (Costs, subscription type, or brand of the ser-
vice/system), task, and technical characteristics These
factors can occur on different levels.
In addition to the three previous IFs (i.e., context level,
system level and user level), Juluri et al. [12] introduced a
fourth IF for video delivery, see Figure 2:
•Content-related Influencing Factors the information
regarding the offered content by the service or appli-
cation under study. They are associated, in the case
of video, with video format, encoding rate, resolution,
duration, motion patterns, type and contents of the
video, etc.
2VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2965099, IEEE Access
Quality of Experience for Streaming Services: Measurements, Challenges and Insights.
Content
Video codec, Resolution, Content
of videos, Type of videos, Motion
patterns, Effectiveness, Popularity,
Age of Content, etc.
Context
Task, Purpose, Social, Cultural
background, Usage history and
efficiency, Personal, Age, Mobility,
Competence, Environmental,
Technological, Cost, etc.
User
Attractiveness, Fidelity,
Enjoyment, Expectations,
Personality, Gender, Initial delays,
Memory effects, etc.
System
Device effectiveness, Device
efficiency, Network Quality, jitter,
Latency, Packet loss, Server
reliability, Server availability,
Transmission network, etc.
QoE
Influencing
Factors
FIGURE 2. Different Factors influencing QoE.
Several works provided other external factors. Like the im-
portance of the application, user’s terminal hardware, and
mobility [11]. Also, five standards of video quality metrics
(i.e., the join time, the buffer ratio, the rate of buffer events,
average bit rate, and rendering quality) were presented in
[13]. As well as the prefetching process, source coding [14]
and the effect of packet reordering [15], [16] studied in [17].
In another perspective, a comparison of the influence of some
metrics the packet loss and bandwidth have a significant
impact than the jitter and delay [18]. In short, it is worth
noting that there are specific IFs relevant for different types
of services and applications.
III. MEASUREMENTS APPROACHES
To consider the user satisfaction in the context of real-time
video streaming applications, QoS is no longer sufficient to
evaluate the quality. Therefore, researches have been con-
ducted to assess the QoE [19]. In this section, we will address
the developed techniques to measure the QoE [20].
Whether using subjective or objective methods or combine
both is discussed in [11] as follows: “Subjective methods are
conducted to obtain information on the quality of multimedia
services using opinion scores, while objective methods are
used to estimate the network performance using models that
approximate the results of subjective quality evaluation.”
A. SUBJECTIVE ASSESSMENT
In [21], subjective assessment is considered as the most
accurate approach to measure the QoE perceived by the end
user. This method gathers human observers in a laboratory to
evaluate sequences of a video and then scores depending on
their point of view and their perception, the average of the
values obtained for each test sequence is known as the Mean
Opinion Score (MOS) [22]; MOS is often used to quantify
these factors. Commonly rated on a five-point discrete scale
as follows [1:bad, 2:poor; 3: fair; 4:good; 5:excellent]. Al-
though MOS is the most known precise assessment, it slows
scoring due thinking and interpretation, as well people are
limited by finite memory and cannot capture users perception
over time. In addition, in a recent research [23] authors have
studied the impact of considering young student (9-17 years
old) as viewers to evaluate the quality of videos (MOS)
subjectively. The results suggested that they are suitable and
can notice different quality issues to the adults. However,
more studies should be performed.
To conduct a subjective quality test, to evaluate a video qual-
ity [24], we introduce some of the widely known standard
methods as follows :
•Double Stimulus Continuous Quality Scale (DSCQS)
[25]: The evaluator is presented twice by reference and
the processed video sequence in alternative fashion,
upon termination of the video he is asked to rate its
quality at a scale of 0 (lowest value)-100 (highest value)
then the difference of the video assessment value is
calculated. In the case of a small value, the quality of
the presented video is close to the reference video else
the quality is low. For a large number of video scenes,
DSCQS needs a very long time to implement quality
assessments.
•Single Stimulus Continuous Quality Evaluation (SS-
CQE) [25] ITU-R recommendation : The user votes
the quality of a continuous video usually of 20 to 30
minutes. This method allows to observe the variation of
the quality over time by calculating the average quality
VOLUME 4, 2016 3

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2965099, IEEE Access
Bouraqia et al.
evaluation of the subjects, SSCQE requires well-trained
observers to attain stable assessment results.
•Absolute Category Rating (ACR) [26] : ACR is recog-
nized as a single stimulus method. The video is watched
for about 10 seconds, and during the next interval up to
10 seconds, the subjects evaluate the video by the five-
grade quality scale expressed as MOS.
•Absolute Category Rating-Hidden Reference (ACR-
HR) [26]. This approach is similar to ACR. Except
that the reference version of each shown distorted test
sequence is also displayed to the participants. After-
ward, they give their scores in the form of MOS, and a
final quality evaluation is computed using a differential
quality score.
•Pair Comparison [26] : Pair of videos are presented to
the subjects to be compared and then evaluated (i.e.,
which one of the pairs has superior quality. The results
vary depending on, which one was shown first, as the
assessments take longer time than the ACR method.
Other standards such as Simultaneous Double Stimulus for
Continuous Evaluation, Subjective Assessment Methodology
for Video Quality, Degradation Category Rating or DoubleS-
timulus Impairment Scale and Comparison Category Rating,
are discussed in [27].
Subjective Assessments are very expansive in term of hu-
man resources, cost and time consumption, however, such a
technique cannot be used as an automatic measurement or
monitoring for real-time applications like video streaming.
Fortunately, there exists another subjective evaluation form
of QoE, that enables new potentialities to conduct web-
based tests. It is more flexible, offers a diverse population as
participants and is cost and time effective. Besides, it creates
a realistic test environment, named Crowdsourcing [28], [29].
Here, we cite some platforms and web-based frameworks:
-Aggregator platforms (e.g., Crowdflower, Crowd-
source): These platforms often delegate the task
to different channels, that provide workers. Such a
system focuses on a limited set of predefined tasks
only. Meanwhile, it might suffer from a significant
drawback as some aspects of the experiment might
not be directly controllable;
-Specialized platforms (e.g., Microtask, TaskRabbit):
This platform focuses on a limited set of tasks or a
specific workers class, as it maintains their workers;
-Crowd providers (e.g., Amazon, Mechanical Turk, Mi-
croworkers, TaskCN): Acknowledged as the most
flexible type, a self-organizing service, maintains a
large work crowd and offers unfiltered access to the
recruited participants;
-Quadrant of Euphoria: Permits for a pairwise compar-
ison of two different stimuli, so the worker could
judge which of the two stimuli has a higher QoE. A
test uncovers fake users and rejects them, but at the
cost of exposing reliable users also to rejection.
On the other hand, an underdeveloped crowdsourcing system
is proposed [30], to evaluate QoE of video on demand
streaming. This system is different from other crowdsourcing
platforms as it can monitor network traffic and the bandwidth,
as well measure the central processing unit(CPU) usage,
Random Access Memory (RAM) utilization, times video
freezes and MOS (i.e., users fill a questionnaire). It proved
to be about a 100% accurate in High Definition display
resolution (HD) and about 81 to 91% in other qualities as
their test shows.
Most of these Crowdsourcing techniques have only allowed
testers to conduct the test on their computers or laptops.
However, Seufert et al. [31] introduced a new application
"CroQoE". It runs on mobile devices to evaluate the QoE
of streaming videos, connected to a Linux back-end server
to dynamically prepare and evaluate the test. Also, they
allowed users to choose the content of videos they would
like to watch. The results proved that this added feature
(i.e., choosing the content) could slightly enhance the QoE
ratings. Still, they utilized , in their tests, only high definition
videos with a duration of fewer minutes. Crowdsourcing
technique has some drawbacks, as there is a little control over
the environment, which may give the participants a chance
to cheat in order to increase their income. Also, as stated
in [32], crowd diversity and expectations, the context, type
of equipment (workers typically use their own devices and
could differ regarding hardware, software, and connectivity)
and the duration and design of the test (small duration will en-
courage the workers while long duration may be unreliable)
impact the QoE assessment.
B. OBJECTIVE ASSESSMENT
A considerable number of objective quality measurements
have been developed using mathematical formulas or algo-
rithms to estimate the QoE on the basis of QoS metrics (
parameters collected from the network). Depending on the
accessibility of the source signal, they are organized into
three approaches:
•Full reference (FR): a reference video is compared
frame-by-frame (e.g., color processing, spatial and tem-
poral features, contrast features) with a distorted video
sequence to obtain the quality (commonly used in lab-
testing environments, e.g., ITU-T J.247).
•Reduced reference (RR): Only some features of the
reference signal are extracted and employed to evaluate
the quality of the distorted signal (e.g., ITU-T J.246).
•No reference (NR): The reference video is inessential
while evaluating the distorted video sequences Quality.
(commonly used for real-time quality assessment of
videos, e.g., ITU-T P.1201).
4VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2965099, IEEE Access
Quality of Experience for Streaming Services: Measurements, Challenges and Insights.
Some of the most known objective quality assessment ap-
proaches are Peak Signal to Noise Ratio (PSNR), Structural
Similarity Metric (SSIM) [33], Multi-Scale Structural SiM-
ilarity [34], SSIMplus [35](supports cross frame rate and
cross resolution), Video Quality Model (VQM) [36], and
Natural Image Quality Evaluator (NIQE) [37]. Despite that
these models outperform PSNR, most researchers commonly
use PSNR [38], the logarithmic ratio between the maximum
value of a signal and the background noise, due to its
simplicity to assess video quality. However, it cannot be
appropriate to be used in a real-time mechanism. A heuristic
TABLE 1. Mean Opinion Score
MOS Quality Perception PSNR (dB)
5 Excellent Imperceptible 37
4 good Perceptible 31 - 37
3 Fair Slightly annoying 25 - 31
2 Poor Annoying 20 - 25
1 Bad Very annoying ≺20
mapping of PSNR to MOS (see Table I) exists though, the
research in [39] revealed that the correlation between the
PSNR and subjective quality would be decreased if the codec
type of the video content changes unless otherwise. PSNR
is a qualified indicator of video quality. Here we exhibit few
PSNR to MOS mapping models:
•The relation between PSNR and MOS for time-variant
of video streams quality on mobile terminals [40]:
P SN R(n) = 10 ·log 2552
MSE(n)(1)
where MSE(n)is defined as follows:
MSE(n) =
N
P
i=1
M
P
j=1
C
P
c=1 Fc
n(i, j)−Rc
n(i, j)2
N·M·C(2)
Further,
\
MOSP S N R(n)is captured using a linear law
\
MOSP S N R(n) = a·P S NR(n) + b(3)
with
a=cMO S,P SN R
σ2
P SN R
and b=µMO S −a.µP SN R
where MSE(n)denotes the mean square error of the n-
th frame Fncompared to the frame Rnof the reference
sequence.
iand jaddress particular pixel values within the frame.
Cis the number of the color components and cis an
index to address them.
cMO S,P SN R represents the sample co-variance between
the PSNR(n), and the MOS(n).
µP SN R and µMO S are the sample means of PSNR
respectively MOS.
σ2PSNR is the sample variance of PSNR.
aand bare respectively the scaling and the shift factors.
•The PSNR-MOS nonlinear mapping model on the wire-
less mobile network for video services as follows [41]:
P SN R = 10 ·log 


2552
a
expRp
b−1+β·P LR 

(4)
aand bare model parameters associated with measured
data, Rptransmitted rate of the the video service and
P LR is the packet loss rate.
MOS =




1,P SN R≤20.
α·th(ξ·P SN R −β) + γ, 20<P S NR<50.
5,≥50.
(5)
α,β,ξand γare parameters that vary with the content
and structure of the video sequences.
•PSNR to MOS mapping using an S-type (sigmoidal)
mapping function [42]:
MOS =1
α+ exp (β(γ−PSNR )) +λ(6)
α,β,γand λare related parameters that can be determined
through many experiments. Moreover, authors in [43], based
on the article [44], have evaluated a relationship between
MOS and the bit rate as follows:
MOSV ideo =




0.5, R < 5kbps.
αlog(β·R),5kbps ≤R < 250kbps
4.0, R ≥250kbps.
(7)
where Ris the bit rate, αand βthe parameters obtained from
the upper and lower limit of MOS values. Based on the paper
[45] α= 2.3473 and β= 0.2667. After presenting PSNR;
Other Frameworks were proposed to measure and predict
future QoE collapses, such as:
•The bit-rate switching mechanism is executed at the
users’ side in a wireless network, to elevate the quality
of the user and determine the QoE metrics. Xu et al.
propose [46] a framework for dynamic adaptive stream-
ing, that, given the bit rate switching logic, computes
the starvation probability of playout buffer, continuous
playback time and mean video quality. It can be used to
predict the QoE metrics of dynamic adaptive streaming.
•YoMoapp [47], a passive android application was
employed in a field study of mobile youtube video
conducted in [48] to monitor the application-level
key performance indicators (i.e., buffer and the video
resolution) of YouTube in the user’s mobile device,
this monitoring application works on JavaScript which
might indicate some errors however it is accurate by
VOLUME 4, 2016 5

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2965099, IEEE Access
Bouraqia et al.
approximately 1second.
•Pytomo [49] evaluates the playback of a played
YouTube [50] video as experienced by users. It collects
the download statistics such as the ping, the downloaded
playback statistics, number of stalling event and the total
buffer duration, then estimates the playout buffer level.
Moreover, Pytomo allows the study of the impact of the
DNS resolution. This tool could be YoMo complemen-
tary. However, it is not feasible, due to the need to access
the user’s device.
•An application for mobile service [51] was proposed
to measure the QoE directly from the user’s device, in
order to transmit the results to the service provider while
preserving the user’s privacy.
•QMON [52]is a network-based approach that moni-
tors and estimate the QoE of the transmitted video
streaming. It focuses on the occurrence and the duration
of playback stalls, also it supports a wide range of
encoding (MP4, FLV et WebM). The study confirmed
that streaming parameters (i.e., stalling times, times
on quality layers) are the best appropriate for QoE
monitoring, as the result ensured to provide an accurate
developed model to estimate QoE.
•The authors in [14] study the quality of streaming
from the aspect of flow dynamic. They develop an
analytical framework that computes the QoE metrics
like dynamics of playout buffer, scheduling duration,
and the video playback variation, in a streaming service
over wireless network. The framework is assumed to
anticipate precisely the distribution of prefetching delay
and the probability of generating a function of the buffer
starvation. The obtained result proved that the flow
dynamics has more influence on QoE metrics. Also, it is
assumed to be suitable in some scenarios like hyper-
exponential video length distribution, heterogeneous
channel gains, mixed data, and streaming flow.
•Network operators may handle long and short views
with different priorities. Thus [53] build a model on
starvation behavior in a bandwidth sharing wireless
network by using a two-dimensional continuous time
Markov process and ordinary differential equations
to determine that progressive downloading increases,
considerably, the starvation probability. Further, they
observe based on their result, that the history of time-
independent streaming traffic pattern can predict the
future traffic, and that the viewing time follows a hyper-
exponential distribution which is validated to be more
accurate than some existing models (i.e., exponential,
Pareto distribution).
•The paper [54] proposes a real-time video QoE software
assessment system. It evaluates the error of network
in the part of video transmission, by testing the value
of the service quality, the quality of transmission, the
encoded videos in various contents and sizes. The
authors indicate that this platform is deployable on a
real network.
•A QoE Index for Streaming Video (SQI) model was
proposed by Duanmu et al. [55] to predict the QoE
instantly. To build their model, they have started by
constructing a video database (effect of initial buffering,
stalling, video compression), then investigate the inter-
actions between video quality and playback stalling.
The SQI seems to be ideal for the optimization of media
streaming systems as well; it is simple in expression
and effective. However, it does not support reporting
function on the degradation of QoE and has limited
monitoring parameters.
•YOUQMON [56] estimates the QoE of YouTube videos
in real time in 3G networks. It combines passive traffic
analysis and a QoE model to detect stalling events
and project them into MOS. Each minute monitoring
system computes the number of stalling as the fraction
of stalling of every detected video, as well it supports
two video formats used by YouTube, AdobeFlash, and
Moving Picture Experts Group (MPEG4). The results
appear to be accurate and similar to the MOS values
and indicate the potentiality of the performance of this
system. Still, it cannot identify the point of the network
that impacts the quality.
•The QoE Doctor tool [57] is an Android tool that can
analyze across different layers (application, transport,
and network), from the app user interface (UI) to the
network. The tool employs a UI automation tool to du-
plicate user behavior and to measure the user-perceived
latency (i.e., identify changes on the screen), mobile
data consumption and network energy consumption.
QoE Doctor can quantify the factors that impact the app
QoE and detect the causes of QoE degradation, although
it is unfit to supervise or control the mobile network, the
component responsible of detecting UI changes has to
be adjusted for each specific app.
•Zabrovskiy et al. [58] presented AdViSE, an Adaptive
Video Streaming Evaluation framework of web-based
media players and adaptation algorithms. It supports
different media formats, various networking parameters
and implementations of adaptation algorithms. AdViSE
contains a set of QoS and QoE metrics gathered and
assessed during the adaptive streaming assessment eval-
uation as well as a log of segment requests, applied to
generate the impaired media sequences employed for
the subjective evaluation. Still, they do not provide a
source code level analysis of familiar Dynamic Adaptive
6VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2965099, IEEE Access
Quality of Experience for Streaming Services: Measurements, Challenges and Insights.
Streaming over HTTP (DASH) players and support for
popular commercial streaming players. In [59], same
authors proposed an end-to-end QoE evaluation to col-
lect and analyze objectively (AdViSE) and subjectively
(Web-based subjective evaluation platform (WESP)
[60]) the streaming performance metrics (e.g., startup
time, stalls, quality). The framework is flexible and
can also determine when players/algorithms compete
for bandwidth in different configurations although it
does not consider content CDNs, Software-Defined
Networking (SDN), nor 5G networks.
•VideoNOC [61] is a video QoE monitoring proto-
type platform for Mobile Network Operators, consid-
ering video QoE metrics (e.g., bitrate, rebuffering).
VideoNOC allows to analyze the impact of network
conditions on video QoE, reveals video demand across
the entire network, to develop and build better networks
and streaming service. Despite, the platform disregard
transport-layer and relevant RAN KPIs data and QoE
inference on encrypted video traffic.
•In the same vein, an online ML named ViCrypt is
introduced [62], to anticipate re-buffering events from
encrypted video streaming traffic in real-time. This
approach, after it subdivides the video streaming session
into a series of time slots, that have the same length.
It employs a fine-grained time slot length of 1second
(for a proper tradeoff between precision and stalling
delay detection), from which, the characteristics are
extracted. Afterward, they are used as an input to the
ML model to predict the stalling occurrence. It should
be mentioned that the initial delay and length of stalling
events can be also be obtained. As an extension to the
later work, the authors have demonstrated in [63] that
ViCrypt can additionally predict the video resolution
and average video bitrate accurately. As an extension
to the later work, the authors have demonstrated in
[63] that ViCrypt can additionally predict the video
resolution and average video bitrate accurately. Also,
Vasilev et al. [64] opted to build an ML model to
anticipates the rebuffering ratio based on the hidden
and context information to enhance the precision of
prediction through Logistic regression.
•Lin et al. [65] applied a supervised ML and support
vector machine to anticipate users’ QoE by considering
the number of active users and channel conditions
experienced by a user. They classify a session in two
categories (i.e., with or without stall events) based on
cell-related information collected at the start of a video
session. Considering the starvation events, mobile users
experience them more than adaptive streaming and
static users. As well these last, are more accurate and
convenient to predict their starvation event. Similarly,
a multistage machine learning cognitive method is
developed by Grazia et al. [66]. Although, this model
combines unsupervised learning of video characteristic
with a supervised classifier trained to extract the quality-
rate features automatically. Their model is supposed to
exceed the other offline video analysis approaches.
•Orsolic et al. [67] proposes YouQ, an android applica-
tion to prognosticate The QoE (i.e., stalls, quality of
playout and its variations ) employing machine learning
(ML) relying on objective metrics like throughput and
packet sizes extracted from the stream of encrypted
packets. Though, the promising result, the majority of
the features depends on TCP, meaning that, in regards
to UDP, these techniques probably will fail.
•Similarly the authors [68], suggested a QoE detector
based on extracted data from networks’ packets em-
ploying a deep learning model. The model is based
on a combination of an RNN, Convolutional Neural
Network, and Gaussian Process (GP) classifier.This
classifier can recognize video abnormalities (i.g., black
pixel, ghost, blockness, columns, chrominance, color
bleeding, and blur) at the current time interval (in 1-
second) and predicts them. The model is supposed to
predict video QoE in a real-time environment; however,
it could encounter a few issues like having a small
amount of training data.
•ECT-QoE framework [69] predicts at the instant the
QoE of streaming over DASH, based on the expectation-
confirmation theory and the video database, they have
built. The model is presumed to defeat several models,
especially when combined with the SSIMplus model.
Despite that, ECT-QoE can be applied only to videos
consisting of view segments.
•Wu’s model [70], contrary to other propositions, ex-
amine the global intensity and local texture metrics
extracted from a decoded video, to predict stalls event
and assess the user’s quality. The algorithm maps the
normalized number and duration of stalls using linear
combinations. When compared to other models (e.g.,
[71], [72], [73]), Wu’s proposition appears more consis-
tent concerning subjective perception.
•A cost-constrained video quality satisfaction (CVQS)
framework is proposed [74] to predict the quality ex-
pected, considering some metrics such as the high cost
of data. Despite that, it indicates satisfactory results the
accuracy of the CVQS could be impacted by the video
encoder as well in their test the client can only obtain
the next video segment after two seconds.
There are a large number of standards, that offer indications
on good and accustomed practices, for certain test applica-
tions, standards do not provide the best or most advanced
VOLUME 4, 2016 7

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2965099, IEEE Access
Bouraqia et al.
TABLE 2. ITU Recommendations on Subjective and Objective measurement
Speech Audio Video
Subjective
P.85(2013) [77]
P.800(2016) [78]
P.805(2007) [79]
P.806(2014) [80]
P.808(2018) [80]
P.830(1996) [81]
P.835(2003) [82]
P.1301(2017) [83]
P.1302(2014) [83]
P.1311(2014) [84]
P.800(2016) [78]
P.830(1996) [81]
P.831(1998) [85]
P.911(1998) [86]
P.913(2016) [87]
P.1301(2017 [83])
P.1302(2014) [88]
BS.1116-1(1997) [89]
BS.1284(2003) [90]
BS.1285(1997) [90]
BS.1534(2014) [91]
BS.1679(2004) [92]
P.910(2009) [26]
P.912(2016) [93]
P.913(2016) [87]
P.917(2019) [94]
BT.500-10(2000) [95]
BT.1663(2003) [96]
BT.1788(2007) [97]
BT.2021(2012) [98]
J.140(1998) [99]
J.245(2008) [100]
J.247(2008) [101]
Objective
P.563(2004) [102]
P.561(2002) [103]
P.562(2004) [103]
P.564(2007) [104]
P.862(2001) [105]
P.863(2018) [106]
G.107(2014) [107]
P.561(2002) [103]
P.564(2007) [104]
P.862(2001) [105]
P.863(2018) [106]
P.920(2000) [108]
P.1201(2012) [109]
P.1305(2016) [110]
P.1310(2017) [111]
P.1311(2014) [84]
P.1312(2014) [112]
BS.1387(2000) [113]
G.1070(2018) [114]
G.1091(2014) [115]
J.343-rev(2018) [116]
P.1201(2012) [109]
P.1202(2012) [117]
P.1203(2017) [118]
P.1401(2012) [119]
BT.1683(2004) [120]
BT.1866(2010) [121]
BT.1867(2010) [122]
BT.1885(2011) [123]
BT.1907(2012) [124]
BT.1908(2012) [125]
J.143(2000) [126]
J.144(2004) [127]
J.246(2008) [128]
J.247(2008) [129]
J.249(2010) [130]
J.341(2016) [131]
J.342(2011) [132]
G.1022(2016) [133]
G.1070(2018) [114]
G.1071(2016) [134]
method available, but it gives solid, common basis which is
accessible to all, like ITU - International Telecommunication
Union [75] (Table II). Furthermore, a survey [76] summa-
rized various ITU- measurement methods to evaluate video
streaming quality.
C. HYBRID ASSESSMENT
According to [135] QoE of a user’s performance can be
estimated based on objective and subjective psychological
measures while using a service or product. Moreover, another
approach exists that consists of a combination of subjec-
tive and objective assessment, referred to as The Hybrid
approach. Using machine learning algorithms [136], [137],
statistics, and other fields. It could be employed in real time,
and it is categorized as the most accurate approach since it
decreases the weaknesses of previous approaches [19].
For instance, the Pseudo Subjective Quality Assessment
(PSQA) was created to give similar results as perceived by
human in real-time, as it provides an accurate QoE mea-
surement [138], [139]. PSQA is based on training a partic-
ular type of statistical learning approach, Random Neural
Network (RNN). To evaluate the quality of the video, the
IFs on the quality are selected to be used to generate sev-
eral distorted video samples. Afterward, these samples are
subjectively assessed. Then the results of the observations
are employed to train the RNN in order to apprehend the
relation between the factors that cause the distortion and the
perceived quality by real humans. The training method is
performed once, after that the trained network can be used
in real time. A comparison study in [138] proved that PSQA
is more effective than subjective (MOS), objective (PSNR),
in the matter of time-consuming, manpower moreover it runs
in real-time. Likewise, a further investigation was done [139]
in the context of Multiple Description Coding (MDC) video
streaming over multiple overlay paths in video distribution
networks, confirms the same result as in [138]. Because, after
training MDC-compatible version of PSQA; PSNR could not
evaluate, and its results did not change a lot corresponding to
the Group of Pictures (GOP) size. On the contrary, PSQA
module considered the size of GOP and differentiated if
MDC is used or not. Nevertheless, this approach is not
applied in wireless mesh networks. Fortunately, another tool
called Hybrid Quality of Experience (HyQoE) can predict
for real-time video streaming applications [140]. It takes into
account six parameters percents losses in I frame, P frame
and B frame, general loss, complexity, and motion. Com-
paring HyQoE to other tools, they have demonstrated that,
PSNR algorithm does not take into consideration the human
visual system and the MPEG structure during the assessment
process. Also SSIM is inadequate to reflect the user opinion
when different patterns of loss, motion, and complexity are
analyzed, and that video quality mode generates low scores.
SSIM (i, k) = (2µiµj+c1)(2σij +c2)
(µ2
i+µ2
j+c1)(σ2
i+σ2
j+c2)(8)
Where µiand µjare respectively, the average value in the
block of the original and the distorted image. c1and c2are the
variables that stabilize the division with weak denominator.
σ2
iand σ2
jare respectively, the variance in the block of the
original and the distorted image. σij denotes the covariance
of the block of the original and the distorted image.
HyQoE gives result quite similar to the one given by MOS.
They believe that it can be used to optimize the QoE by
improving the usage of the network’s resources. Likewise,
Chenet al. [141] proposed a framework that seizes the
users’ perception while using network applications named
Oneclick. If ever the user is displeased, he can click a
button to indicate his feedback. Then the collected data is
analyzed to determine the user’s perception under variable
network conditions. The tool is supposed to be intuitive,
lightweight, time-aware, and it is convenient for multi-modal
QoE assessment and management studies considering its
application independent nature. The framework considered
to give the same result as MOS but faster. Furthermore, the
authors in [142] employed four ML algorithm (i.e., Decision
Tree, neural network, kNN, and random forest) to evaluate
MOS value, Based on VQM and SSIM values (i.e., the effect
of video distortion and structural similarity). Thus, to assess
the performance of these algorithms, the Pearson correlation
8VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2965099, IEEE Access
Quality of Experience for Streaming Services: Measurements, Challenges and Insights.
coefficient and the Root Mean Square Error are employed.
According to the results, the Random Forest algorithm was
the best in anticipating user perception. However, network
parameters like transmission delay and response time are not
taken into account.
MLQoE is a modular user-centric algorithm developed by
Charonyktakis et al. [143], based on supervised learning
to correlate the QoE and network parameters such as aver-
age delay, packet loss, average jitter. The framework uses
multiple machine learning algorithms (i.e., Artificial Neural
Networks, Support Vector Regression Machines, Decision
Trees, and Gaussian Naive Bayes.). The one that outperforms
the others, as well as its parameters, will be selected automat-
ically considering the dataset employed as input. According
to their result, MLQoE can predict precisely the score of the
QoE compared to other existing machine learning model.
As well, in [144] the authors have suggested a trained ML
model that predicts the MoS value in SDN, based on network
parameters (e.g., bandwidth, jitter, and delay), Their proposal
seems to be efficient.
YoMoApp (YouTube Monitoring App) [145] is an under
improvements tool. It monitors the application and the net-
work layer (i.e., the total amount of uploaded and down-
loaded data, is logged periodically) for both mobile and
WiFi networks streaming parameters. As well to obtain
subjective QoE ratings from end-users (MOS). The data
is, anonymously uploaded, to an external database. Then
a map is generated from the uploaded data of all users
to reveal how every network operator function and how
to be employed to benchmark them. YoMoApp performs
accurate measurements on an adequately small time scale ( 1
second). They recommended that QoE measurements have to
consider more extended video clips. However, the tool uses
JavaScript, which can occasionally cause inconsistencies and
errors. The latter was employed as well as another Android-
based passive monitoring tool to investigate the precision
of different approaches. Consequently, streaming parameters
revealed high correlations to the subjectively than for the
objective experienced quality, which proves that it is better
suited for QoE monitoring. [48]. Also, authors in [146] have
used YoMoApp to monitor video sessions and obtain several
features from end-user smartphones(e.g., the signal strength
and the number of incoming and outgoing bytes). They,
using ML, introduce a lightweight approach to predict Video
streaming QoE metrics such as initial delay, number, and
the ratio of stalling and user engagement. According to their
evaluation, network layer features is enough to get accurate
results. Recently, [147] propose an ML model called Video
Assessment of Temporal Artifacts and Stalls (ATLAS). It
uses an objective video quality assessment (VQA) method by
combine QoE-related features and memory features sources
of information to predict QoE. They have also adopted, a
subjective assessment, LIVE-Netflix Video QoE Database
[148] to evaluate their model. Although the model is only
apt to deliver overall QoE scores and cannot be used for real-
time bit-rate decisions.
To sum up, the hybrid approach can collect metrics simulta-
neously from both the network and user-end. Such methods
would help to correlate the QoS metrics on the QoE and
generate a better MOS prediction tool. Also, hybrid studies
will allow the study of the impact of the variations in the
performance of the network on the users’ QoE [12].
Moreover, little research has been conducted in this area.
Like in [149], authors have examined the effect of user
behavior (e.g., seeking, pausing, and video skipping) on
the accuracy of the trained QoE/KPI estimation models.
They have concluded that when including user’s various
interactions, much better results will be obtained. However,
more studies should be done.
We summarize some of the measurement approaches dis-
cussed above in Table III.
IV. CONTROLLING QUALITY OF EXPERIENCE
As previously presented, various metrics influence the QoE.
In this section, several approaches and observations will be
discussed to enhance and control the QoE of video streaming
services. Some may presume that increasing QoS, means
precisely a higher QoE as stated in [21]. Except that the user
could be content if he is expectations and requirements are
fulfilled especially if the context of the video is interesting.
The previous findings were confirmed by [151] as their re-
sults indicated that even frame freezes and shorter playbacks
are acceptable by viewers.
Although it was proven that as the number of starvation
increase the experience decrease, which the user is unable
to endure and finally deserts the video [51]. Hence, to avoid
starvation, prefetching/Start-up delay and re-buffering delay,
a model was proposed in [152], to optimize the QoE by
computing the optimum start-up threshold that influences the
number of starvation, which allows the content provider to
achieve its QoE requirements choosing the right QoE metrics
and to avoid starvation. Likewise, authors in [153], when
analyzing the buffer starvation, have suggested that service
providers should configure different start-up threshold for
different categories of media files. Furthermore, based on the
observations in [53] they advice network operators, that to
enhance the QoE of short views, they should be configured
in a higher scheduling priority to reduce the starvation signif-
icantly and start-up delays, in the other hand the probability
of starvation will slightly increase for long views. However
content providers are unwilling to share statistics of views
with network providers.
The authors in [154] adopted Lagrange Multiplier, after
studying the probability of starvation (ps) of different file
distribution to exploit the trade-off between psand the startup
VOLUME 4, 2016 9

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2965099, IEEE Access
Bouraqia et al.
TABLE 3. QoE Measurement Approaches.
Related
literature
Measurement Approaches Monitoring
Point
Methods and Techniques Complexity Accuracy Challenges
Subjective Objective Hybrid
[14] XNetwork
Both CBR and VBR streaming
were considered under static and
fast fading channels. Ordinary
differential equations were con-
structed over a Markov process,
to determine the prefetching de-
lay distribution and the starvation
probability.
High Yes Computational
complexity
[28] XUser
Platforms and web-based frame-
works performed online gather
submitted opinions about the sub-
jected test from different partici-
pants around the world..
Low 90% users’ fairness
[141] XUser Collect data after the users’ feed-
back, then analyze them to deter-
mine the users’ perception.
Low Yes users’ fairness
[46] XNetwork
They work on a slow channel fad-
ing or shared by multiple flows
over a wireless network, modeled
by a continuous time Markov pro-
cess. After formulating the ordi-
nary differential equations they
solve them with Laplace Trans-
form, to be presented as star-
vation probability and the mean
continuous playback time.
High Yes
Uncontrollable
network
conditions
Computational
complexity
[150] XUser
Detect YouTube’s flow and ana-
lyze the packets in order to calcu-
late the buffer status, then mon-
itor it constantly; if the status
falls below a critical threshold, it
raises the alarm.
High Yes
User’s fairness
User’s privacy
[49] XUser
Computes and stores the down-
loaded statistics in a database of
each video, by resolving the IP
address of the video server; that
is used to perform the analysis.
High Yes Data
limitations
[52] XNetwork
First, they extract the playout
timestamp, and then the algo-
rithm calculates the actual buffer
fill level and the duration of the
stalling event.
High Yes
Access to the
user’s device
Algorithm
limited to
YouTube
[54] XUser and
Network
After collecting the viewers’ ex-
perience, through a mathematical
model, QoE scores are evaluated.
High Yes
Computational
cost
User’s fairness
[140] XUser
HyQoE evaluates the quality of
the video, based on static and
trained learning (random neural
network) over a wireless mesh.
High Yes Computational
complexity
10 VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2965099, IEEE Access
Quality of Experience for Streaming Services: Measurements, Challenges and Insights.
delay. They were able to optimize the start-up delay by 40%.
In contrast, dynamic adaptive bit-rate was not considered in
their scenario. In the same manner, another work [155] used
KKT-conditions based on a Resource Allocation Algorithm
[156] to optimize the problem (i.e., reduce the occurrence
of stalling events, assure fairness among users (whether
utilizing dynamic adaptive streaming or not)). Compared to
other proposals(e.g., Proportional Fair Resource Allocation
[157]and Base Station Optimization [158]) theirs indicate
better performance. For example, in a disturbed traffic net-
work, authors [17] proposed to keep the packet reordering
percentage below 20% to maintain an acceptable level of
QoE. Still, they have streamed the video using UDP protocol
in their study.
The streaming service has adopted a new protocol that
answers to the massive demand on network requirements
like bandwidth, entitled DASH [159], [160]. It is proved to
adapt the quality of the requested video, based on the current
bandwidth and devices qualification, but it is affected by
many factors based on [161], [162], initial delay, stalling
and level variation (frame rate, bit rate and resolution),
besides other factors like video length and the number of
motions in the video. Consequently, to derive an effective
the trade-off between the network variations and dynamic
videos streaming behaviour, they [163] introduce a queue-
based model to analyze the video buffer (GI/GI/1 queue) with
pq-policy (pausing or continuing the video download) using
discrete-time analysis. Suggesting to adjust the buffering
thresholds according to the bandwidth fluctuations to reduce
the stalling vents. In the same aspect, authors [164], after
studying the impact of variable and fixed segment duration
(HAS streaming services commonly use segments of equal
duration) on the stalling probability, proposed a variable
segmentation approach that effectively increases the content
encoding (i.e., reduced bit-rate per video cliP. However,
the segment duration can affect the QoE of the streaming
behavior of DASH. Besides, authors in [74] suggested a
trade-off between profit and service, to network operators
and mobile providers. It states that based on several metrics
like cost of data and encoding, they can decide the suitable
quality level to transfer data to the end user and thereby,
reduce the video storage and optimize resource allocation.
In the same context, the framework named QUVE [165]
intended to increase the QoE of video streaming services.
It comprises two principal sections The first approach, the
QoE estimation model, considers encoding parameters, re-
buffering conditions and content time to assess the QoE
for constant bit rate (CBR) video streaming. The second,
QoE parameter estimation approach, it predicts the network
quality, re-buffering time and count for the proposed model.
The results attest that QUVE is adequate to improve the QoE
by choosing the adequate encoding based on a user network
conditions.
In another context, users usually find it troubling to decide
the next segments quality level to maintain a high QoE. Thus
an extension of DASH player is presented [166] to take the
decision based on Markov Decision Process (MDP) called
MDP-based DASH. It requires a bandwidth model and a
learning process, so after adequate training, the player pa-
rameters are tuned to be employed. It is shown that adopting
MDP to adapt video quality will reduce notably the video
freezing and buffering events.
There exist also a bit-rate switching mechanism permitting
users to choose among different switching algorithms to
control the starvation probability, which is difficult to define
its behaviour, as the wrong choice affect the QoE. In [46] a
framework is proposed to assist the user in finding the opti-
mal bit-rate to optimize the QoE, taking into consideration
all the future occurrences. Also to provide the QoE expected
from video streaming HTTP adaptive Bitrate (ABR) was
adopted, caching many streaming files to meet up with the
QoE requirements. ABR encountered a problem of storage
to control it, an optimal subset of playback rates that would
be cashed is chosen. As a solution to this problem, the authors
in [54] developed a model for QoE driven cache management
to offer the best QoE and avoid the content storage to be filled
up rapidly.
Regarding the increase in energy consumption in a cellular
network and mobile devices authors in [167], have conducted
a study in the subject. They have asserted that to maintain
a good balance between QoE and energy consumption,
while watching a video from a mobile phone over Long-
Term Evolution (LTE) networks, a new design of video
streaming service will decrease the energy consumption by
30%. Though; some points (increasing the length of video
segments, increasing the buffer size, the strength of the signal
and using appropriate DASH sittings) should be taking into
consideration. In another paper by Song et al. [168] they
propose an Energy-aware DASH (EDASH) framework over
LTE to optimize network throughput and to find an excellent
balance between energy consumption of the users’ device and
QoE, that proves based on their experiments its efficiency.
The authors in [169] have determined the mathematical for-
mula expressed by two QoE metrics (video rate, the probabil-
ity of timely delivery of video packets), in order to compute
the probability of time delivery of DASH over a wireless
access cell (LTE) to determine the bandwidth assigned to
the mobile user to maintain a satisfactory QoE. Moving cell
phones between wireless access networks make it hard to
maintain a good QoE. Thus in [170] they have proposed an
adaptive streaming protocol consisting of network adaption
and buffer management block that dynamically adapts the
bit rate according to network conditions fluctuations, to
provide a stable QoE over 5G. The protocol is designed
independently of the operating system (OS) version and CPU
performance of the mobile device. The result indicates that
the proposed protocol seemed to enhance the users’ QoE, as
it has been deployed commercially in South Korea for more
VOLUME 4, 2016 11

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2965099, IEEE Access
Bouraqia et al.
than five years over commercial LTE/3G and wifi networks.
To improve DASH efficiency under different network condi-
tions, they suggest [171] a dynamic adaptive algorithm that
can be utilized in both bandwidth and buffer based methods.
It depends on current bandwidth fluctuation to choose the
best quality video, guarantees the continuity and real-time
video streaming to keep a high QoE. To test their model they
have to utilize Google ExoPlayer [172] an Android-based
mobile DASH as a video player for Android. The results
obtained attest that the approach attains a significant average
QoE and performs steadily under various networks as no
rebuffering happens except in the initial buffering stage ( 0.35
seconds ).
In addition, to address the problem of network delays for
CBR and variable bit rate over 5G mobile networks. In this
paper [173] they describe an analytic method that addresses
this challenge, also they present the method to compute the
users’ QoE based on an exponential hypothesis for streaming
traffic using delay and packet loss rate as metrics. This
approach decreases the network delays of traffic by less than
1 ms, therefore improve the QoE.
Furthermore, in some bidirectional streaming services, the
up-link capacity might also be required as much as down-
link capacity. For instance, the authors of [174] propose a
piggyback mechanism for audio-video IP transmission over
the uplink channel to enhance the QoE, which seems to
perform well. The result obtained shows that the mechanism
is rather more effective in adaptive allocation schemes than
under static allocation schemes. However, it seems Nunome
and Tasaka have tested their proposition on other classes of
contents.
Dutta et al. [175], to face the challenges encountered in
5G networks (i.e., arranging the connectivity of high data
rate to an expanding mobile data traffic), suggest an ap-
proach to allow the cloud infrastructure to dynamically and
automatically change the resources of a virtual environment,
to use the resources efficiently and to provide an adequate
QoE. The approach seems to be able to ensure a real elastic
infrastructure and promising in handling unexpected load
surges while reducing resource, demanding real-time values
of PSQA.
Other research efforts suggest that a better quality perception
might be met when the quality should be controlled. In [176],
[177], the authors apply provisioning-delivery hysteresis for
QoE in video streaming case, in order to predict the behavior
of the throughput and the QoE to control the quality, using the
SSIM. Another mechanism [178] is proposed to control the
quality, as congestion degradation affects QoS which impacts
thereby the QoE of users. The authors introduce an Admis-
sion Control (AC) mechanism based on QoS and QoE met-
rics, using a joint QoS/QoE that is predicted by a QoS/QoE
mapper. Based on these metrics the AC decides whether the
user should be accepted within the small-cell network on
not. Though the results obtained are encouraging, AC is only
simulated and has not been implemented in realistic network
as far as we know. In addition, an introduction of SELFNET
5G project [179] provides a self-organized capability into 5G
networks achieving autonomic management of network in-
frastructure. It designs and implements an adaptable network
management framework to provide scalability, extensibility
and smart network management reducing and detecting some
of the network problems. The framework improves the QoE
also and reduces the operational expenditure (OPEX).
V. BRINGING QOE AT THE EDGE
In a typical scenario when a mobile device requests a video
content, it is issued from the servers of Content Delivery
Network (CDN), then crossing the mobile carrier Core
Network (CN) and Radio Access Network (RAN). Clearly,
a massive number of simultaneous streams would generate
a colossal demand at backhaul side. Moreover, the wireless
channel uncontrollable conditions (e.g., fading, multi-user
interface, peak traffic loads, etc.) might be a challenging
issue for the monitoring of user’s QoE and would be an
additional load on the cellular network. Yet, delivering a
streaming content is rather difficult, giving that the channel
between servers providing the desired content and users can
cause delays when transporting data, which would impact
the user’s experience. Bringing the content closer to the user
via caching promises to overcome several obstacles like the
network load and delays resulting in an enhanced QoE [180].
To improve users’ QoE when using dynamic rate adapta-
tion control over information-centric networks, StreamCache
[181] is proposed. This latter periodically collects statistics
(i.e., video requests) from edge routers to make a video cache
decision. The results indicate that this approach offers a near-
optimal solution for real-time caching as it enhances the QoE
by increasing the average throughput. However, the cache
size at routerss level might influence the performance. Also,
a mobile Edge Computing (MEC) scheme was suggested
[182] to permit network edge assisted video adaption based
on DASH. The MEC server locally caches the most popular
segments at an appropriate quality based on collected data
from the network edge (i.e., throughput, latency, error rate,
etc.). To solve the problem of cache storage a context-aware
cache replacement algorithm, replaces old segments by new
popular ones, which leads to maximizing the users’ QoE as
it ensures a steady playback minimizing frequent switching.
Proactive service replication is a promising technique to
decrease the handover time and to meet the desired QoE
between different edge nodes. However, the distribution of
replicas inflates resource consumption of constrained edge
nodes and deployment cost. In [183], the authors have
proposed two integer linear problem optimization schemes.
The first scheme aims to reduce the QoE collapse during the
12 VOLUME 4, 2016

Quality of Experience for Streaming Services: Measurements, Challenges and Insights.
handover; whereas the second scheme aims to reduce the
cost of service replication. Evaluating this scheme in MEC,
mentioned above, the authors believe the effectiveness of
their solutions as will they could provide more information
about the network (i.e., predict the user’s mobility pattern).
Furthermore, to manage calls’ handover in wireless mesh
networks, a testbed technique [184], combines RSSI (mea-
sure the strength of the received signal) and RTR (as an
indicator of transmission rate quality) to compute the quality
of a wireless link (every 1 second). This allows to monitor
and take the decision of handover (select the access point
with the highest quality level). On one hand, this scheme is
assumed to improve the QoE by 70 %. On the other hand,
it might increase the amount of updates, delays besides it
disregard variable bit rate (VBR).
In another research piece [185], the authors propose a cloud
encoding service and a Hypertext Markup Language revi-
sion 5 (HTML5) for adaptive streaming player. The built
player is a client framework that could be integrated in any
browser. The server side is implemented within a public
cloud infrastructure. It has been claimed that this scheme
promises the elasticity and the scalability needed to suit the
clients, although this approach is specifically destined for
MPEG-DASH. Due to a rapid growth of mobile data traffic
(e.g., mobile videos), the authors of [186] develop some
optimum storage schemes and some dynamic streaming poli-
cies to optimize the video quality, combing caching on a
device and D2D communication to offload the traffic from
cellular network as well as the available storage on mobile
devices. They introduce a framework called reactive Mobile
device Caching (rMDC). Hence, instead of requesting a video
from the base station, in D2D caching network, the user
can request it from neighboring users and might be served
over an unlicensed band. In such a way, D2D candidates
are detected before starting communication session between
devices, using assigned beacon (synchronization or reference
signal sequence) resources by the network. This beacon will
be broadcast in the cell area to allow devices to advertise
their presence and identify each others [187]. Thereby, in
the occurrence of a video request, the device starts searching
its cache and afterwards, it explores the neighbors’ caches
locally to retrieve the desired video. If it does not appear,
the cache agent at the e-NodeB attempts to locate another
mobile device in another group that belongs to the same area.
Finally, if the video is not located in any other neighboring
device, the cache agent will program to get the missing
chunks of the videos from the cache of the e-NodeB if
they exist, else they will be downloaded from the CDN.
Figure -3- indicates the different transition that the mobile
device might take before obtaining the desired video. Here,
the authors have proved that using rMDC along with user
preference profile-based caching, their framework seems to
perform well and reaches high network capacity and better
video QoE for mobile devices. Besides, the distance between
CDN
Situation 1:
Situation 2:
Situation 3:
Situation 4:
Users group 1 Users group 2
4
4
3
2
1
?
E-Node B
FIGURE 3. Request a video file from a neighbor device and get served.
the mobile device and the server hosting the video might be
long and could impact the QoE. In [188], the authors propose
two mechanisms for files duplication: 1) caching (duplicate
copy of a file in different places); and 2) fetching (retrieving
the video to another place or zone) simulated separately in
different scenarios. Based on the observed demand on a given
file, it is selected and the duplication algorithm is activated to
duplicate it at the operators sharing server, to be closer and
more accessible to the user with good quality and minimum
cost. The content fetching seems to be more efficient than
caching, and combining these mechanisms might produce
even better results.
To efficiently bring a given content to end users with a
satisfactory QoE level, the CDN administrator should ensure
that this content is strategically stored/cached across the Web
[189], [190], as this profoundly impacts the user experience.
Storage policy also influences the cost, both in terms of
CAPEX and OPEX, to be paid by the CDN owner. It also
plays a crucial role in offering of CDN as a service (CDNaaS)
[191]. CDNaaS is a platform that could establish virtual
machines (VMs) over a network of data centers and provides
a customized slice of CDN to end users. Moreover, it can
handle a significant number of videos through caches and
streamers hosted at different VMs. The authors formulate
two linear integer solutions for VM placement problem, that
was implemented using Gurobi optimization tool, Efficient
Cost Solution (ECS) and Efficient QoE Solution (EQS).
In terms of maximizing QoE, EQS algorithm shows the
best performance. However, regarding time, ECS algorithm
exhibits better performance, disregarding the number of data
centers and the number of flavors per location.
In order to deliver virtual server resources in a CDNaas archi-
tecture, [192] presents a QoE estimation solution that can be
employed as a part of a QoE-aware system. The developed
system discovers how many users can simultaneously be
VOLUME 4, 2016 13

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2965099, IEEE Access
Bouraqia et al.
handled by a server while granting a satisfactory service
quality level. It aims to capture how the QoE of a video
stream is affected by different factors. The results, based
on PSQA reveals that stream segment duration is an influ-
ential factor, and needs to be taken into account throughout
resource optimization. The system might be used as a part of
the QoE-optimized resource. However, the authors seem to
have overlooked the effect of network bandwidth.
From a different perspective, an optimal rate allocation was
designed by [193] to limit the co-channel interference and
manage resources between D2D and cellular users. Using a
joint encoding rate allocation and a description distribution
optimization forwarded to BS and D2D users (predefined
candidates, who already cached the content, and who are
selected based on their available storage and battery level)
before transmitting video segments to the requester. They
believe that the scheme improves the QoE of video streaming
delivery. Despite, The authors did not consider the additional
delays that would be generated by the optimization process
at the BS level. Also, a dynamic allocation method is adopted
in [43], implementing the shortest path tree, to allocate
joint resources (i.e., video streaming, files, etc.). The results
conclude that selecting the appropriate transmission rate and
the dynamic allocation, could result in an enhanced QoE.
Still, the authors suppose assume the content chunks have
the same size and the transmission rate is the same for all
active nodes, which is not true in real networks scenarios.
The end-to-end communications in Next-Generation Net-
works (NGN) between users and application servers may
cross different networks belonging to different operators and
implementing different technologies, which is challenging in
term of measuring, monitoring and managing the QoE.
According to [194], optimizing the QoE requires that some
factors should be considered like application-level QoS,
allocated capacity, customer premise factors and subjective
user factors. These factors are hard to figure out due to
the difficulties of measuring subjective factors, and some
of the elements degrading the QoE may not be available
for diagnoses. Moreover, crossing several heterogeneous
networks/links makes it hard to determine the element that
induces a poor QoS level. In this regard, the authors build
a framework that can be implemented in NGN, where the
user is able to report the perceived QoE and QoS via
software, which allows the operator to allocate the resources
and reconfigure them accordingly. Nevertheless, the cost
in term of reporting, and the changing in the parameter
might affect the performance significantly. Moreover, some
networks might refuse to join and prefer to manage their
QoE independently. A new dynamic and a reconfigurable
machine-to-machine network is proposed by [195] where
they two metrics are introduced allowing to manage the
wireless network, operational quality of applications and
efficiency of wireless resource utilization. These metrics
allow the network to cover more applications running with
higher QoS level and enhanced QoE metrics. The authors
consider a multiple layer sensing to the proposed system,
so as the platform collects information from each wireless
node in the wide area and then forwards the resulting control
information to the management network entity. Thereby,
the network management decides to optimize the network
topology and so on.
Mobile network operators have a limited spectrum/bandwidth,
and they pay billions of Dollars to obtain time-limited
licenses. Hence, obtaining efficient spectrum usage to get
the required capacity is of great interest both for opera-
tors and end users. Thus, communication network needs
to increase the capacity to cope with the growing demand
for data transmission. The authors of [196], have described
and clearly formulated this problem, and the new areas of
research on infrastructureless communication (e.g., D2D,
M2M, etc.) and small-cells. They also emphasize some
innovative spectrum management options, that permit more
flexible use of spectrum while enabling D2D communication
and deploying small-cells to be candidates to ease such a
flexible usage of spectrum.
Long Term Evolution-Advanced (LTE-A) significantly en-
hanced the spectral efficiency. Yet, the imbalanced traffic
distribution among different cells and the severe congestion
of some of them still a challenging issue. Techniques like
smart-cells [197] and biasing [198] seem to be promising
and might partially solve such a problem. Yet, although they
cannot deal with real-time traffic distribution, authors of
[196] propose a D2D communication-based load balancing
algorithm to increase the ratio of user equipment (UE) that
can access Internet at the same time. This mainly helps
offloading traffic of macro cells via small cells. However
unfortunately, this algorithm could only be utilized for net-
work applications/services, and is not adapted to streaming
service as it suffers from some drawbacks like security issues
and interference management. [199] presents a resource
management algorithm Media Service Ressource Allocation
(MSRA). This scheme schedules limited cellular network
resources based on content popularity, while considering
channel conditions and packet loss rate of D2D direct links.
It also allows to achieve an interesting tradeoff between
the amount of video service delivered and available cellular
resources. Compared to other schemes, MSRA benefits from
a rapid users’ services distribution adjustment, reduces the
impact of D2D underlying interference and enhances the
QoE level. For better QoE fairness over services in LTE/LTE-
A, a self-tuning algorithm [200] is proposed. The key idea
is to repeatedly change the service priority parameters to
(re)prioritize services, and guarantee that all users achieve
the same average QoE regardless of the type of running
service. Depending on whether the objective is to improve
the average service QoE or the individual QoE, the authors
present two algorithms: 1) QoE unweighted approach, and 2)
QoE weighted approach. This way, the appropriate algorithm
14 VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2965099, IEEE Access
Quality of Experience for Streaming Services: Measurements, Challenges and Insights.
is selected according to the preferred objective function.
Thus, if fairness between services is desirable despite the
number of users per service, the unweighted algorithm is
used. Otherwise, the weighted algorithm priorities the pop-
ular services to enhance the user’s QoE by around 15%.
LTE wireless network supports most of M2M Communica-
tion classes. Yet, it faces many challenges like dealing with
a massive number of M2M devices without influencing the
users’ QoE. While LTE scheduler plays an important role,
it does not distinguish between M2M terminals and legacy
UEs. It follows that the radio resources scheduler which
turns to be in favour of M2M terminals over user equipment.
As a solution [201] suggests an M2M-aware hybrid uplink
scheduler to balance the radio resources allocation, which
provides adequate scheduling of M2M terminals without
affecting standard UEs and the perceived QoE. Machine Type
Communication (MTC) allows communication of machines
or devices to machines over mobile networks. It is expected
to exceed billions of M2M connections, still it might overload
the system when a massive number of MTC devices attempt
to connect simultaneously to the mobile network. The prob-
lem is addressed in [202] regarding a Lightweight Evolved
Packet Core (LightEPC) to organize the on-demand creation
of cloud-based lightweight mobile core networks dedicated
for MTC and to simplify the network attach procedure, by
creating an NFV MTC function that implements all the con-
ventional procedures. The latter scheme is shown to exhibit
some nice efficiency and scalability features.
VI. OPEN ISSUES
Although QoE modeling has gained a tremendous attention
recently, it is still a challenging topic due to its multidisci-
plinary and subjective nature. For instance, it is hard to get
access to operators networks data and traces, which makes
it hard to experiment in realistic environments. Also, lack of
open source video database to test quality metrics is being a
high barrier towards understanding, assessing, improving and
controlling the QoE.
A. NEED TO DEVELOP ROBUST AND REALISTIC
MODELS
Most of existing QoE models consider only a few parameters
and not all QoE impacting factors. Whilst many IFs have
been identified, such as user and context (e.g., habits, cultural
background, environment, etc.), should be taken into account
to design a robust and holistic model. Moreover, most re-
viewed articles do not offer a full study on the complexity
of the proposed models from resource allocation (e.g., com-
puting capacity, storage, energy consumption, etc.) perspec-
tive, specifically for handsets like smartphones, reducing the
performance of the suggested assessment applications.
B. NEED TO CONSIDER POWERFUL TOOLS TO
PREDICT AND ASSESS QOE
Throughout this article, we have surveyed a long list of
methods aiming to assess QoE and control it. Unfortunately
none of them seems to perform well under general realistic
settings. Namely, most of schemes suggested in related lit-
erature are only valid for some specific cases, under some
strong assumptions in terms of content, user profile, hand-
set, environment, etc.. Artificial intelligence and machine
learning algorithms have been recently used to measure the
QoE objectively or to improve it. For instance, DASH use
machine learning to set the appropriate resolution and/or bit-
rate according to the channel state. Allowing this way to
continuously track the QoE and proactively take appropriate
actions to keep good user experience. However unfortunately,
few works have used machine learning for hybrid assess-
ment which gives similar results to subjective measurement
approach. This performance collapse is probably due to the
massive amount of required data, computation, verification
and the complexity of the training model. We believe this re-
search direction is still in its infancy and needs to be explored
in depth. Furthermore other powerful tools could be used to
provide a better understanding of the QoE evolution over
space and time. For instance, we believe mean-field game
theory is a promising framework that may allow to model
and track the QoE variation, while capturing the interaction
among active users. More precisely, mean-field game theory
turns to be very efficient in analyzing the behavior of a
massive number of actors under uncertainty (e.g., random
channel, random number of active users, unknown locations
of attractive contents, etc.), by averaging over degrees of free-
dom allowing hereby to deal with a much simpler problem
equivalent to the original complex problem.
C. NEED TO CONSIDER HUMAN AT THE CENTER OF
SERVICE DESIGN PROCESS
Recently new applications/technologies have emerged [203],
requiring an unprecedented requirement in terms of high data
rate and extremely low latency. Consequently, promising the
best possible experience is non-trivial due to diverse factors.
As future applications like Virtual Reality (VR) Augmented
Reality (AR), Mission Critical (MC) services, Tactile Internet
(TI) and teleportation will require a colossal amount of
resources, end users will keep asking for high QoE while
using these apps [204]. The international telecommunication
union [205] has highlighted numerous requirements for the
developing agreement on the usage states and needs of the
emerging services (e.g., e-health, remote tactile control, etc.).
Additionally, the technical infrastructure developments of the
5G communication systems have been evaluated in the con-
text of recent system requirements (e.g., high bandwidth, low
latency, and high-resolution content) and new experiences of
users such as 4K resolution video streaming, TI, and AR/VR.
AR allows people to add digital elements into their exist-
ing environment (e.g., Snapchat, Instagram, PokemonGO,
VOLUME 4, 2016 15

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2965099, IEEE Access
Bouraqia et al.
etc.). Billions of mobile users already heavily used, and
many companies like Apple and Google Glass, Microsoft
HoloLens are encouraging developers to build AR-Apps.
Conversely, VR changes the real world into a virtual one
requiring specific special hardware such as Oculus Rift gear
(expensive and not-portable), which is slowing down its
adoption rate by end users. Moreover, TI [206] will combine
many technologies such as mobile edge computing, AR/VR,
automation, robotics, telepresence etc.. Also, it will permit
the control of the Internet of things (IoT) in real time
while moving and within a particular communication range.
Further, a new dimension will be added to human-to-machine
interaction by enabling tactile and haptic ( sense of touch, in
essence, the manipulation and perception of objects utilizing
touch and proprioception) sensations, and at the same time
transform the interaction of machines. Therefore, assessing
the QoE of such an application would need to consider all
new parameters and will extremely specific QoS (e.g., ultra-
reliability and low-latency) [207].
Inevitably, these emerging applications are changing our
daily life and surrounding environment (e.g., home, work,
etc.), which impacts our perception and understanding of
space and time. Indeed, numerous study such as [210] have
proven that AR increases the learning ability. Earlier to this,
more research must be conducted in various demographic,
geographic areas. To incentivize users to experience and
interact with immersive environments, it is fundamental to
provide seamless services with perfect audio/video data pro-
cessing capabilities. The most crucial performance metrics of
these applications are typically high energy consumption and
long processing delay [208]. To overcome the computational
resource shortage of mobile devices novel techniques like
mobile cloud computing and mobile edge computing are to
be examined to allow users offloading the intensive computa-
tion tasks to several robust cloud servers. However, for more
efficiency, a convenient edge-to-cloud architecture should
be constructed. In this aspect, machine learning techniques
can be applied to approach these difficulties possibly by
using available traces. For example, to anticipate computa-
tional requirements so that devices could minimize latency,
proactive scheduling of the computational resources could be
performed in advance [209].
As mentioned earlier, TI, MC, VR and AR, are new classes of
applications that completely change the way we interact with
reality. It is essential to keep in mind, that they can massively
impact the brain, and affect its perceptions and reasoning,
directly in an obvious manner (e.g., motion sickness, ad-
diction, discomfort, eyestrain, nausea, migraine, etc.) [211].
Thus more studies have to consider these critical issues.
D. ECONOMICS OF QOE
Economics of telecom services has reached maturity as a
tremendous research effort has been spent in developing joint
QoS and pricing models. Most of these models capture the
interaction among competing operators over a shared market
under homogeneous services and inhomogeneous services.
However, all these models only consider strategic pricing
for delivered QoS, and only deals with optimizing CAPEX
and OPEX. Thus, interactive models considering QoE and its
influencing parameters are still to be build. More precisely,
charging end users according to the QoE they receive is of
great importance. Of course, the pricing is assumed to depend
on the delivered QoS but also on the end users’ satisfac-
tion level and context. A deep analysis of the interaction
among content provider, service provider, network provider,
broker and end users is becoming of grand importance. This
interesting research direction is highly inter-disciplinary as
it involves: economics, logistics and demand-supply opti-
mization, flow theory, cognitive science, psychological and
behavioral science.
VII. CONCLUSION
In this article we provide a comprehensive literature review
on QoE, by presenting standard definitions as well the
influencing factors of QoE, that depends mostly on the type
of network, the type of device, content, services and users.
Next, we list major tools and techniques allowing to monitor
and measure/estimate the QoE of a given service. We also
discuss the challenges encountered in wireless networks
and mobile networks (e.g., LTE, LTE-A and 5G), such as
network capacity and varying channel conditions. Then,
we exhibit most impactful solutions from literature. Many
improvement mechanisms and controlling approaches with
promising potential and even effective, are also cited and
analyzed.
With 5G being deployed around the world, providing re-
sponsive networks able to grant high throughput and low
latency is not a challenging issue anymore. However, sup-
porting extremely latency/reliability demanding applications
such as VR/AR and tactile Internet is still to be addressed.
Thus, we believe considerable research efforts need to deal
with developing efficient mechanisms allowing to meet these
requirements.
REFERENCES
[1] W. Cheng, X. Zhang, and H. Zhang, “Statistical-qos driven energy-
efficiency optimization over green 5g mobile wireless networks,” IEEE
Journal on Selected Areas in Communications, vol. 34, no. 12, pp. 3092–
3107, 2016.
[2] M. Seufert, S. Wassermann, and P. Casas, “Considering user behavior
in the quality of experience cycle: Towards proactive qoe-aware traffic
management,” IEEE Communications Letters, 2019.
[3] I. Recommendation, “E. 800: Terms and definitions related to quality
of service and network performance including dependability,” ITU-T
August, vol. 1994, 1994.
[4] O. Markaki, D. Charilas, and D. Nikitopoulos, “Enhancing quality of
experience in next generation networks through network selection mech-
anisms,” in Personal, Indoor and Mobile Radio Communications, 2007.
PIMRC 2007. IEEE 18th International Symposium on, pp. 1–5, 2007.
[5] T. Zhao, Q. Liu, and C. W. Chen, “Qoe in video transmission: A user
experience-driven strategy,” IEEE Communications Surveys & Tutorials,
vol. 19, no. 1, pp. 285–302, 2016.
16 VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2965099, IEEE Access
Quality of Experience for Streaming Services: Measurements, Challenges and Insights.
[6] K. U. R. Laghari and K. Connelly, “Toward total quality of experience: A
QoE model in a communication ecosystem,” Communications Magazine,
IEEE, vol. 50, no. 4, pp. 58–65, 2012.
[7] D. Hands, O. V. Barriac, and F. Telecom, “Standardization activities in
the itu for a QoE assessment of iptv,” IEEE Communications Magazine,
vol. 46, no. 2, pp. 78–84, 2008.
[8] M. Fiedler, K. Kilkki, and P. Reichl, “09192 executive summary–from
quality of service to quality of experience,” in Dagstuhl Seminar Pro-
ceedings, Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 2009.
[9] K. Brunnström, S. A. Beker, K. De Moor, A. Dooms, S. Egger, M.-N.
Garcia, T. Hossfeld, S. Jumisko-Pyykkö, C. Keimel, C. Larabi, et al.,
“Qualinet white paper on definitions of quality of experience,” 2013.
[10] J. Barakovi´
c Husi´
c, S. Barakovi´
c, E. Cero, N. Slamnik, M. O´
cuz, A. De-
dovi´
c, and O. Zupˇ
ci´
c, “Quality of experience for unified communications:
A survey,” International Journal of Network Management, p. e2083,
2019.
[11] A. Mellouk, H. A. Tran, and S. Hoceini, Quality-of-Experience for
Multimedia. John Wiley & Sons, 2013.
[12] P. Juluri, V. Tamarapalli, and D. Medhi, “Measurement of quality of ex-
perience of video-on-demand services: A survey,” IEEE Communications
Surveys & Tutorials, vol. 18, no. 1, pp. 401–418, 2015.
[13] F. Dobrian, A. Awan, D. Joseph, A. Ganjam, J. Zhan, V. Sekar, I. Stoica,
and H. Zhang, “Understanding the impact of video quality on user
engagement,” Communications of the ACM, vol. 56, no. 3, pp. 91–99,
2013.
[14] Y. Xu, S. Elayoubi, E. Altman, R. El-Azouzi, and Y. Yu, “Flow
level QoE of video streaming in wireless networks,” arXiv preprint
arXiv:1406.1255, 2014.
[15] M. A. Usman, S. Y. Shin, M. Shahid, and B. Lövström, “A no reference
video quality metric based on jerkiness estimation focusing on multiple
frame freezing in video streaming,” IETE Technical Review, vol. 34,
no. 3, pp. 309–320, 2017.
[16] S. Jaiswal, G. Iannaccone, C. Diot, J. Kurose, and D. Towsley, “Mea-
surement and classification of out-of-sequence packets in a tier-1 ip
backbone,” IEEE/ACM Transactions on Networking (ToN), vol. 15,
no. 1, pp. 54–66, 2007.
[17] A. A. Laghari, H. He, and M. I. Channa, “Measuring effect of packet re-
ordering on quality of experience (qoe) in video streaming,” 3D Research,
vol. 9, no. 3, p. 30, 2018.
[18] M. T. Vega, C. Perra, and A. Liotta, “Resilience of video streaming
services to network impairments,” IEEE Transactions on Broadcasting,
2018.
[19] M. Yang, S. Wang, R. N. Calheiros, and F. Yang, “Survey on qoe assess-
ment approach for network service,” IEEE Access, vol. 6, pp. 48374–
48390, 2018.
[20] ITU, “Itu g.1011: Reference guide to quality of experience assessment
methodologies,” ITU-T Recommendation, 2016.
[21] M. Matulin and S. Mrvelj, “Subjective evaluation of quality of experi-
ence for video streaming service,” in Research Conference In Technical
Disciplines (RCITD) 2014, 2014.
[22] ITU, “Itu p.801: Mean opinion score (MOS) terminology,” ITU Telecom,
2006.
[23] M. Nilsson, B. Allan, and S. Appleby, “The suitability of young students
for video subjective testing,” in 2018 Tenth International Conference on
Quality of Multimedia Experience (QoMEX), pp. 1–3, IEEE, 2018.
[24] I. Rec, “Bt. 500-11 (2002). methodology for the subjective assessment
of the quality of television pictures,” International Telecommunication
Union, vol. 39, 2009.
[25] I. Recommendation, “500-11,“methodology for the subjective assess-
ment of the quality of television pictures,” recommendation itu-r bt. 500-
11,” ITU Telecom. Standardization Sector of ITU, vol. 7, 2002.
[26] I. Recommendation, “Subjective video quality assessment methods for
multimedia applications,” 2009.
[27] K. Gu, J. Zhou, J.-F. Qiao, G. Zhai, W. Lin, and A. C. Bovik, “No-
reference quality assessment of screen content pictures,” IEEE Transac-
tions on Image Processing, vol. 26, no. 8, pp. 4005–4018, 2017.
[28] T. Hobfeld, M. Hirth, P. Korshunov, P. Hanhart, B. Gardlo, C. Keimel, and
C. Timmerer, “Survey of web-based crowdsourcing frameworks for sub-
jective quality assessment,” in Multimedia Signal Processing (MMSP),
2014 IEEE 16th International Workshop on, pp. 1–6, IEEE, 2014.
[29] B. Naderi, Motivation of workers on microtask crowdsourcing platforms.
Springer, 2018.
[30] Y. Lin and Z. Shih, “Crowdsourcing system on measuring quality of ex-
perience for internet video streaming,” in Consumer Electronics-Taiwan
(ICCE-TW), 2016 IEEE International Conference on, pp. 1–2, IEEE,
2016.
[31] M. Seufert, N. Wehner, and P. Casas, “App for dynamic crowdsourced
QoE studies of http adaptive streaming on mobile devices,” in 2018
Network Traffic Measurement and Analysis Conference (TMA), pp. 1–2,
IEEE, 2018.
[32] S. Egger-Lampl, J. Redi, T. Hoßfeld, M. Hirth, S. Möller, B. Naderi,
C. Keimel, and D. Saupe, “Crowdsourcing quality of experience experi-
ments,” in Evaluation in the Crowd. Crowdsourcing and Human-Centered
Experiments, pp. 154–190, Springer, 2017.
[33] Z. Wang, L. Lu, and A. C. Bovik, “Video quality assessment based on
structural distortion measurement,” Signal processing: Image communi-
cation, vol. 19, no. 2, pp. 121–132, 2004.
[34] Z. Wang, E. P. Simoncelli, and A. C. Bovik, “Multiscale structural
similarity for image quality assessment,” in The Thrity-Seventh Asilomar
Conference on Signals, Systems & Computers, 2003, vol. 2, pp. 1398–
1402, Ieee, 2003.
[35] A. Rehman, K. Zeng, and Z. Wang, “Display device-adapted video
quality-of-experience assessment,” in Human Vision and Electronic
Imaging XX, vol. 9394, p. 939406, International Society for Optics and
Photonics, 2015.
[36] M. H. Pinson and S. Wolf, “A new standardized method for objectively
measuring video quality,” IEEE Transactions on broadcasting, vol. 50,
no. 3, pp. 312–322, 2004.
[37] A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a" completely
blind" image quality analyzer.,” IEEE Signal Process. Lett., vol. 20, no. 3,
pp. 209–212, 2013.
[38] ITUPSNR, “Objective perceptual multimedia video quality measurement
in the presence of a full reference,” ITU Telecom, 2008.
[39] Q. Huynh-Thu and M. Ghanbari, “Scope of validity of psnr in im-
age/video quality assessment,” Electronics Letters, vol. 44, pp. 800–801,
June 2008.
[40] O. Nemethova, M. Ries, M. Zavodsky, and M. Rupp, PSNR-based
estimation of subjective time-variant video quality for mobiles. na, 2006.
[41] “QoE prediction-based interference shaping method,” 2014. CN Patent
App. CN 201,410,231,209.
[42] “Mapping algorithm from psnr (peak signal-to-noise ratio) to MOS
(mean opinion score) in video system,” Aug. 8 2012. CN Patent App.
CN 201,210,109,040.
[43] X. He, K. Wang, H. Huang, T. Miyazaki, Y. Wang, and Y.-f. Sun, “QoE-
driven joint resource allocation for content delivery in fog computing en-
vironment,” in 2018 IEEE International Conference on Communications
(ICC), pp. 1–6, IEEE, 2018.
[44] K. Wang, L. Gu, S. Guo, H. Chen, V. C. Leung, and Y. Sun,
“Crowdsourcing-based content-centric network: a social perspective,”
IEEE network, vol. 31, no. 5, pp. 28–34, 2017.
[45] K. Wang, M. Du, D. Yang, C. Zhu, J. Shen, and Y. Zhang, “Game-theory-
based active defense for intrusion detection in cyber-physical embedded
systems,” ACM Transactions on Embedded Computing Systems (TECS),
vol. 16, no. 1, p. 18, 2016.
[46] Y. Xu, Y. Zhou, and D.-M. Chiu, “Analytical QoE models for bit-rate
switching in dynamic adaptive streaming systems,” Mobile Computing,
IEEE Transactions on, vol. 13, no. 12, pp. 2734–2748, 2014.
[47] F. Wamser, M. Seufert, P. Casas, R. Irmer, P. Tran-Gia, and R. Schatz,
“Yomoapp: A tool for analyzing QoE of youtube http adaptive streaming
in mobile networks,” in Networks and Communications (EuCNC), 2015
European Conference on, pp. 239–243, IEEE, 2015.
[48] M. Seufert, P. Casas, F. Wamser, N. Wehner, R. Schatz, and P. Tran-Gia,
“Application-layer monitoring of QoE parameters for mobile youtube
video streaming in the field,” in Communications and Electronics (ICCE),
2016 IEEE Sixth International Conference on, pp. 411–416, IEEE, 2016.
[49] P. Juluri, L. Plissonneau, and D. Medhi, “Pytomo: a tool for analyz-
ing playback quality of youtube videos,” in Proceedings of the 23rd
International Teletraffic Congress, pp. 304–305, International Teletraffic
Congress, 2011.
[50] K. Bouraqia, E. Sabir, and M. Sadik, “Youtube context-awareness to
enhance quality of experience between yesterday, today and tomorrow:
Survey,” in Wireless Networks and Mobile Communications (WIN-
COM), 2017 International Conference on, pp. 1–7, IEEE, 2017.
[51] B. Chihani, E. Bertin, D. Collange, N. Crespi, T. H. Falk, et al., “User-
centric quality of experience measurement,” in Mobile Computing, Ap-
plications, and Services, pp. 33–46, Springer, 2014.
[52] M. Eckert and T. M. Knoll, “QoE monitoring (QMON) demo proposal.,”
in IM, pp. 1082–1083, 2013.
VOLUME 4, 2016 17

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2965099, IEEE Access
Bouraqia et al.
[53] Y. Xu, Z. Xiao, H. Feng, T. Yang, B. Hu, and Y. Zhou, “Modeling QoE
of video streaming in wireless networks with large-scale measurement of
user behavior,” arXiv preprint arXiv:1604.01675, 2016.
[54] Y. Liu, H. Liu, Y. Li, Y. Shen, J. Wu, and D. Yang, “A software platform
design for objective video QoE assessment,” Journal of Software, vol. 9,
no. 11, pp. 2955–2965, 2014.
[55] Z. Duanmu, K. Zeng, K. Ma, A. Rehman, and Z. Wang, “A quality-of-
experience index for streaming video,” IEEE Journal of Selected Topics
in Signal Processing, vol. 11, no. 1, pp. 154–166, 2017.
[56] P. Casas, M. Seufert, and R. Schatz, “Youqmon: A system for on-
line monitoring of youtube QoE in operational 3G networks,” ACM
SIGMETRICS Performance Evaluation Review, vol. 41, no. 2, pp. 44–
46, 2013.
[57] Q. A. Chen, H. Luo, S. Rosen, Z. M. Mao, K. Iyer, J. Hui, K. Sontineni,
and K. Lau, “Qoe doctor: Diagnosing mobile app qoe with automated ui
control and cross-layer analysis,” in Proceedings of the 2014 Conference
on Internet Measurement Conference, pp. 151–164, ACM, 2014.
[58] A. Zabrovskiy, E. Kuzmin, E. Petrov, C. Timmerer, and C. Mueller, “Ad-
vise: Adaptive video streaming evaluation framework for the automated
testing of media players,” in Proceedings of the 8th ACM on Multimedia
Systems Conference, pp. 217–220, ACM, 2017.
[59] C. Timmerer, A. Zabrovskiy, and A. Begen, “Automated objective
and subjective evaluation of http adaptive streaming systems,” in 2018
IEEE Conference on Multimedia Information Processing and Retrieval
(MIPR), IEEE, 2018.
[60] B. Rainer, M. Waltl, and C. Timmerer, “A web based subjective evalu-
ation platform,” in Quality of Multimedia Experience (QoMEX), 2013
Fifth International Workshop on, pp. 24–25, IEEE, 2013.
[61] T. Mangla, E. Zegura, M. Ammar, E. Halepovic, K.-W. Hwang, R. Jana,
and M. Platania, “Videonoc: assessing video QoE for network operators
using passive measurements,” in Proceedings of the 9th ACM Multimedia
Systems Conference, pp. 101–112, ACM, 2018.
[62] M. Seufert, P. Casas, N. Wehner, L. Gang, and K. Li, “Stream-based
machine learning for real-time qoe analysis of encrypted video streaming
traffic,” in 2019 22nd Conference on Innovation in Clouds, Internet and
Networks and Workshops (ICIN), pp. 76–81, IEEE, 2019.
[63] S. Wassermann, M. Seufert, P. Casas, L. Gang, and K. Li, “Let me decrypt
your beauty: Real-time prediction of video resolution and bitrate for
encrypted video streaming,” 2019.
[64] V. Vasilev, J. Leguay, S. Paris, L. Maggi, and M. Debbah, “Predicting
QoE factors with machine learning,” in 2018 IEEE International Confer-
ence on Communications (ICC), pp. 1–6, IEEE, 2018.
[65] Y.-T. Lin, E. M. R. Oliveira, S. B. Jemaa, and S. E. Elayoubi, “Machine
learning for predicting qoe of video streaming in mobile networks,” in
2017 IEEE International Conference on Communications (ICC), pp. 1–6,
IEEE, 2017.
[66] M. D. F. De Grazia, D. Zucchetto, A. Testolin, A. Zanella, M. Zorzi, and
M. Zorzi, “Qoe multi-stage machine learning for dynamic video stream-
ing,” IEEE Transactions on Cognitive Communications and Networking,
vol. 4, no. 1, pp. 146–161, 2017.
[67] I. Orsolic, D. Pevec, M. Suznjevic, and L. Skorin-Kapov, “A machine
learning approach to classifying youtube qoe based on encrypted network
traffic,” Multimedia tools and applications, vol. 76, no. 21, pp. 22267–
22301, 2017.
[68] M. Lopez-Martin, B. Carro, J. Lloret, S. Egea, and A. Sanchez-
Esguevillas, “Deep learning model for multimedia quality of experience
prediction based on network flow packets,” IEEE Communications Mag-
azine, vol. 56, no. 9, pp. 110–117, 2018.
[69] Z. Duanmu, K. Ma, and Z. Wang, “Quality-of-experience for adap-
tive streaming videos: An expectation confirmation theory motivated
approach,” IEEE Transactions on Image Processing, vol. 27, no. 12,
pp. 6135–6146, 2018.
[70] Q. Wu, H. Li, F. Meng, and K. N. Ngan, “Toward a blind quality
metric for temporally distorted streaming video,” IEEE Transactions on
Broadcasting, 2018.
[71] T. Hoßfeld, R. Schatz, E. Biersack, and L. Plissonneau, “Internet video
delivery in youtube: from traffic measurements to quality of experience,”
in Data Traffic Monitoring and Analysis, pp. 264–301, Springer, 2013.
[72] R. K. Mok, E. W. Chan, and R. K. Chang, “Measuring the quality of
experience of http video streaming,” in Integrated Network Management
(IM), 2011 IFIP/IEEE International Symposium on, pp. 485–492, IEEE,
2011.
[73] D. Z. Rodriguez, J. Abrahao, D. C. Begazo, R. L. Rosa, and G. Bressan,
“Quality metric to assess video streaming service over tcp considering
temporal location of pauses,” IEEE Transactions on Consumer Electron-
ics, vol. 58, no. 3, pp. 985–992, 2012.
[74] F. Li, F. Shuang, Z. Liu, and X. Qian, “A cost-constrained video quality
satisfaction study on mobile devices,” IEEE Transactions on Multimedia,
vol. 20, no. 5, pp. 1154–1168, 2018.
[75] R. Streijl, S. Winkler, and D. Hands, “Mean opinion score (MOS) revis-
ited: methods and applications, limitations and alternatives,” Multimedia
Systems, pp. 1–15, 2014.
[76] D. Pal and T. Triyason, “A survey of standardized approaches towards the
quality of experience evaluation for video services: An itu perspective,”
International Journal of Digital Multimedia Broadcasting, vol. 2018,
2018.
[77] I.-T. T13-SG12, “Temporary document 132-gen (revision 1): Proposed
text for appendix i to rec. p.85 (for agreement),” International Telecom-
munication Union, 2014.
[78] I.-T. R. P.800.1, “Methods for objective and subjective assessment of
speech,” ITU-T July, 2016.
[79] I.-T. R. P.805, “Itu-t recommendation p. 805" in subjective evaluation of
conversational quality,” International Telecommunication Union, 2007.
[80] I.-T. R. P.806, “A subjective quality test methodology using multiple
rating scales,” International Telecommunication Union, 2014.
[81] I.-T. R. P.830, “Subjective performance assessment of telephone-band
and wideband digital codecs,” International Telecommunication Union,
1996.
[82] I.-T. R. P.835, “Subjective test methodology for evaluating speech com-
munication systems that include noise suppression algorithm,” Interna-
tional Telecommunication Union, 2003.
[83] I.-T. R. P.1301, “Subjective quality evaluation of audio and audiovisual
multiparty telemeetings,” International Telecommunication Union, 2017.
[84] I.-T. R. P.1311, “Method for determining the intelligibility of multiple
concurrent talkers,” International Telecommunication Union, 2014.
[85] I.-T. P.831, “Subjective performance evaluation of network echo can-
cellers,” International Telecommunication Union, 1998.
[86] I.-T. P.911, “Subjective audiovisual quality assessment methods for mul-
timedia applications,” International Telecommunication Union, 1998.
[87] I.-T. P.913, “Methods for the subjective assessment of video quality,
audio quality and audiovisual quality of internet video and distribution
quality television in any environment,” International Telecommunication
Union, 2016.
[88] I.-T. R. P.1302, “Subjective method for simulated conversation tests
addressing speech and audiovisual call quality,” International Telecom-
munication Union, 2014.
[89] I. Rec, “Bs. 1116-1: Methods for the subjective assessment of small
impairments in audio systems including multichannel sound systems,”
Int. Telecomm. Union, Geneva Std, 1997.
[90] T. Assembly, “Itu-r bs. 1284-1: En-general methods for the subjective
assessment of sound quality,” tech. rep., ITU, Tech. Rep, 2003.
[91] R. S. ITU, “Recommendation bs. 1534-2: Method for the subjective
assessment of intermediate quality level of audio systems,” 2014.
[92] I. T. Union, “Subjective assessment of the quality of audio in large-screen
digital imagery applications intended for presentation in a theatrical
environment,” 2004.
[93] I.-T. P.912, “Subjective video quality assessment methods for recognition
tasks,” International Telecommunication Union, 2016.
[94] I.-T. P.917, “Subjective test methodolgy for assessing impact of initial
loading delay on user experience,” International Telecommunication
Union, 2019.
[95] I.-R. Recommendatios, “Recommendation 500-10; methodology for the
subjective assessment of the quality of television pictures,” ITU-R Rec.
BT. 500-10, 2000.
[96] I.-T. R. I.-R. BT.1663, “Expert viewing methods to assess the quality of
systems for the digital display of large screen digital imagery in theatres,”
International Telecommunication Union, 2003.
[97] I. Rec, “Bt. 1788,“,” Methodology for the subjective assessment of video
quality in multimedia applications, 2007.
[98] I.-R. B. 2021-1, “Subjective methods for the assessment of stereoscopic
3dtv systems,” 2012.
[99] I.-T. Rec.J.140, “Subjective picture quality assessment for digital cable
television systems,” 1998.
[100] I.-T. Rec.J.245, “Subjective assessment method of picture quality as-
sessed in the home,” 2008.
[101] I.-T. Rec.J.247, “Objective perceptual multimedia video quality measure-
ment in the presence of a full reference,” 2008.
18 VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2965099, IEEE Access
Quality of Experience for Streaming Services: Measurements, Challenges and Insights.
[102] I.-T. P.563, “Single-ended method for objective speech quality assess-
ment in narrow-band telephony applications,” 2004.
[103] I.-T. P.561, “In-service non-intrusive measurement device – voice service
measurements,” 2002.
[104] I.-T. P.561, “Conformance testing for voice over ip transmission quality
assessment models,” 2007.
[105] I.-T. Recommendation, “Perceptual evaluation of speech quality (pesq):
An objective method for end-to-end speech quality assessment of narrow-
band telephone networks and speech codecs,” Rec. ITU-T P. 862, 2001.
[106] I.-T. Recommendation, “Perceptual objective listening quality predic-
tion,” Rec. ITU-T P. 863, 2018.
[107] R. I.-T. G.107, “Transmission systems and media, digital systems and
networks,” 2014.
[108] I.-T. P.920, “Interactive test methods for audiovisual communications,”
2000.
[109] I.-T. P. 1201, “Parametric non-intrusive assessment of audiovisual media
streaming quality,” 2012.
[110] I. T. Union, “Effect of delays on telemeeting quality,” 2016.
[111] I. T. Union, “Spatial audio meetings quality evaluation,” 2017.
[112] I. T. Union, “Method for the measurement of the communication effec-
tiveness of multiparty telemeetings using task performance,” 2014.
[113] T. Thiede, W. C. Treurniet, R. Bitto, C. Schmidmer, T. Sporer, J. G.
Beerends, and C. Colomes, “Peaq-the itu standard for objective mea-
surement of perceived audio quality,” Journal of the Audio Engineering
Society, vol. 48, no. 1/2, pp. 3–29, 2000.
[114] I.-T. G.1070, “Opinion model for video-telephony applications,” Interna-
tional Telecommunication Union, 2018.
[115] I. T. Union, “Quality of experience requirements for telepresence ser-
vices,” 2014.
[116] I. T. Union, “Hybrid perceptual/bitstream models for objective video
quality measurements,” 2018.
[117] I.-T. P. 1202, “Parametric non-intrusive bitstream assessment of video
media streaming quality,” 2012.
[118] A. Raake, M.-N. Garcia, W. Robitza, P. List, S. Göring, and B. Feiten,
“Scalable video quality model for itu-t p. 1203 (aka p. nats) for bitstream-
based monitoring of http adaptive streaming,” QoMEX 2017. to appear.
IEEE., 2017.
[119] I.-T. P. 1401, “Methods, metrics and procedures for statistical evaluation,
qualification and comparison of objective quality prediction models,”
2012.
[120] I.-R. BT.1683, “Itu-r objective perceptual video quality measurement
techniques for standard definition digital broadcast television in the
presence of a full reference,” 2004.
[121] I.-R. BT.1866, “Objective perceptual visual quality measurement tech-
niques for broadcasting applications using low definition television in the
presence of a full reference signal,” 2010.
[122] I.-R. BT.1867, “Objective perceptual video quality measurement tech-
niques for broadcasting applications using low definition television in the
presence of a reduced bandwidth reference,” 2010.
[123] I.-R. BT.1885, “Objective perceptual video quality measurement tech-
niques for standard definition digital broadcast television in the presence
of a reduced bandwidth reference,” 2011.
[124] B. Series, “Objective perceptual video quality measurement techniques
for broadcasting applications using hdtv in the presence of a full reference
signal,” 2012.
[125] I. Recommendation, “Objective video quality measurement techniques
for broadcasting applications using hdtv in the presence of a reduced
reference signal,” 2012.
[126] I.-T. J.143, “This recommendation describes user requirements for objec-
tive perceptual video quality measurements in digital cable television and
similar applications.,” International Telecommunication Union, 2000.
[127] I.-T. J.144, “Objective perceptual video quality measurement techniques
for digital cable television in the presence of a full reference,” Interna-
tional Telecommunication Union, 2004.
[128] I.-T. J.246, “Perceptual audiovisual quality measurement techniques for
multimedia services over digital cable television networks in the presence
of a reduced bandwidth reference,” International Telecommunication
Union, 2008.
[129] I.-T. J.247, “Objective perceptual multimedia video quality measurement
in the presence of a full reference,” International Telecommunication
Union, 2008.
[130] I.-T. J.249, “Perceptual video quality measurement techniques for digital
cable television in the presence of a reduced reference,” International
Telecommunication Union, 2010.
[131] I.-T. J.341, “Objective perceptual multimedia video quality measurement
of hdtv for digital cable television in the presence of a full reference,”
2016.
[132] I. t J.342, “Objective multimedia video quality measurement of hdtv
for digital cable television in the presence of a reduced reference,”
International Telecommunication Union, 2011.
[133] I.-T. G.1022, “Buffer models for media streams on tcp transport,” Inter-
national Telecommunication Union, 2016.
[134] I.-T. G.1071, “Opinion model for network planning of video and audio
streaming applications,” International Telecommunication Union, 2016.
[135] P. Brooks and B. Hestnes, “User measures of quality of experience: why
being objective and quantitative is important,” Network, IEEE, vol. 24,
no. 2, pp. 8–13, 2010.
[136] J. Xie, F. R. Yu, T. Huang, R. Xie, J. Liu, C. Wang, and Y. Liu, “A survey
of machine learning techniques applied to software defined networking
(sdn): Research issues and challenges,” IEEE Communications Surveys
& Tutorials, vol. 21, no. 1, pp. 393–430, 2018.
[137] A. BinSahaq, T. Sheltami, and K. Salah, “A survey on autonomic provi-
sioning and management of qos in sdn networks,” IEEE Access, 2019.
[138] K. Piamrat, C. Viho, J.-M. Bonnin, and A. Ksentini, “Quality of ex-
perience measurements for video streaming over wireless networks,”
in Information Technology: New Generations, 2009. ITNG’09. Sixth
International Conference on, pp. 1184–1189, IEEE, 2009.
[139] M. Ghareeb and C. Viho, “Hybrid QoE assessment is well-suited for
multiple description coding video streaming in overlay networks,” in
Communication Networks and Services Research Conference (CNSR),
2010 Eighth Annual, pp. 327–333, IEEE, 2010.
[140] A. F. Riker, E. S. Aguiar, D. C. Cardoso, P. A. de Oliveira, E. Cerqueira,
and A. J. Abelém, “A hybrid prediction and assessment quality of
experience approach for videostreaming applications over wireless mesh
networks,” in 29th Brazilian Symposium on Computer Networks and
Distributed Systems, 2011.
[141] K.-T. Chen, C.-C. Tu, and W.-C. Xiao, “Oneclick: A framework for
measuring network quality of experience,” in INFOCOM 2009, IEEE,
pp. 702–710, IEEE, 2009.
[142] T. Abar, A. B. Letaifa, and S. El Asmi, “Machine learning based QoE
prediction in sdn networks,” in 2017 13th International Wireless Commu-
nications and Mobile Computing Conference (IWCMC), pp. 1395–1400,
IEEE, 2017.
[143] P. Charonyktakis, M. Plakia, I. Tsamardinos, and M. Papadopouli, “On
user-centric modular QoE prediction for voip based on machine-learning
algorithms,” IEEE Transactions on mobile computing, vol. 15, no. 6,
pp. 1443–1456, 2015.
[144] A. B. Letaifa, “Adaptive QoE monitoring architecture in sdn networks:
Video streaming services case,” in 2017 13th International Wireless Com-
munications and Mobile Computing Conference (IWCMC), pp. 1383–
1388, IEEE, 2017.
[145] M. Seufert, N. Wehner, F. Wamser, P. Casas, A. D’Alconzo, and P. Tran-
Gia, “Unsupervised QoE field study for mobile youtube video streaming
with yomoapp,” in Quality of Multimedia Experience (QoMEX), 2017
Ninth International Conference on, pp. 1–6, IEEE, 2017.
[146] S. Wassermann, N. Wehner, and P. Casas, “Machine learning models
for youtube qoe and user engagement prediction in smartphones,” ACM
SIGMETRICS Performance Evaluation Review, vol. 46, no. 3, pp. 155–
158, 2019.
[147] C. G. Bampis and A. C. Bovik, “Learning to predict streaming
video QoE: Distortions, rebuffering and memory,” arXiv preprint
arXiv:1703.00633, 2017.
[148] C. G. Bampis, Z. Li, A. K. Moorthy, I. Katsavounidis, A. Aaron, and
A. C. Bovik, “Study of temporal effects on subjective video quality of
experience,” IEEE Transactions on Image Processing, vol. 26, no. 11,
pp. 5217–5231, 2017.
[149] I. Bartolec, I. Orsolic, and L. Skorin-Kapov, “In-network youtube per-
formance estimation in light of end user playback-related interactions,”
in 2019 Eleventh International Conference on Quality of Multimedia
Experience (QoMEX), pp. 1–3, IEEE, 2019.
[150] B. Staehle, M. Hirth, R. Pries, F. Wamser, and D. Staehle, “Yomo:
a youtube application comfort monitoring tool,” New Dimensions in
the Assessment and Support of Quality of Experience for Multimedia
Applications, Tampere, Finland, pp. 1–3, 2010.
[151] P. A. Kara, M. G. Martini, and S. Rossi, “One spoonful or multiple
drops investigation of stalling distribution and temporal information for
quality of experience over time,” in Telecommunications and Multimedia
(TEMU), 2016 International Conference on, pp. 1–6, IEEE, 2016.
VOLUME 4, 2016 19

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2965099, IEEE Access
Bouraqia et al.
[152] Z. Ye, R. EL-Azouzi, T. Jimenez, and Y. Xu, “Computing the quality
of experience in network modeled by a markov modulated fluid model,”
arXiv preprint arXiv:1412.2600, 2014.
[153] Y. Xu, E. Altman, R. El Azouzi, M. Haddad, S. Elayoubi, and T. Jimenez,
“Analysis of buffer starvation with application to objective QoE optimiza-
tion of streaming services,” Multimedia, IEEE Transactions on, vol. 16,
no. 3, pp. 813–827, 2014.
[154] K. Bouraqia, W. Attaoui, and E. Sabir, “Solving startup-delay-QoE
dilemma for video streaming services in 5g networks,” in Proceedings
of the Future Technologies Conference, pp. 758–770, Springer, 2018.
[155] H. J. Kim, Y. S. Son, and J. T. Kim, “Kkt-conditions based resource
allocation algorithm for dash streaming service over lte,” in Consumer
Electronics (ICCE), 2018 IEEE International Conference on, pp. 1–3,
IEEE, 2018.
[156] M. S. Bazaraa, H. D. Sherali, and C. M. Shetty, Nonlinear programming:
theory and algorithms. John Wiley & Sons, 2013.
[157] R. Kwan, C. Leung, and J. Zhang, “Proportional fair multiuser scheduling
in lte,” IEEE Signal Processing Letters, vol. 16, no. 6, pp. 461–464, 2009.
[158] K. Xiao, S. Mao, and J. K. Tugnait, “Qoe-driven resource allocation
for dash over ofdma networks,” in Global Communications Conference
(GLOBECOM), 2016 IEEE, pp. 1–6, IEEE, 2016.
[159] C. Timmerer, A. Zabrovskiy, E. Kuzmin, and E. Petrov, “Quality of
experience of commercially deployed adaptive media players,” in Open
Innovations Association (FRUCT), 2017 21st Conference of, pp. 330–
335, IEEE, 2017.
[160] D. Stohr, A. Frömmgen, A. Rizk, M. Zink, R. Steinmetz, and W. Effels-
berg, “Where are the sweet spots?: A systematic approach to reproducible
dash player comparisons,” in Proceedings of the 2017 ACM on Multime-
dia Conference, pp. 1113–1121, ACM, 2017.
[161] Y. Liu, S. Dey, D. Gillies, F. Ulupinar, and M. Luby, “User experience
modeling for dash video,” in Packet Video Workshop (PV), 2013 20th
International, pp. 1–8, IEEE, 2013.
[162] N. Barman and M. G. Martini, “Qoe modeling for HTTP adaptive
video streaming–a survey and open challenges,” IEEE Access, vol. 7,
pp. 30831–30859, 2019.
[163] V. Burger, T. Zinner, L. Dinh-Xuan, F. Wamser, and P. Tran-Gia, “A
generic approach to video buffer modeling using discrete-time analysis,”
ACM Transactions on Multimedia Computing, Communications, and
Applications (TOMM), vol. 14, no. 2s, p. 33, 2018.
[164] S. Schwarzmann, T. Zinner, S. Geissler, and C. Sieber, “Evaluation of the
benefits of variable segment durations for adaptive streaming,” in 2018
Tenth International Conference on Quality of Multimedia Experience
(QoMEX), pp. 1–6, IEEE, 2018.
[165] T. Kimura, M. Yokota, A. Matsumoto, K. Takeshita, T. Kawano, K. Sato,
H. Yamamoto, T. Hayashi, K. Shiomoto, and K. Miyazaki, “QUVE: QoE
maximizing framework for video-streaming,” IEEE Journal of Selected
Topics in Signal Processing, vol. 11, no. 1, pp. 138–153, 2017.
[166] A. Bokani, S. A. Hoseini, M. Hassan, and S. S. Kanhere, “Implementa-
tion and evaluation of adaptive video streaming based on markov decision
process,” in Communications (ICC), 2016 IEEE International Conference
on, pp. 1–6, IEEE, 2016.
[167] J. Zhang, G. Fang, C. Peng, M. Guo, S. Wei, and V. Swaminathan,
“Profiling energy consumption of dash video streaming over 4G LTE
networks,” in Proceedings of the 8th International Workshop on Mobile
Video, p. 3, ACM, 2016.
[168] J. Song, Y. Cui, Z. Li, Y. Bao, L. Zhang, and Y. Zhang, “Edash:
Energy-aware QoE optimization for adaptive video delivery over LTE
networks,” in Computer Communication and Networks (ICCCN), 2016
25th International Conference on, pp. 1–9, IEEE, 2016.
[169] S. Colonnese, S. Russo, F. Cuomo, T. Melodia, and I. Rubin, “Timely
delivery versus bandwidth allocation for dash-based video streaming over
LTE,” IEEE Communications Letters, vol. 20, no. 3, pp. 586–589, 2016.
[170] M. H. Park, J. K. Choi, and J. Choi, “Quality of experience pro-
visioned mobile streaming protocol for the hyper-connected internet
of things devices over the heterogeneous wireless access networks,”
International Journal of Distributed Sensor Networks, vol. 12, no. 9,
p. 1550147716665514, 2016.
[171] S. Zhao, Z. Li, D. Medhi, P. Lai, and S. Liu, “Study of user QoE
improvement for dynamic adaptive streaming over http (mpeg-dash),”
in Computing, Networking and Communications (ICNC), 2017 Interna-
tional Conference on, pp. 566–570, IEEE, 2017.
[172] Google, “Exoplayer.” https://google.github.io/ExoPlayer/guide.html, On-
line; accessed Mar-2016.
[173] M. S. Mushtaq, S. Fowler, B. Augustin, and A. Mellouk, “QoE in 5g
cloud networks using multimedia services,” in IEEE Wireless Communi-
cation and Networking Conference (WCNC’16), 2016.
[174] T. Nunome and S. Tasaka, “QoE enhancement by capacity allocation
and piggyback bandwidth request in audio-video ip transmission over the
ieee 802.16 be service,” in Wireless Communications and Networking
Conference (WCNC), 2012 IEEE, pp. 1245–1250, IEEE, 2012.
[175] S. Dutta, T. Taleb, and A. Ksentini, “QoE-aware elasticity support in
cloud-native 5g systems,” in Communications (ICC), 2016 IEEE Inter-
national Conference on, pp. 1–6, IEEE, 2016.
[176] T. Zinner, T. Hossfeld, T. N. Minash, and M. Fiedler, “Controlled vs.
uncontrolled degradations of QoE–the provisioning-delivery hysteresis
in case of video,” New Dimensions in the Assessment and Support of
Quality of Experience (QoE) for Multimedia Applications, 2010.
[177] M. Fiedler, T. Hoßfeld, and P. Tran-Gia, “Quality of experience-related
differential equations and provisioning-delivery hysteresis,” in IEEE
Network, Special requirements for Mobile TV, 13th ACM, International
Conference on Multimedia, p. 3, 2010.
[178] T. Taleb and A. Ksentini, “QoS/QoE predictions-based admission con-
trol for femto communications,” in Communications (ICC), 2012 IEEE
International Conference on, pp. 5146–5150, IEEE, 2012.
[179] J. Nightingale, Q. Wang, J. M. Alcaraz Calero, E. Chirivella-Perez,
M. Ulbricht, J. A. Alonso-López, R. Preto, T. Batista, T. Teixeira,
M. J. Barros, et al., “QoE-driven, energy-aware video adaptation in 5g
networks: The selfnet self-optimisation use case,” International Journal
of Distributed Sensor Networks, vol. 2016, p. 4, 2016.
[180] M. Shefkiu, S. Cakaj, and A. Maraj, “Quality of experience (qoe) im-
provement by video caching implementation,” in 2017 South Eastern Eu-
ropean Design Automation, Computer Engineering, Computer Networks
and Social Media Conference (SEEDA-CECNSM), pp. 1–5, IEEE, 2017.
[181] W. Li, S. M. Oteafy, and H. S. Hassanein, “Streamcache: Popularity-
based caching for adaptive streaming over information-centric networks,”
in Communications (ICC), 2016 IEEE International Conference on,
pp. 1–6, IEEE, 2016.
[182] C. Ge, N. Wang, S. Skillman, G. Foster, and Y. Cao, “QoE-driven dash
video caching and adaptation at 5G mobile edge,” in Proceedings of
the 2016 conference on 3rd ACM Conference on Information-Centric
Networking, pp. 237–242, ACM, 2016.
[183] I. Farris, T. Taleb, M. Bagaa, and H. Flick, “Optimizing service repli-
cation for mobile delay-sensitive applications in 5g edge network,” in
Communications (ICC), 2017 IEEE International Conference on, pp. 1–
6, IEEE, 2017.
[184] I. V. S. Brito and G. B. Figueiredo, “Improving QoS and QoE through
seamless handoff in software-defined IEEE 802.11 mesh networks,”
IEEE Communications Letters, vol. 21, no. 11, pp. 2484–2487, 2017.
[185] C. Timmerer, D. Weinberger, M. Smole, R. Grandl, C. Müller, and
S. Lederer, “Transcoding and streaming-as-a-service for improved video
quality on the web,” in Proceedings of the 7th International Conference
on Multimedia Systems, p. 37, ACM, 2016.
[186] A. F. Molisch, G. Caire, D. Ott, J. R. Foerster, D. Bethanabhotla,
and M. Ji, “Caching eliminates the wireless bottleneck in video aware
wireless networks,” Advances in Electrical Engineering, vol. 2014, 2014.
[187] G. Fodor, E. Dahlman, G. Mildh, S. Parkvall, N. Reider, G. Miklós,
and Z. Turányi, “Design aspects of network assisted device-to-device
communications,” Communications Magazine, IEEE, pp. 170–177, 2012.
[188] E. Abd-Elrahman, T. Rekik, and H. Afifi, “Optimization of quality
of experience through file duplication in video sharing servers,” arXiv
preprint arXiv:1502.01120, 2015.
[189] S. Dutta, T. Taleb, P. A. Frangoudis, and A. Ksentini, “On-the-fly QoE-
aware transcoding in the mobile edge,” in Global Communications Con-
ference (GLOBECOM), 2016 IEEE, pp. 1–6, IEEE, 2016.
[190] P. A. Frangoudis, L. Yala, A. Ksentini, and T. Taleb, “An architecture
for on-demand service deployment over a telco cdn,” in Communications
(ICC), 2016 IEEE International Conference on, pp. 1–6, IEEE, 2016.
[191] S. Retal, M. Bagaa, T. Taleb, and H. Flinck, “Content delivery network
slicing: QoE and cost awareness,” in Proc. IEEE ICC, 2017.
[192] L. Koskimies, T. Taleb, and M. Bagaa, “QoE estimation-based server
benchmarking for virtual video delivery platform,” in Communications
(ICC), 2017 IEEE International Conference on, pp. 1–6, IEEE, 2017.
[193] N.-S. Vo, T. Q. Duong, H. D. Tuan, and A. Kortun, “Optimal video
streaming in dense 5g networks with D2D communications,” IEEE
Access, vol. 6, pp. 209–223, 2018.
20 VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2020.2965099, IEEE Access
Quality of Experience for Streaming Services: Measurements, Challenges and Insights.
[194] J. Zhang and N. Ansari, “On assuring end-to-end QoE in next generation
networks: challenges and a possible solution,” Communications Maga-
zine, IEEE, vol. 49, no. 7, pp. 185–191, 2011.
[195] K. Yano, M. Uno, T. Miyasaka, K. Oshima, Y. Tanaka, K. Murakami,
T. Yamamoto, H. Yamamoto, Y. Okada, K. Kobayashi, et al., “Dynamic
and reconfigurable M2M wireless network with QoE-based control,”
in Future Network and Mobile Summit (FutureNetworkSummit), 2013,
pp. 1–9, IEEE, 2013.
[196] J. Liu, Y. Kawamoto, H. Nishiyama, N. Kato, and N. Kadowaki, “Device-
to-device communications achieve efficient load balancing in LTE-
advanced networks,” Wireless Communications, IEEE, vol. 21, no. 2,
pp. 57–65, 2014.
[197] T. Nakamura, S. Nagata, A. Benjebbour, Y. Kishiyama, T. Hai, S. Xi-
aodong, Y. Ning, and L. Nan, “Trends in small cell enhancements in LTE
advanced,” Communications Magazine, IEEE, vol. 51, no. 2, pp. 98–105,
2013.
[198] A. Bou Saleh, O. Bulakci, S. Redana, B. Raaf, and J. Hämäläinen,
“Enhancing LTE-advanced relay deployments via biasing in cell selection
and handover decision,” in Personal Indoor and Mobile Radio Com-
munications (PIMRC), 2010 IEEE 21st International Symposium on,
pp. 2277–2281, IEEE, 2010.
[199] M. Chen, L. Wang, J. Chen, and X. Wei, “QoE-driven D2D media
services distribution scheme in cellular networks,” Wireless Communi-
cations and Mobile Computing, vol. 2017, 2017.
[200] P. Oliver-Balsalobre, M. Toril, S. Luna-Ramírez, and J. M. R. Avilés,
“Self-tuning of scheduling parameters for balancing the quality of expe-
rience among services in LTE,” EURASIP Journal on Wireless Commu-
nications and Networking, vol. 2016, no. 1, p. 1, 2016.
[201] I. Abdalla and S. Venkatesan, “A QoE preserving M2M-aware hy-
brid scheduler for LTE uplink,” in Mobile and Wireless Network-
ing (MoWNeT), 2013 International Conference on Selected Topics in,
pp. 127–132, IEEE, 2013.
[202] T. Taleb, A. Ksentini, and A. Kobbane, “Lightweight mobile core
networks for machine type communications,” Access, IEEE, vol. 2,
pp. 1128–1137, 2014.
[203] A. Lounis, F. Alilat, and N. Agoulmine, “Neural network model of qoe
for estimation video streaming over 5g network,” in 6th International
Workshop on ADVANCEs in ICT Infrastructures and Services (AD-
VANCE 2018), 2018.
[204] Z. Akhtar, K. Siddique, A. Rattani, S. L. Lutfi, and T. H. Falk, “Why
is multimedia quality of experience assessment a challenging problem?,”
IEEE Access, vol. 7, pp. 117897–117915, 2019.
[205] I. Vision, “Framework and overall objectives of the future development
of imt for 2020 and beyond,” International Telecommunication Union
(ITU), Document, Radiocommunication Study Groups, 2015.
[206] O. Holland, E. Steinbach, R. V. Prasad, Q. Liu, Z. Dawy, A. Aijaz,
N. Pappas, K. Chandra, V. S. Rao, S. Oteafy, et al., “The ieee 1918.1
"tactile internet"standards working group and its standards,” Proceedings
of the IEEE, 2019.
[207] Y. Kabalci, “5g mobile communication systems: Fundamentals, chal-
lenges, and key technologies,” in Smart Grids and Their Communication
Systems, pp. 329–359, Springer, 2019.
[208] J. Ren, Y. He, G. Huang, G. Yu, Y. Cai, and Z. Zhang, “An edge-
computing based architecture for mobile augmented reality,” IEEE Net-
work, 2019.
[209] H. Zhang, Y. Ren, K.-C. Chen, L. Hanzo, et al., “Thirty years of machine
learning: The road to pareto-optimal next-generation wireless networks,”
arXiv preprint arXiv:1902.01946, 2019.
[210] S. Cai, E. Liu, Y. Yang, and J.-C. Liang, “Tablet-based ar technology:
Impacts on students’ conceptions and approaches to learning mathe-
matics according to their self-efficacy,” British Journal of Educational
Technology, 2019.
[211] D. You, T. V. Doan, R. Torre, M. Mehrabi, A. Kropp, V. Nguyen,
H. Salah, G. T. Nguyen, and F. H. Fitzek, “Fog computing as an enabler
for immersive media: Service scenarios and research opportunities,”
IEEE Access, vol. 7, pp. 65797–65810, 2019.
VOLUME 4, 2016 21