Content uploaded by Pedro Luis Mateo Navarro
Author content
All content in this area was uploaded by Pedro Luis Mateo Navarro on Jan 04, 2014
Content may be subject to copyright.
A Context-aware Model for the Analysis of User
Interaction and QoE in Mobile Environments
Pedro Mateo
C´
atedra SAES Laboratories
University of Murcia
Murcia, Spain
email: pedromateo@um.es
Diego Sevilla Ruiz
Computer Engineering Department
University of Murcia
Murcia, Spain
email: dsevilla@um.es
Gregorio Mart´
ınez P´
erez
Communications Engineering Department
University of Murcia
Murcia, Spain
email: gregorio@um.es
Abstract—This paper describes a novel approach to model
quality of experience (QoE) in mobile environments. A meta-
model is created to establish a set of parameters to dynamically
describe the interaction between the user and the system, as well
as the context in which it is carried and the attractiveness of that
process. A uniform representation of user-system interaction in
mobile contexts is provided. This helps user-related applications
to determine QoE of users, and allows the comparison between
different interaction records. Its run-time nature also allows
context-aware applications to make model-based decisions in
real-time to adapt themselves, and thus providing a better
experience to users. As a result, this meta-model provides unified
criteria for the inference and analysis of QoE in mobile contexts,
as well as for implementing user profiling based on successful-
QoE experiences.
Index Terms—quality of experience; context-aware; QoE mod-
eling; interaction modeling; user profiling.
I. INTRODUCTION
Quality of Experience (QoE) is a subjective measure of
users’ experiences with a service. It focuses on those aspects
that users directly perceive as quality parameters, and that
finally decide the acceptability of a service. QoE is not
just related to Quality of Service (QoS), but it is a broader
construct beyond technical and objective system performance
metrics [1], [2], [3], [4]. It encompasses users’ behavioural,
cognitive, and psychological states along with the context in
which the services are provided to them. This is particularly
true in mobile contexts, where applications are dynamically
used in different scenarios and social contexts [4].
Moreover, context-aware systems extract, interpret, and use
context information to adapt their functionality to the current
context of use [5]. By “context” we mean any information
used to characterize the situation of an entity, e.g., person,
place, or object, and that is considered relevant for the user-
system interaction analysis. Hong et al. [6] differentiate among
external context, which involves data referring to the physical
environment, e.g., location, sound, time of the day, etc., and
internal context, which involves data related to the cognitive
domains of the user, e.g., emotional state.
Complexity of interaction within mobile scenarios has in-
creased dramatically in the last few years. Users and their
handheld devices are continuously moving in several simul-
taneous fuzzy contexts [6]. This dynamic environment sets
special requirements for mobile applications’ usability and
the acceptance of such systems. A close relationship between
interaction, its context, and QoE can be found in these environ-
ments. However, the lack of a uniform approach for modeling
the information related to interaction within a specific context
is obvious [7]. This is why this work proposes incorporating
user ratings and context-aware information into user-system
interaction analysis methods, providing a uniform basis to
quantify interaction within mobile scenarios, and use it to
determine QoE.
However, incorporating these data into user-system interac-
tion analysis processes poses several problems. One of them is
deciding what parameters are useful to capture QoE in mobile
contexts [4]. We consider essential that such parameters have
to be collected, as far as possible, by using current devices’
capabilities. The low standardization of technologies used in
context-aware systems is also a problem [6]. A common rep-
resentation of the context of applications is needed to support
standard analysis and decision processes, as well as to support
cooperation between different QoE analysis applications. A
related problem is how to build “standard” user-interaction
profiles, which are used by applications or systems to adapt
themselves to provide a better QoE.
According to these problems, the following research ques-
tions are posed:
Q1: How can rating and context information be properly
incorporated into interaction analysis processes?
Q2: How can QoE of different users be compared to each
other, as well as QoE inferred from different systems and/or
contexts?
Q3: Is it possible to build a user’s interaction profile based on
successful-QoE experiences?
To answer these questions, this paper briefly describes
the design of a meta-model arranging dynamic interaction
parameters. This meta-model is augmented with parameters
describing users’ perception of interaction quality, as well as
context information. It provides a uniform representation of
user-system interaction within real mobile contexts. Instances
of this meta-model provide a basis to determine and compare
QoE of users in such contexts, as well as to make decisions
at run-time to provide a better users’ experience.
The rest of the document is structured as follows. Section II
124Copyright (c) IARIA, 2012. ISBN: 978-1-61208-232-5
CENTRIC 2012 : The Fifth International Conference on Advances in Human-oriented and Personalized Mechanisms, Technologies, and Services
describes an overview of the meta-model design mentioned
above. Section III describes some considerations about the
implementation and the applications of the meta-model in
current mobile scenarios. Finally, Section IV includes some
conclusion and future lines of work.
II. QOE-AWARE INTERACTION ANALYSIS
The approach described in this paper tries to give an answer
to the three research questions posed above. First, to answer
research question Q1, the design of a meta-model including
interaction, user-rating and context data is proposed. It struc-
tures all these data to be the basis for the implementation of
QoE analysis and inference processes. We propose to base this
meta-model design on an existing one [8], [9] which is being
developed in parallel to this work, in a joint effort between
the C´
atedra SAES [10] and the Telekom Innovation Labora-
tories [11] to quantify interaction in multimodal contexts.
The base meta-model describes interaction by turn, i.e.,
each time the user or the system take part in the dialog,
following a dialog structure, i.e., a set of ordered system-
and user-turns. This “step by step” description of interaction
creates a relationship between data and time, providing new
opportunities for the dynamic analysis of interaction. This
meta-model handles different modalities at the same level
using common metrics to describe interaction. These metrics
are structured within a common representation, allowing the
comparison among different interaction records. Collected
data (turn content, meta-communication, I/O information, and
modality description) are partly based on the well validated
parameters and concepts described in [12], [13].
Since the base meta-model only quantifies user-system
multimodal interaction, it was extended in to ways. On one
hand, user-rating parameters as the used in questionnaires like
AttrakDiff [14] were added to measure how attractive and
user-friendly is the product under test. On the other hand,
the model was extended with new parameters to describe the
interaction context in mobile scenarios. Thus, the model not
only provides a link between interaction data and time, but new
links between interaction and, for example, user’s opinions,
information about location, social context, device features, etc.
are created. Figure 1 depicts the kind of parameters considered
by the proposed design.
Human-computer interaction parameters —included in the
base meta-model— are used to quantify the interaction of the
user with the system, e.g., quantity of information provided by
the system, average reaction time of the user. User rating pa-
rameters are used to measure the experience of users with the
system under test, e.g., motivating, human, clearly structured.
They measure how attractive the system/application is in terms
of usability and appearance. The validity of questionnaires like
AttrakDiff as a method to extract users’ experience is shown
in related work, e.g., [15].
Communication parameters describe the features of device
connectivity, e.g., if the user is on-line or not, connection
bandwidth. Location and time parameters describe the posi-
tion of the user while interacting the system, as well as a
Figure 1. Design overview of the proposed meta-model.
time reference, e.g., the user is moving, it is Friday night.
User parameters describe peculiarities of the person using the
application and the device, e.g., gender, age, disabilities. Social
context parameters are those that can be extracted from social
media, and are interesting for the interaction analysis, e.g., if
the user is along with their friends, if he/she is within an office
context. Finally, Device parameters describe the peculiarities
of the device being used, e.g., screen size, input and output
methods.
Some of these parameters are run-time and have to be
collected many times during interaction, e.g., quantity of user
input at a specific time. Others are not, and are collected
only once, e.g., screen resolution. This feature is specially
relevant to know where the parameters are included in the
model design. Run-time ones are included at turn-level, while
static ones are included at dialog level.
Instances of this meta-model describe user-system interac-
tion within its mobile context, and are ready to support further
analysis, comparison, transformation, and decision processes.
The great majority of the parameters described above are
intended to be collected automatically, e.g., by using tools
like the Android HCI Extractor [16] which extracts multi-
modal interaction data at run-time. However, those based on
subjective judgments of the user or the expert have to be
manually annotated in most of cases. Anyway, automatically
collected or not, the same metrics —structured into a common
representation— are used to quantify the interaction between
the user and the system. This provides experts and tools with
unified criteria to describe the interaction process. Different
interaction records can be analyzed and compared regardless
of the system/application under testing, the interaction context,
even the modalities used to provide input and output data. This
answers research question Q2.
To answer question Q3, the reader can consider an instance
of the meta-model described above as a three-dimensional rep-
resentation of an interaction “occurrence”. These dimensions
125Copyright (c) IARIA, 2012. ISBN: 978-1-61208-232-5
CENTRIC 2012 : The Fifth International Conference on Advances in Human-oriented and Personalized Mechanisms, Technologies, and Services
are the user who is performing the interaction (u∈U,Uthe
set of all users under study), the application or system with
which the user interacts (a∈A,Athe set of all applications or
systems under study), and the context in which the interaction
process is performed (c∈C,Cthe set of all the possible
contexts under study.) Let Ibe a set of instances of the model
described above, each instance in∈Irepresents one of the
interaction occurrences in the space (ui, ai, ci). If we consider
all the instances inin the space (ui), i.e., all the interaction
occurrences of the user ui, these can be used to build a “user
interaction profile” representing the features of the behavior
of this specific user.
Maybe using only one dimension to analyze interaction is
not useful to make decisions. However, if for example the
interaction occurrences in the space (ui, ai)are used to build
a profile, we can analyze the behavior and ratings of a user
using the same application in different contexts. Thus, the
application can be adapted to the current context, as mentioned
in [4]. Using the occurrences in the space (ui, ci)allow us
to analyze how the user behaves in a specific context, and
those in the space (ai, ci)are valid to analyze how different
users interact with an application in a specific context. The
process gets more complex if sub-dimensions of these three
dimensions are considered, but this is not the goal of this
paper. This shows that instances of the proposed meta-model
are valid to create different interaction profiles not only related
to users, but also related to the system/application in use or
the interaction context.
Finally, we consider essential the easy incorporation of the
proposed solution into current mobile devices, i.e., tablets,
smartphones, etc. From a practical point of view, developed
solutions should be easily built into the kind of devices used
nowadays. This not only fosters using such a kind of testing
tools into current applications and systems, but it will ease
the full implementation of context- and QoE-aware methods in
real life, and not only for laboratory environments [6]. This is
why we argue for using current devices capabilities to collect
interaction and context data, e.g., the windowing system to
collect touch interaction metrics, internet-based applications
to get social information, device’s sensors as GPS to get
position. Even users’ perception of interaction quality when
possible, e.g., encouraging the user to rate interaction at run-
time. Therefore, advanced sensors are not required to fill the
model instances.
III. META- MO DE L IMPLEMENTATION AND APPLICATIONS
Our implementation of the base meta-model will use the
facilities offered by the Eclipse Modeling Framework (EMF).
The widely used EMF allows the definition of comprehen-
sible, flexible and extensible meta-models, as well as the
syntactically validation of concrete model instances. Tools like
EMF help to make the modeling process more effective, and
provide indispensable functionality to validate and extend the
meta-model [17]. EMF provides model transformation and
automatic code generation functionality as well. The design
of the meta-model will be used to automatically generate the
source code, which has to be integrated into the applications
in which new model instances will be created.
To collect interaction and context parameters automatically,
it is proposed using a tool based on the Android HCI Ex-
tractor. [16] This open-source tool can be extended to collect,
as far as the device allows, all the interaction and context
data necessary to fill model instances. User ratings might be
automatically collected as well, e.g., by showing question-
naires after test trials. This tool will be used to create model
instances at run-time. Such instances are valid to represent
many different interaction scenarios, from the usage of an
application during some minutes, to the usage of a device
during hours. Despite the architecture and behavior of the HCI
Extractor can be ported to other mobile systems —it uses
a reduced EMF Java implementation that can run in many
platforms— currently only Android is supported. However,
Android can run in many different mobile platforms, e.g.,
smartphones, tablets, netbooks, smart-tv, etc.
Once created, the instances provide a basis on which to
implement different analysis and evaluation processes. QoE in-
ference is the first result that comes to mind. The data included
into a model instance can be used to systematically determine
the QoE of a user within a mobile context. Interaction data can
be fused with context information and users/experts ratings to
determine QoE, e.g., by using Bayesian networks as in [4]
Moreover, thanks to the run-time nature of the meta-model,
QoE can be estimated in real-time. In case the resulting value
is not the expected, the interaction history and the context can
be analyzed for a specific interval of time to make a decision
that makes QoE to improve, e.g., by adjusting microphone
settings, changing screen brightness.
As a common representation is used, different instances can
be easily compared to each other, e.g., to detect why QoE
worsens when using an application in a different scenario,
to know why an application provides a better QoE than
another. Finally, model transformation processes can be im-
plemented using the high expressiveness of EMF-based tools,
e.g., ATL [18]. Original model instances can be transformed
into instances of other meta-models, which provide different
perspectives of the data collected during the interaction pro-
cess, e.g., a summary meta-model linking only context and
ratings data, a statistical meta-model aggregating users ratings.
Thus, model transformation is valid to build user interaction
profiles as well. Let Model Bbe a meta-model describing the
user interaction profile. Several instances of the meta-model
proposed in this paper (say Model A) can be used to build an
instance of the new model by simply using ATL transformation
rules. The process is completely automatic, as data in Model A
instances are used by ATL to fill data fields in the Model B
instance according to the rules.
Some validations tests were conducted in the context of the
PALADIN project [8], [9]. The participants used multimodal
input (speech + touch) to book a restaurant within an Android
smartphone. These tests where used to show the validity of
the interaction meta-model on which the proposed solution is
based. Similar tests can be conducted to show the validity of
126Copyright (c) IARIA, 2012. ISBN: 978-1-61208-232-5
CENTRIC 2012 : The Fifth International Conference on Advances in Human-oriented and Personalized Mechanisms, Technologies, and Services
the solution proposed in this work, but now using a context-
dependant application and collecting real or simulated context
data. Then, the model instances will be used to determine QoE
of each participant using different modalities or a combination
of them.
IV. CONCLUSIONS AND FUTURE WOR K
The design of a meta-model including data about user-
system interaction and its context is described. Most of the
parameters included in this design are collected using current
devices’ capabilities; subjective ones have to be annotated
manually. This meta-model provides a common representation
of interaction, allowing the inference, analysis and comparison
of QoE in mobile contexts. A relationship between user-system
interaction, its context, and users’ perception of quality is
created. Moreover, a strong relationship between these data
and time is provided as well, opening up new opportunities for
the dynamic analysis of QoE. Its dynamic nature also allows
to make QoE-aware decisions at run-time. Instances of this
meta-model are also valid to create user interaction profiles
based on successful-QoE experiences.
This approach poses lots of challenges with which to deal
until reaching its final implementation. One of them is treating
the variety, diversity and big amount of interaction and context
data. This solution is not aimed at modeling “the entire world”,
but only data that is relevant for the analysis of QoE have to
be considered. A well balanced set of parameters has to be
chosen for the model design. This set should be as small as
possible, but enough to determine QoE in mobile contexts.
Choosing an adequate abstraction level for the parameters
is also very important, e.g., the user is in a specific geographic
coordinate vs. the user is in the office. Dealing with cognitive
data automatically is also challenging. Emotional state of the
user and cognitive elements have to be incorporated into the
meta-model, as well as users’ quality perception and experts’
verdicts. Users’ security and privacy problem should be also
posed and discussed.
ACK NOW LE DG EM EN T
This paper has been partially funded by the C´
atedra SAES
of the University of Murcia initiative, a joint effort between
SAES [19] and the University of Murcia [20] to work on
open-source software, and real-time and critical information
systems.
REFERENCES
[1] S. M¨
oller, K.-P. Engelbrecht, C. K¨
uhnel, I. Wechsung, and B. Weiss, “A
Taxonomy of Quality of Service and Quality of Experience of Multi-
modal Human-Machine Interaction,” in First International Workshop on
Quality of Multimedia Experience (QoMEX’09), July 2009, pp. 7–12.
[2] W. Wu, M. A. Arefin, R. Rivas, K. Nahrstedt, R. M. Sheppard, and
Z. Yang, “Quality of experience in distributed interactive multimedia
environments: toward a theoretical framework,” in ACM Multimedia,
W. Gao, Y. Rui, A. Hanjalic, C. Xu, E. G. Steinbach, A. El-Saddik,
and M. X. Zhou, Eds. ACM, 2009, pp. 481–490. [Online]. Available:
http://dblp.uni-trier.de/db/conf/mm/mm2009.html#WuARNSY09
[3] K.-T. Chen, C.-J. Chang, C.-C. Wu, Y.-C. Chang, and C.-L. Lei,
“Quadrant of euphoria: a crowdsourcing platform for qoe assessment.”
IEEE Network, vol. 24, no. 2, pp. 28–35, 2010. [Online]. Available: http:
//dblp.uni-trier.de/db/journals/network/network24.html#ChenCWCL10
[4] K. Mitra, A. B. Zaslavsky, and C. ˚
Ahlund, “A probabilistic context-
aware approach for quality of experience measurement in pervasive
systems.” in SAC, W. C. Chu, W. E. Wong, M. J. Palakal, and
C.-C. Hung, Eds. ACM, 2011, pp. 419–424. [Online]. Available:
http://dblp.uni-trier.de/db/conf/sac/sac2011.html#MitraZA11
[5] H. Byun and K. Cheverst, “Utilizing context history to provide dynamic
adaptations.” Applied Artificial Intelligence, vol. 18, no. 6, pp. 533–548,
2004. [Online]. Available: http://dblp.uni-trier.de/db/journals/aai/aai18.
html\#ByunC04
[6] J. Hong, E. Suh, and S. Kim, “Context-aware systems: A literature
review and classification,” Expert Systems with Applications, vol. 36,
no. 4, pp. 8509 – 8522, 2009. [Online]. Available: http://www.
sciencedirect.com/science/article/pii/S0957417408007574
[7] C. Bolchini, C. A. Curino, E. Quintarelli, F. A. Schreiber, and
L. Tanca, “A data-oriented survey of context models,” SIGMOD
Rec., vol. 36, pp. 19–26, December 2007. [Online]. Available:
http://doi.acm.org/10.1145/1361348.1361353
[8] P. Mateo and S. Hillmann, “PALADIN: a Run-time Model for
Automatic Evaluation of Multimodal Interfaces,” 2011. [Online].
Available: http://www.catedrasaes.org/wiki/MIM
[9] ——, “Model-based Measurement of Human-Computer Interaction in
Mobile Multimodal Environments,” in NordiCHI. ACM, 2012. [On-
line]. Available: http://www.prometei.de/fileadmin/ammi-nordichi2012/
04ammi12 mateo hillmann.pdf
[10] C´
atedra SAES de la Unviersidad de Murcia, http://www.catedrasaes.org,
[Online; accessed Oct. 2012].
[11] Telekom Innovation Laboratories, http://www.laboratories.telekom.com,
[Online; accessed Oct. 2012].
[12] ITU-T Suppl. 24 to P-Series Rec., “Parameters describing the interaction
with spoken dialogue systems,” Geneva, Switzerland, October 2005.
[13] C. K¨
uhnel, B. Weiss, and S. M. ller, “Parameters describing multimodal
interaction - Definitions and three usage scenarios,” in Proceedings of the
11th Annual Conference of the ISCA (Interspeech 2010), T. Kobayashi,
K. Hirose, and S. Nakamura, Eds. Makuhari, Japan: ISCA, 2010, pp.
2014–2017.
[14] M. Hassenzahl, M. Burmester, and F. Koller, “AttrakDiff:
Ein Fragebogen zur Messung wahrgenommener hedonischer und
pragmatischer Qualit¨
at. (A questionnaire for measuring perceived
hedonic and pragmatic quality.),” in Mensch & Computer, G. Szwillus
and J. Ziegler, Eds. Teubner, 2003, pp. 187–196. [Online]. Available:
http://dblp.uni-trier.de/db/conf/mc/mc2003.html#HassenzahlBK03
[15] I. Wechsung, K.-P. Engelbrecht, A. B. Naumann, S. Schaffer, J. Seebode,
F. Metze, and S. M¨
oller, “Predicting the quality of multimodal systems
based on judgments of single modalities.” in INTERSPEECH. ISCA,
2009, pp. 1827–1830. [Online]. Available: http://dblp.uni-trier.de/db/
conf/interspeech/interspeech2009.html#WechsungENSSMM09
[16] P. Mateo, “Android HCI Extractor,” 2011. [Online]. Available:
http://code.google.com/p/android-hci- extractor
[17] J. W. Kaltz, J. Ziegler, and S. Lohmann, “Context-aware web
engineering: Modeling and applications.” Revue d’Intelligence
Artificielle, vol. 19, no. 3, pp. 439–458, 2005. [Online]. Available:
http://dblp.uni-trier.de/db/journals/ria/ria19.html#KaltzZL05
[18] F. Jouault, F. Allilaire, J. Bezivin, and I. Kurtev, “Atl: A
model transformation tool,” Science of Computer Programming,
vol. 72, no. 1-2, pp. 31–39, June 2008. [Online]. Available:
http://dx.doi.org/10.1016/j.scico.2007.08.002
[19] Sociedad An´
onima de Electr´
onica Submarina (SAES), http://www.
electronica-submarina.com, [Online; accessed Oct. 2012].
[20] University of Murcia, http://www.um.es, [Online; accessed Oct. 2012].
127Copyright (c) IARIA, 2012. ISBN: 978-1-61208-232-5
CENTRIC 2012 : The Fifth International Conference on Advances in Human-oriented and Personalized Mechanisms, Technologies, and Services
