Conference PaperPDF Available

A Context-aware Model for the Analysis of User Interaction and QoE in Mobile Environments

Authors:
A Context-aware Model for the Analysis of User
Interaction and QoE in Mobile Environments
Pedro Mateo
C
´
atedra SAES Laboratories
University of Murcia
Murcia, Spain
email: pedromateo@um.es
Diego Sevilla Ruiz
Computer Engineering Department
University of Murcia
Murcia, Spain
email: dsevilla@um.es
Gregorio Mart
´
ınez P
´
erez
Communications Engineering Department
University of Murcia
Murcia, Spain
email: gregorio@um.es
Abstract—This paper describes a novel approach to model
quality of experience (QoE) in mobile environments. A meta-
model is created to establish a set of parameters to dynamically
describe the interaction between the user and the system, as well
as the context in which it is carried and the attractiveness of that
process. A uniform representation of user-system interaction in
mobile contexts is provided. This helps user-related applications
to determine QoE of users, and allows the comparison between
different interaction records. Its run-time nature also allows
context-aware applications to make model-based decisions in
real-time to adapt themselves, and thus providing a better
experience to users. As a result, this meta-model provides unified
criteria for the inference and analysis of QoE in mobile contexts,
as well as for implementing user profiling based on successful-
QoE experiences.
Index Terms—quality of experience; context-aware; QoE mod-
eling; interaction modeling; user profiling.
I. INTRODUCTION
Quality of Experience (QoE) is a subjective measure of
users’ experiences with a service. It focuses on those aspects
that users directly perceive as quality parameters, and that
finally decide the acceptability of a service. QoE is not
just related to Quality of Service (QoS), but it is a broader
construct beyond technical and objective system performance
metrics [1], [2], [3], [4]. It encompasses users’ behavioural,
cognitive, and psychological states along with the context in
which the services are provided to them. This is particularly
true in mobile contexts, where applications are dynamically
used in different scenarios and social contexts [4].
Moreover, context-aware systems extract, interpret, and use
context information to adapt their functionality to the current
context of use [5]. By “context” we mean any information
used to characterize the situation of an entity, e.g., person,
place, or object, and that is considered relevant for the user-
system interaction analysis. Hong et al. [6] differentiate among
external context, which involves data referring to the physical
environment, e.g., location, sound, time of the day, etc., and
internal context, which involves data related to the cognitive
domains of the user, e.g., emotional state.
Complexity of interaction within mobile scenarios has in-
creased dramatically in the last few years. Users and their
handheld devices are continuously moving in several simul-
taneous fuzzy contexts [6]. This dynamic environment sets
special requirements for mobile applications’ usability and
the acceptance of such systems. A close relationship between
interaction, its context, and QoE can be found in these environ-
ments. However, the lack of a uniform approach for modeling
the information related to interaction within a specific context
is obvious [7]. This is why this work proposes incorporating
user ratings and context-aware information into user-system
interaction analysis methods, providing a uniform basis to
quantify interaction within mobile scenarios, and use it to
determine QoE.
However, incorporating these data into user-system interac-
tion analysis processes poses several problems. One of them is
deciding what parameters are useful to capture QoE in mobile
contexts [4]. We consider essential that such parameters have
to be collected, as far as possible, by using current devices’
capabilities. The low standardization of technologies used in
context-aware systems is also a problem [6]. A common rep-
resentation of the context of applications is needed to support
standard analysis and decision processes, as well as to support
cooperation between different QoE analysis applications. A
related problem is how to build “standard” user-interaction
profiles, which are used by applications or systems to adapt
themselves to provide a better QoE.
According to these problems, the following research ques-
tions are posed:
Q1: How can rating and context information be properly
incorporated into interaction analysis processes?
Q2: How can QoE of different users be compared to each
other, as well as QoE inferred from different systems and/or
contexts?
Q3: Is it possible to build a user’s interaction profile based on
successful-QoE experiences?
To answer these questions, this paper briefly describes
the design of a meta-model arranging dynamic interaction
parameters. This meta-model is augmented with parameters
describing users’ perception of interaction quality, as well as
context information. It provides a uniform representation of
user-system interaction within real mobile contexts. Instances
of this meta-model provide a basis to determine and compare
QoE of users in such contexts, as well as to make decisions
at run-time to provide a better users’ experience.
The rest of the document is structured as follows. Section II
124Copyright (c) IARIA, 2012. ISBN: 978-1-61208-232-5
CENTRIC 2012 : The Fifth International Conference on Advances in Human-oriented and Personalized Mechanisms, Technologies, and Services
describes an overview of the meta-model design mentioned
above. Section III describes some considerations about the
implementation and the applications of the meta-model in
current mobile scenarios. Finally, Section IV includes some
conclusion and future lines of work.
II. QOE-AWARE INTERACTION ANALYSIS
The approach described in this paper tries to give an answer
to the three research questions posed above. First, to answer
research question Q1, the design of a meta-model including
interaction, user-rating and context data is proposed. It struc-
tures all these data to be the basis for the implementation of
QoE analysis and inference processes. We propose to base this
meta-model design on an existing one [8], [9] which is being
developed in parallel to this work, in a joint effort between
the C
´
atedra SAES [10] and the Telekom Innovation Labora-
tories [11] to quantify interaction in multimodal contexts.
The base meta-model describes interaction by turn, i.e.,
each time the user or the system take part in the dialog,
following a dialog structure, i.e., a set of ordered system-
and user-turns. This “step by step” description of interaction
creates a relationship between data and time, providing new
opportunities for the dynamic analysis of interaction. This
meta-model handles different modalities at the same level
using common metrics to describe interaction. These metrics
are structured within a common representation, allowing the
comparison among different interaction records. Collected
data (turn content, meta-communication, I/O information, and
modality description) are partly based on the well validated
parameters and concepts described in [12], [13].
Since the base meta-model only quantifies user-system
multimodal interaction, it was extended in to ways. On one
hand, user-rating parameters as the used in questionnaires like
AttrakDiff [14] were added to measure how attractive and
user-friendly is the product under test. On the other hand,
the model was extended with new parameters to describe the
interaction context in mobile scenarios. Thus, the model not
only provides a link between interaction data and time, but new
links between interaction and, for example, user’s opinions,
information about location, social context, device features, etc.
are created. Figure 1 depicts the kind of parameters considered
by the proposed design.
Human-computer interaction parameters —included in the
base meta-model— are used to quantify the interaction of the
user with the system, e.g., quantity of information provided by
the system, average reaction time of the user. User rating pa-
rameters are used to measure the experience of users with the
system under test, e.g., motivating, human, clearly structured.
They measure how attractive the system/application is in terms
of usability and appearance. The validity of questionnaires like
AttrakDiff as a method to extract users’ experience is shown
in related work, e.g., [15].
Communication parameters describe the features of device
connectivity, e.g., if the user is on-line or not, connection
bandwidth. Location and time parameters describe the posi-
tion of the user while interacting the system, as well as a
Figure 1. Design overview of the proposed meta-model.
time reference, e.g., the user is moving, it is Friday night.
User parameters describe peculiarities of the person using the
application and the device, e.g., gender, age, disabilities. Social
context parameters are those that can be extracted from social
media, and are interesting for the interaction analysis, e.g., if
the user is along with their friends, if he/she is within an office
context. Finally, Device parameters describe the peculiarities
of the device being used, e.g., screen size, input and output
methods.
Some of these parameters are run-time and have to be
collected many times during interaction, e.g., quantity of user
input at a specific time. Others are not, and are collected
only once, e.g., screen resolution. This feature is specially
relevant to know where the parameters are included in the
model design. Run-time ones are included at turn-level, while
static ones are included at dialog level.
Instances of this meta-model describe user-system interac-
tion within its mobile context, and are ready to support further
analysis, comparison, transformation, and decision processes.
The great majority of the parameters described above are
intended to be collected automatically, e.g., by using tools
like the Android HCI Extractor [16] which extracts multi-
modal interaction data at run-time. However, those based on
subjective judgments of the user or the expert have to be
manually annotated in most of cases. Anyway, automatically
collected or not, the same metrics —structured into a common
representation— are used to quantify the interaction between
the user and the system. This provides experts and tools with
unified criteria to describe the interaction process. Different
interaction records can be analyzed and compared regardless
of the system/application under testing, the interaction context,
even the modalities used to provide input and output data. This
answers research question Q2.
To answer question Q3, the reader can consider an instance
of the meta-model described above as a three-dimensional rep-
resentation of an interaction “occurrence”. These dimensions
125Copyright (c) IARIA, 2012. ISBN: 978-1-61208-232-5
CENTRIC 2012 : The Fifth International Conference on Advances in Human-oriented and Personalized Mechanisms, Technologies, and Services
are the user who is performing the interaction (u U , U the
set of all users under study), the application or system with
which the user interacts (a A, A the set of all applications or
systems under study), and the context in which the interaction
process is performed (c C, C the set of all the possible
contexts under study.) Let I be a set of instances of the model
described above, each instance i
n
I represents one of the
interaction occurrences in the space (u
i
, a
i
, c
i
). If we consider
all the instances i
n
in the space (u
i
), i.e., all the interaction
occurrences of the user u
i
, these can be used to build a “user
interaction profile” representing the features of the behavior
of this specific user.
Maybe using only one dimension to analyze interaction is
not useful to make decisions. However, if for example the
interaction occurrences in the space (u
i
, a
i
) are used to build
a profile, we can analyze the behavior and ratings of a user
using the same application in different contexts. Thus, the
application can be adapted to the current context, as mentioned
in [4]. Using the occurrences in the space (u
i
, c
i
) allow us
to analyze how the user behaves in a specific context, and
those in the space (a
i
, c
i
) are valid to analyze how different
users interact with an application in a specific context. The
process gets more complex if sub-dimensions of these three
dimensions are considered, but this is not the goal of this
paper. This shows that instances of the proposed meta-model
are valid to create different interaction profiles not only related
to users, but also related to the system/application in use or
the interaction context.
Finally, we consider essential the easy incorporation of the
proposed solution into current mobile devices, i.e., tablets,
smartphones, etc. From a practical point of view, developed
solutions should be easily built into the kind of devices used
nowadays. This not only fosters using such a kind of testing
tools into current applications and systems, but it will ease
the full implementation of context- and QoE-aware methods in
real life, and not only for laboratory environments [6]. This is
why we argue for using current devices capabilities to collect
interaction and context data, e.g., the windowing system to
collect touch interaction metrics, internet-based applications
to get social information, device’s sensors as GPS to get
position. Even users’ perception of interaction quality when
possible, e.g., encouraging the user to rate interaction at run-
time. Therefore, advanced sensors are not required to fill the
model instances.
III. META-MODEL IMPLEMENTATION AND APPLICATIONS
Our implementation of the base meta-model will use the
facilities offered by the Eclipse Modeling Framework (EMF).
The widely used EMF allows the definition of comprehen-
sible, flexible and extensible meta-models, as well as the
syntactically validation of concrete model instances. Tools like
EMF help to make the modeling process more effective, and
provide indispensable functionality to validate and extend the
meta-model [17]. EMF provides model transformation and
automatic code generation functionality as well. The design
of the meta-model will be used to automatically generate the
source code, which has to be integrated into the applications
in which new model instances will be created.
To collect interaction and context parameters automatically,
it is proposed using a tool based on the Android HCI Ex-
tractor. [16] This open-source tool can be extended to collect,
as far as the device allows, all the interaction and context
data necessary to fill model instances. User ratings might be
automatically collected as well, e.g., by showing question-
naires after test trials. This tool will be used to create model
instances at run-time. Such instances are valid to represent
many different interaction scenarios, from the usage of an
application during some minutes, to the usage of a device
during hours. Despite the architecture and behavior of the HCI
Extractor can be ported to other mobile systems —it uses
a reduced EMF Java implementation that can run in many
platforms— currently only Android is supported. However,
Android can run in many different mobile platforms, e.g.,
smartphones, tablets, netbooks, smart-tv, etc.
Once created, the instances provide a basis on which to
implement different analysis and evaluation processes. QoE in-
ference is the first result that comes to mind. The data included
into a model instance can be used to systematically determine
the QoE of a user within a mobile context. Interaction data can
be fused with context information and users/experts ratings to
determine QoE, e.g., by using Bayesian networks as in [4]
Moreover, thanks to the run-time nature of the meta-model,
QoE can be estimated in real-time. In case the resulting value
is not the expected, the interaction history and the context can
be analyzed for a specific interval of time to make a decision
that makes QoE to improve, e.g., by adjusting microphone
settings, changing screen brightness.
As a common representation is used, different instances can
be easily compared to each other, e.g., to detect why QoE
worsens when using an application in a different scenario,
to know why an application provides a better QoE than
another. Finally, model transformation processes can be im-
plemented using the high expressiveness of EMF-based tools,
e.g., ATL [18]. Original model instances can be transformed
into instances of other meta-models, which provide different
perspectives of the data collected during the interaction pro-
cess, e.g., a summary meta-model linking only context and
ratings data, a statistical meta-model aggregating users ratings.
Thus, model transformation is valid to build user interaction
profiles as well. Let Model
B be a meta-model describing the
user interaction profile. Several instances of the meta-model
proposed in this paper (say Model A) can be used to build an
instance of the new model by simply using ATL transformation
rules. The process is completely automatic, as data in Model A
instances are used by ATL to fill data fields in the Model B
instance according to the rules.
Some validations tests were conducted in the context of the
PALADIN project [8], [9]. The participants used multimodal
input (speech + touch) to book a restaurant within an Android
smartphone. These tests where used to show the validity of
the interaction meta-model on which the proposed solution is
based. Similar tests can be conducted to show the validity of
126Copyright (c) IARIA, 2012. ISBN: 978-1-61208-232-5
CENTRIC 2012 : The Fifth International Conference on Advances in Human-oriented and Personalized Mechanisms, Technologies, and Services
the solution proposed in this work, but now using a context-
dependant application and collecting real or simulated context
data. Then, the model instances will be used to determine QoE
of each participant using different modalities or a combination
of them.
IV. CONCLUSIONS AND FUTURE WORK
The design of a meta-model including data about user-
system interaction and its context is described. Most of the
parameters included in this design are collected using current
devices’ capabilities; subjective ones have to be annotated
manually. This meta-model provides a common representation
of interaction, allowing the inference, analysis and comparison
of QoE in mobile contexts. A relationship between user-system
interaction, its context, and users’ perception of quality is
created. Moreover, a strong relationship between these data
and time is provided as well, opening up new opportunities for
the dynamic analysis of QoE. Its dynamic nature also allows
to make QoE-aware decisions at run-time. Instances of this
meta-model are also valid to create user interaction profiles
based on successful-QoE experiences.
This approach poses lots of challenges with which to deal
until reaching its final implementation. One of them is treating
the variety, diversity and big amount of interaction and context
data. This solution is not aimed at modeling “the entire world”,
but only data that is relevant for the analysis of QoE have to
be considered. A well balanced set of parameters has to be
chosen for the model design. This set should be as small as
possible, but enough to determine QoE in mobile contexts.
Choosing an adequate abstraction level for the parameters
is also very important, e.g., the user is in a specific geographic
coordinate vs. the user is in the office. Dealing with cognitive
data automatically is also challenging. Emotional state of the
user and cognitive elements have to be incorporated into the
meta-model, as well as users’ quality perception and experts’
verdicts. Users’ security and privacy problem should be also
posed and discussed.
ACKNOWLEDGEMENT
This paper has been partially funded by the C
´
atedra SAES
of the University of Murcia initiative, a joint effort between
SAES [19] and the University of Murcia [20] to work on
open-source software, and real-time and critical information
systems.
REFERENCES
[1] S. M
¨
oller, K.-P. Engelbrecht, C. K
¨
uhnel, I. Wechsung, and B. Weiss, “A
Taxonomy of Quality of Service and Quality of Experience of Multi-
modal Human-Machine Interaction, in First International Workshop on
Quality of Multimedia Experience (QoMEX’09), July 2009, pp. 7–12.
[2] W. Wu, M. A. Arefin, R. Rivas, K. Nahrstedt, R. M. Sheppard, and
Z. Yang, “Quality of experience in distributed interactive multimedia
environments: toward a theoretical framework, in ACM Multimedia,
W. Gao, Y. Rui, A. Hanjalic, C. Xu, E. G. Steinbach, A. El-Saddik,
and M. X. Zhou, Eds. ACM, 2009, pp. 481–490. [Online]. Available:
http://dblp.uni-trier.de/db/conf/mm/mm2009.html#WuARNSY09
[3] K.-T. Chen, C.-J. Chang, C.-C. Wu, Y.-C. Chang, and C.-L. Lei,
“Quadrant of euphoria: a crowdsourcing platform for qoe assessment.
IEEE Network, vol. 24, no. 2, pp. 28–35, 2010. [Online]. Available: http:
//dblp.uni-trier.de/db/journals/network/network24.html#ChenCWCL10
[4] K. Mitra, A. B. Zaslavsky, and C.
˚
Ahlund, A probabilistic context-
aware approach for quality of experience measurement in pervasive
systems. in SAC, W. C. Chu, W. E. Wong, M. J. Palakal, and
C.-C. Hung, Eds. ACM, 2011, pp. 419–424. [Online]. Available:
http://dblp.uni-trier.de/db/conf/sac/sac2011.html#MitraZA11
[5] H. Byun and K. Cheverst, “Utilizing context history to provide dynamic
adaptations. Applied Artificial Intelligence, vol. 18, no. 6, pp. 533–548,
2004. [Online]. Available: http://dblp.uni-trier.de/db/journals/aai/aai18.
html\#ByunC04
[6] J. Hong, E. Suh, and S. Kim, “Context-aware systems: A literature
review and classification, Expert Systems with Applications, vol. 36,
no. 4, pp. 8509 8522, 2009. [Online]. Available: http://www.
sciencedirect.com/science/article/pii/S0957417408007574
[7] C. Bolchini, C. A. Curino, E. Quintarelli, F. A. Schreiber, and
L. Tanca, A data-oriented survey of context models, SIGMOD
Rec., vol. 36, pp. 19–26, December 2007. [Online]. Available:
http://doi.acm.org/10.1145/1361348.1361353
[8] P. Mateo and S. Hillmann, “PALADIN: a Run-time Model for
Automatic Evaluation of Multimodal Interfaces, 2011. [Online].
Available: http://www.catedrasaes.org/wiki/MIM
[9] ——, “Model-based Measurement of Human-Computer Interaction in
Mobile Multimodal Environments, in NordiCHI. ACM, 2012. [On-
line]. Available: http://www.prometei.de/fileadmin/ammi-nordichi2012/
04ammi12 mateo hillmann.pdf
[10] C
´
atedra SAES de la Unviersidad de Murcia, http://www.catedrasaes.org,
[Online; accessed Oct. 2012].
[11] Telekom Innovation Laboratories, http://www.laboratories.telekom.com,
[Online; accessed Oct. 2012].
[12] ITU-T Suppl. 24 to P-Series Rec., “Parameters describing the interaction
with spoken dialogue systems, Geneva, Switzerland, October 2005.
[13] C. K
¨
uhnel, B. Weiss, and S. M. ller, “Parameters describing multimodal
interaction - Definitions and three usage scenarios,” in Proceedings of the
11th Annual Conference of the ISCA (Interspeech 2010), T. Kobayashi,
K. Hirose, and S. Nakamura, Eds. Makuhari, Japan: ISCA, 2010, pp.
2014–2017.
[14] M. Hassenzahl, M. Burmester, and F. Koller, AttrakDiff:
Ein Fragebogen zur Messung wahrgenommener hedonischer und
pragmatischer Qualit
¨
at. (A questionnaire for measuring perceived
hedonic and pragmatic quality.), in Mensch & Computer, G. Szwillus
and J. Ziegler, Eds. Teubner, 2003, pp. 187–196. [Online]. Available:
http://dblp.uni-trier.de/db/conf/mc/mc2003.html#HassenzahlBK03
[15] I. Wechsung, K.-P. Engelbrecht, A. B. Naumann, S. Schaffer, J. Seebode,
F. Metze, and S. M
¨
oller, “Predicting the quality of multimodal systems
based on judgments of single modalities. in INTERSPEECH. ISCA,
2009, pp. 1827–1830. [Online]. Available: http://dblp.uni-trier.de/db/
conf/interspeech/interspeech2009.html#WechsungENSSMM09
[16] P. Mateo, “Android HCI Extractor, 2011. [Online]. Available:
http://code.google.com/p/android-hci-extractor
[17] J. W. Kaltz, J. Ziegler, and S. Lohmann, “Context-aware web
engineering: Modeling and applications. Revue d’Intelligence
Artificielle, vol. 19, no. 3, pp. 439–458, 2005. [Online]. Available:
http://dblp.uni-trier.de/db/journals/ria/ria19.html#KaltzZL05
[18] F. Jouault, F. Allilaire, J. Bezivin, and I. Kurtev, Atl: A
model transformation tool, Science of Computer Programming,
vol. 72, no. 1-2, pp. 31–39, June 2008. [Online]. Available:
http://dx.doi.org/10.1016/j.scico.2007.08.002
[19] Sociedad An
´
onima de Electr
´
onica Submarina (SAES), http://www.
electronica-submarina.com, [Online; accessed Oct. 2012].
[20] University of Murcia, http://www.um.es, [Online; accessed Oct. 2012].
127Copyright (c) IARIA, 2012. ISBN: 978-1-61208-232-5
CENTRIC 2012 : The Fifth International Conference on Advances in Human-oriented and Personalized Mechanisms, Technologies, and Services
Thesis
Full-text available
1 Introduction 1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1.1 Human-Computer Interaction . . . . . . . . . . . . . . . . . . . . 2 1.1.2 Software Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.1.3 Data Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.1.4 Software Usability . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.1.5 Quality of Experience . . . . . . . . . . . . . . . . . . . . . . . . . 10 Enhancing Software Quality . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.2.1 Block 1: Achieving Quality in Interaction Components Separately . 12 1.2.2 Block 2: Achieving Quality of User-System Interaction as a Whole . 14 1.3 Goals of this PhD Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.4 Publications Related to this PhD Thesis . . . . . . . . . . . . . . . . . . . . 19 1.5 Software Contributions of this PhD Thesis . . . . . . . . . . . . . . . . . . 22 1.5.1 OHT: Open HMI Tester . . . . . . . . . . . . . . . . . . . . . . . . 23 1.5.2 S-DAVER: Script-based Data Verification . . . . . . . . . . . . . . 24 1.5.3 PALADIN: Practice-oriented Analysis and Description of Multi-modal Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 CARIM: Context-Aware and Ratings Interaction Metamodel . . . . 25 1.6 Summary of Research Goals, Publications, and Software Contributions . . 25 1.7 Context of this PhD Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 1.8 Structure of this PhD Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2 Related Work 2.1 Group 1: Approaches Assuring Quality of a Particular Interaction Component 30 2.2 Validation of Software Output . . . . . . . . . . . . . . . . . . . . 30 2.1.1.1 Methods Using a Complete Model of the GUI . . . . . . 31 2.1.1.2 Methods Using a Partial Model of the GUI . . . . . . . . 32 2.1.1.3 Methods Based on GUI Interaction . . . . . . . . . . . . 32 Validation of User Input . . . . . . . . . . . . . . . . . . . . . . . . 33 2.1.2.1 Data Verification Using Formal Logic . . . . . . . . . . . 34 2.1.2.2 Data Verification Using Formal Property Monitors . . . . 35 2.1.2.3 Data Verification in GUIs and in the Web . . . . . . . . . 36 Group 2: Approaches Describing and Analyzing User-System Interaction as a Whole . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 2.2.1 Analysis of User-System Interaction . . . . . . . . . . . . . . . . . 37 2.2.1.1 Analysis for the Development of Multimodal Systems . . 37 2.2.1.2 Evaluation of Multimodal Interaction . . . . . . . . . . . 41 2.2.1.3 Evaluation of User Experiences . . . . . . . . . . . . . . 44 Analysis of Subjective Data of Users . . . . . . . . . . . . . . . . . 45 2.2.2.1 User Ratings Collection . . . . . . . . . . . . . . . . . . 45 2.2.2.2 Users Mood and Attitude Measurement . . . . . . . . . . 47 Analysis of Interaction Context . . . . . . . . . . . . . . . . . . . . 49 2.2.3.1 Interaction Context Factors Analysis . . . . . . . . . . . 49 2.2.3.2 Interaction Context Modeling . . . . . . . . . . . . . . . 50 3 Evaluating Quality of System Output 3.1 Introduction and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.2 GUI Testing Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.3 Preliminary Considerations for the Design of a GUI Testing Architecture . 57 3.3.1 Architecture Actors . . . . . . . . . . . . . . . . . . . . . . . . . . 57 3.3.2 Organization of the Test Cases . . . . . . . . . . . . . . . . . . . . 57 3.3.3 Interaction and Control Events . . . . . . . . . . . . . . . . . . . . 58 The OHT Architecture Design . . . . . . . . . . . . . . . . . . . . . . . . . 58 3.4.1 The HMI Tester Module Architecture . . . . . . . . . . . . . . . . 60 3.4.2 The Preload Module Architecture . . . . . . . . . . . . . . . . . . . 61 3.4.3 The Event Capture Process . . . . . . . . . . . . . . . . . . . . . . 63 3.4.4 The Event Playback Process . . . . . . . . . . . . . . . . . . . . . . 64 The OHT Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 3.5.1 Implementation of Generic and Final Functionality . . . . . . . . . 66 3.5.1.1 Generic Data Model . . . . . . . . . . . . . . . . . . . . 66 3.5.1.2 Generic Recording and Playback Processes . . . . . . . . 66 Implementation of Specific and Adaptable Functionality . . . . . . 67 3.5.2.1 Using the DataModelAdapter . . . . . . . . . . . . . . . 68 3.5.2.2 The Preloading Process . . . . . . . . . . . . . . . . . . . 68 3.5.2.3 Adapting the GUI Event Recording and Playback Processes 69 3.7 Technical Details About the OHT Implementation . . . . . . . . . 70 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 3.6.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 3.6.2 The Test Case Generation Process . . . . . . . . . . . . . . . . . . 73 3.6.3 Validation of Software Response . . . . . . . . . . . . . . . . . . . 74 3.6.4 Tolerance to Modifications, Robustness, and Scalability . . . . . . . 75 3.6.5 Performance Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 76 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 4 Evaluating Quality of Users Input 4.1 Introduction and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . 80 4.2 Practical Analysis of Common GUI Data Verification Approaches . . . . . 82 4.3 Monitoring GUI Data at Runtime . . . . . . . . . . . . . . . . . . . . . . . 83 4.4 Verification Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 4.4.1 Rule Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 4.4.2 Using the Rules to Apply Correction . . . . . . . . . . . . . . . . . 87 4.4.3 Rule Arrangement . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 4.4.4 Rule Management . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 4.4.4.1 88 Loading the Rules . . . . . . . . . . . . . . . . . . . . . xviiContents 4.4.4.2 Evolution of the Rules and the GUI . . . . . . . . . . . . 89 Correctness and Consistency of the Rules . . . . . . . . . . . . . . 90 4.5 The Verification Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 4.6 S-DAVER Architecture Design . . . . . . . . . . . . . . . . . . . . . . . . . 92 4.6.1 Architecture Details . . . . . . . . . . . . . . . . . . . . . . . . . . 92 4.6.2 Architecture Adaptation . . . . . . . . . . . . . . . . . . . . . . . . 94 4.7 S-DAVER Implementation and Integration Considerations . . . . . . . . . 95 4.8 Practical Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 4.8.1 Integration, Configuration, and Deployment of S-DAVER . . . . . 99 4.8.2 Defining the Rules in Qt Bitcoin Trader . . . . . . . . . . . . . . . 100 4.8.3 Defining the Rules in Transmission . . . . . . . . . . . . . . . . . . 103 4.8.4 Development and Verification Experience with S-DAVER . . . . . 106 4.9 Performance Analysis of S-DAVER . . . . . . . . . . . . . . . . . . . . . . 106 4.10 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 4.10.1 A Lightweight Data Verification Approach . . . . . . . . . . . . . 108 4.10.2 The S-DAVER Open-Source Implementation . . . . . . . . . . . . . 110 4.10.3 S-DAVER Compared with Other Verification Approaches . . . . . . 111 4.11 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 5 Modeling and Evaluating Quality of Multimodal User-System Interaction 115 5.1 Introduction and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . 116 5.2 A Model-based Framework to Evaluate Multimodal Interaction . . . . . . . 118 5.2.1 Classification of Dialog Models by Level of Abstraction . . . . . . 119 5.2.2 The Dialog Structure . . . . . . . . . . . . . . . . . . . . . . . . . 120 5.2.3 Using Parameters to Describe Multimodal Interaction . . . . . . . 121 5.2.3.1 Adaptation of Base Parameters . . . . . . . . . . . . . . 121 5.2.3.2 Defining new Modality and Meta-communication Param- eters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 5.2.3.3 Defining new Parameters for GUI and Gesture Interaction 123 5.2.3.4 Classification of the Multimodal Interaction Parameters . 124 5.3 Design of PALADIN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 5.4 Implementation, Integration, and Usage of PALADIN . . . . . . . . . . . . 129 5.5 Application Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 5.6 Assessment of PALADIN as an Evaluation Tool . . . . . . . . . . . 132 5.5.1.1 Participants and Material . . . . . . . . . . . . . . . . . 134 5.5.1.2 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . 136 5.5.1.3 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . 137 Usage of PALADIN in a User Study . . . . . . . . . . . . . . . . . 140 5.5.2.1 Participants and Material . . . . . . . . . . . . . . . . . 140 5.5.2.2 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . 144 5.5.2.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 5.6.1 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 146 5.6.2 Practical Application of PALADIN . . . . . . . . . . . . . . . . . . 147 5.6.3 Completeness of PALADIN According to Evaluation Guidelines . . 148 5.6.4 Limitations in Automatic Logging of Interactions Parameters . . . 151 5.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 5.8 Parameters Used in PALADIN . . . . . . . . . . . . . . . . . . . . . . . . . 152 6 Modeling and Evaluating Mobile Quality of Experience 163 6.1 Introduction and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . 164 6.2 Context- and QoE-aware Interaction Analysis . . . . . . . . . . . . . . . . 166 6.2.1 Incorporating Context Information and User Ratings into Interaction Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 6.2.2 Arranging the Parameters for the Analysis of Mobile Experiences . 168 6.2.3 Using CARIM for QoE Assessment . . . . . . . . . . . . . . . . . . 169 Context Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 6.3.1 Quantifying the Surrounding Context . . . . . . . . . . . . . . . . 170 6.3.2 Arranging Context Parameters into CARIM . . . . . . . . . . . . . 173 User Perceived Quality Parameters . . . . . . . . . . . . . . . . . . . . . . 173 6.4.1 Measuring the Attractiveness of Interaction . . . . . . . . . . . . . 173 6.4.2 Measuring Users Emotional State and Attitude toward Technology Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 6.5 Arranging User Parameters into CARIM . . . . . . . . . . . . . . . 177 CARIM Model Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 6.5.1 The Base Design: PALADIN . . . . . . . . . . . . . . . . . . . . . 177 6.5.2 The New Proposed Design: CARIM . . . . . . . . . . . . . . . . . 178 6.6 CARIM Model Implementation . . . . . . . . . . . . . . . . . . . . . . . . 181 6.7 Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 6.7.1 Participants and Material . . . . . . . . . . . . . . . . . . . . . . . 183 6.7.2 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 6.7.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 6.9 Comparing the Two Interaction Designs for UMU Lander 185 Validating the User Behavior Hypotheses . . . . . . . . . 186 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 6.8.1 Modeling Mobile Interaction and QoE . . . . . . . . . . . . . . . . 188 6.8.3 CARIM Implementation and Experimental Validation . . . . . . . 190 CARIM Compared with Other Representative Approaches . . . . . 191 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 7 Conclusions and Further Work 7.2 Conclusions of this PhD Thesis . . . . . . . . . . . . . . . . . . . . . . . . 196 7.1.2 Driving Forces of this PhD Thesis . . . . . . . . . . . . . . . . . . 196 Work and Research in User-System Interaction Assessment . . . . 197 7.1.3 Goals Achieved in this PhD Thesis . . . . . . . . . . . . . . . . . . 200 Future Lines of Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 Bibliography 205 A List of Acronyms 231
Article
Full-text available
Existing quality of experience assessment methods, subjective or objective, suffer from either or both problems of inaccurate experiment tools and expensive personnel cost. The panacea for them, as we have come to realize, lies in the joint application of paired comparison and crowdsourcing, the latter being a Web 2.0 practice of organizations asking ordinary unspecific Internet users to carry out internal tasks. We present in this article Quadrant of Euphoria, a user-friendly Web-based platform facilitating QoE assessments in network and multimedia studies, which features low cost, participant diversity, meaningful and interpretable QoE scores, subject consistency assurance, and a burdenless experiment process.
Conference Paper
Full-text available
The past decades have witnessed a rapid growth of Dis- tributed Interactive Multimedia Environments (DIMEs). De- spite their intensity of user-involved interaction, the exist- ing evaluation frameworks remain very much system-centric. As a step toward the human-centric paradigm, we present a conceptual framework of Quality of Experience (QoE) in DIMEs, to model, measure, and understand user experience and its relationship with the traditional Quality of Service (QoS) metrics. A multi-displinary approach is taken to build up the framework based on the theoretical results from var- ious fields including psychology, cognitive sciences, sociol- ogy, and information technology. We introduce a mapping methodology to quantify the correlations between QoS and QoE, and describe our controlled and uncontrolled stud- ies as illustrating examples. The results present the first deep study to model the multi-facet QoE construct, map the QoS-QoE relationship, and capture the human-centric quality modalities in the context of DIMEs.
Conference Paper
Full-text available
In this paper, we develop a novel context-aware approach for quality of experience (QoE) modeling, reasoning and inferencing in mobile and pervasive computing environments. The proposed model is based upon a state-space approach and Bayesian networks for QoE modeling and reasoning. We further extend this context model to incorporate influence diagrams for efficient QoE inferencing. Our approach accommodates user, device and quality of service (QoS) related context parameters to determine the overall QoE of the user. This helps in user-related media, network and device adaptation. We perform experimentation to validate the proposed approach and the results verify its modeling and inferencing capabilities.
Article
Full-text available
Context-aware systems are pervading everyday life, there fore context modeling is becoming a relevant issue and an expanding research field. This survey has the goal to pro vide a comprehensive evaluation framework, allowing application designers to compare context models with respect to a given target application; in particular we stress the analysis of those features which are relevant for the problem of data tailoring. The contribution of this paper is twofold: a general analysis framework for context models and an up-to-date comparison of the most interesting, data-oriented approaches available in the literature
Article
Multimodal interfaces are expected to improve input and output capabilities of increasingly sophisticated applications. Several approaches are aimed at formally describing multimodal interaction. However, they rarely treat it as a continuous flow of actions, preserving its dynamic nature and considering modalities at the same level. This work proposes a model-based approach called Practice-oriented Analysis and Description of Multimodal Interaction (PALADIN) aimed at describing sequential multimodal interaction beyond such problems. It arranges a set of parameters to quantify multimodal interaction as a whole, in order to minimise the existing differences between modalities. Furthermore, interaction is described stepwise to preserve the dynamic nature of the dialogue process. PALADIN defines a common notation to describe interaction in different multimodal contexts, providing a framework to assess and compare the usability of systems. Our approach was integrated into four real applications to conduct two experiments with users. The experiments show the validity and prove the effectiveness of the proposed model for analysing and evaluating multimodal interaction.
Conference Paper
Quality of Service (QoS) and Quality of Experience (QoE) are not only important for services transmitting multimedia data, but also for services involving multimodal human-machine interaction. In order to guide the assessment and evaluation of such services, we developed a taxonomy of the most relevant QoS and QoE aspects which result from multimodal human-machine interactions. It consists of three layers: (1) The QoS-influencing factors related to the user, the system, and the context of use; (2) the QoS interaction performance aspects describing user and system behavior and performance; and (3) the QoE aspects related to the quality perception and judgment processes taking place inside the user. For each of these layers, we provide metrics which make system evaluation more systematic and comparable.
Article
In the context of Model Driven Engineering, models are the main development artifacts and model transformations are among the most important operations applied to models. A number of specialized languages have been proposed, aimed at specifying model transformations. Apart from the software engineering properties of transformation languages, the availability of high quality tool support is also of key importance for the industrial adoption and ultimate success of MDE. In this paper we present ATL: a model transformation language and its execution environment based on the Eclipse framework. ATL tools provide support for the major tasks involved in using a language: editing, compiling, executing, and debugging.
Conference Paper
While multimodal systems are an active research field, there is no agreed-upon set of multimodal interaction parameters, which would allow to quantify the performance of such systems and their underlying modules, and would there for be necessary for a systematic evaluation. In this paper we propose an extension to established parameters describing the interaction with spoken dialog systems [1] in order to be used for multimodal systems. Focussing on the evaluation of a multimodal system, three usage scenarios for these parameters are given.
Conference Paper
This paper investigates the relationship between user ratings of multimodal systems and user ratings of its single modalities. Based on previous research showing precise predictions of ratings of multimodal systems based on ratings of single modality, it was hypothesized that the accuracy might have been caused by the participants' efforts to rate consistently. We address this issue with two new studies. In the first study, the multimodal system was presented before the single modality versions were known by the users. In the second study, the type of system was changed, and age effects were investigated. We apply linear regression and show that models get worse when the order is changed. In addition, models for younger users perform better than those for older users. We conclude that ratings can be impacted by the effort of users to judge consistently, as well as their ability to do so.
Article
This article presents an approach to Web Engineering which aims to account for context-awareness in a comprehensive and integrated fashion, thus enabling an enhanced adaptation of the application to the end-user. A conceptual model, permitting the combination of a domain ontology with context-relevant parameters and a degree of relevance, is presented. Subsequently, the use of such a model in a Web Engineering process is discussed, including appropriate modeling software, and requirements for a runtime system.