1 Introduction
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 Human-Computer Interaction . . . . . . . . . . . . . . . . . . . . 2
1.1.2 Software Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.3 Data Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.1.4 Software Usability . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.1.5 Quality of Experience . . . . . . . . . . . . . . . . . . . . . . . . . 10
Enhancing Software Quality . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.2.1 Block 1: Achieving Quality in Interaction Components Separately . 12
1.2.2 Block 2: Achieving Quality of User-System Interaction as a Whole . 14
1.3 Goals of this PhD Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.4 Publications Related to this PhD Thesis . . . . . . . . . . . . . . . . . . . . 19
1.5 Software Contributions of this PhD Thesis . . . . . . . . . . . . . . . . . . 22
1.5.1 OHT: Open HMI Tester . . . . . . . . . . . . . . . . . . . . . . . . 23
1.5.2 S-DAVER: Script-based Data Verification . . . . . . . . . . . . . . 24
1.5.3 PALADIN: Practice-oriented Analysis and Description of Multi-modal Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
CARIM: Context-Aware and Ratings Interaction Metamodel . . . . 25
1.6 Summary of Research Goals, Publications, and Software Contributions . . 25
1.7 Context of this PhD Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
1.8 Structure of this PhD Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2 Related Work
2.1 Group 1: Approaches Assuring Quality of a Particular Interaction Component 30
2.2 Validation of Software Output . . . . . . . . . . . . . . . . . . . . 30
2.1.1.1 Methods Using a Complete Model of the GUI . . . . . . 31
2.1.1.2 Methods Using a Partial Model of the GUI . . . . . . . . 32
2.1.1.3 Methods Based on GUI Interaction . . . . . . . . . . . . 32
Validation of User Input . . . . . . . . . . . . . . . . . . . . . . . . 33
2.1.2.1 Data Verification Using Formal Logic . . . . . . . . . . . 34
2.1.2.2 Data Verification Using Formal Property Monitors . . . . 35
2.1.2.3 Data Verification in GUIs and in the Web . . . . . . . . . 36
Group 2: Approaches Describing and Analyzing User-System Interaction
as a Whole . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.2.1 Analysis of User-System Interaction . . . . . . . . . . . . . . . . . 37
2.2.1.1 Analysis for the Development of Multimodal Systems . . 37
2.2.1.2 Evaluation of Multimodal Interaction . . . . . . . . . . . 41
2.2.1.3 Evaluation of User Experiences . . . . . . . . . . . . . . 44
Analysis of Subjective Data of Users . . . . . . . . . . . . . . . . . 45
2.2.2.1 User Ratings Collection . . . . . . . . . . . . . . . . . . 45
2.2.2.2 Users Mood and Attitude Measurement . . . . . . . . . . 47
Analysis of Interaction Context . . . . . . . . . . . . . . . . . . . . 49
2.2.3.1 Interaction Context Factors Analysis . . . . . . . . . . . 49
2.2.3.2 Interaction Context Modeling . . . . . . . . . . . . . . . 50
3 Evaluating Quality of System Output
3.1 Introduction and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.2 GUI Testing Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.3 Preliminary Considerations for the Design of a GUI Testing Architecture . 57
3.3.1 Architecture Actors . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.3.2 Organization of the Test Cases . . . . . . . . . . . . . . . . . . . . 57
3.3.3 Interaction and Control Events . . . . . . . . . . . . . . . . . . . . 58
The OHT Architecture Design . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.4.1 The HMI Tester Module Architecture . . . . . . . . . . . . . . . . 60
3.4.2 The Preload Module Architecture . . . . . . . . . . . . . . . . . . . 61
3.4.3 The Event Capture Process . . . . . . . . . . . . . . . . . . . . . . 63
3.4.4 The Event Playback Process . . . . . . . . . . . . . . . . . . . . . . 64
The OHT Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
3.5.1 Implementation of Generic and Final Functionality . . . . . . . . . 66
3.5.1.1 Generic Data Model . . . . . . . . . . . . . . . . . . . . 66
3.5.1.2 Generic Recording and Playback Processes . . . . . . . . 66
Implementation of Specific and Adaptable Functionality . . . . . . 67
3.5.2.1 Using the DataModelAdapter . . . . . . . . . . . . . . . 68
3.5.2.2 The Preloading Process . . . . . . . . . . . . . . . . . . . 68
3.5.2.3 Adapting the GUI Event Recording and Playback Processes 69
3.7 Technical Details About the OHT Implementation . . . . . . . . . 70
Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.6.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.6.2 The Test Case Generation Process . . . . . . . . . . . . . . . . . . 73
3.6.3 Validation of Software Response . . . . . . . . . . . . . . . . . . . 74
3.6.4 Tolerance to Modifications, Robustness, and Scalability . . . . . . . 75
3.6.5 Performance Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 76
Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
4 Evaluating Quality of Users Input
4.1 Introduction and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.2 Practical Analysis of Common GUI Data Verification Approaches . . . . . 82
4.3 Monitoring GUI Data at Runtime . . . . . . . . . . . . . . . . . . . . . . . 83
4.4 Verification Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
4.4.1 Rule Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
4.4.2 Using the Rules to Apply Correction . . . . . . . . . . . . . . . . . 87
4.4.3 Rule Arrangement . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.4.4 Rule Management . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
4.4.4.1 88
Loading the Rules . . . . . . . . . . . . . . . . . . . . .
xviiContents
4.4.4.2 Evolution of the Rules and the GUI . . . . . . . . . . . . 89
Correctness and Consistency of the Rules . . . . . . . . . . . . . . 90
4.5 The Verification Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.6 S-DAVER Architecture Design . . . . . . . . . . . . . . . . . . . . . . . . . 92
4.6.1 Architecture Details . . . . . . . . . . . . . . . . . . . . . . . . . . 92
4.6.2 Architecture Adaptation . . . . . . . . . . . . . . . . . . . . . . . . 94
4.7 S-DAVER Implementation and Integration Considerations . . . . . . . . . 95
4.8 Practical Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
4.8.1 Integration, Configuration, and Deployment of S-DAVER . . . . . 99
4.8.2 Defining the Rules in Qt Bitcoin Trader . . . . . . . . . . . . . . . 100
4.8.3 Defining the Rules in Transmission . . . . . . . . . . . . . . . . . . 103
4.8.4 Development and Verification Experience with S-DAVER . . . . . 106
4.9 Performance Analysis of S-DAVER . . . . . . . . . . . . . . . . . . . . . . 106
4.10 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
4.10.1 A Lightweight Data Verification Approach . . . . . . . . . . . . . 108
4.10.2 The S-DAVER Open-Source Implementation . . . . . . . . . . . . . 110
4.10.3 S-DAVER Compared with Other Verification Approaches . . . . . . 111
4.11 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
5 Modeling and Evaluating Quality of Multimodal User-System Interaction 115
5.1 Introduction and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . 116
5.2 A Model-based Framework to Evaluate Multimodal Interaction . . . . . . . 118
5.2.1 Classification of Dialog Models by Level of Abstraction . . . . . . 119
5.2.2 The Dialog Structure . . . . . . . . . . . . . . . . . . . . . . . . . 120
5.2.3 Using Parameters to Describe Multimodal Interaction . . . . . . . 121
5.2.3.1 Adaptation of Base Parameters . . . . . . . . . . . . . . 121
5.2.3.2 Defining new Modality and Meta-communication Param-
eters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
5.2.3.3 Defining new Parameters for GUI and Gesture Interaction 123
5.2.3.4 Classification of the Multimodal Interaction Parameters . 124
5.3 Design of PALADIN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
5.4 Implementation, Integration, and Usage of PALADIN . . . . . . . . . . . . 129
5.5 Application Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
5.6 Assessment of PALADIN as an Evaluation Tool . . . . . . . . . . . 132
5.5.1.1 Participants and Material . . . . . . . . . . . . . . . . . 134
5.5.1.2 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . 136
5.5.1.3 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . 137
Usage of PALADIN in a User Study . . . . . . . . . . . . . . . . . 140
5.5.2.1 Participants and Material . . . . . . . . . . . . . . . . . 140
5.5.2.2 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . 144
5.5.2.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
5.6.1 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 146
5.6.2 Practical Application of PALADIN . . . . . . . . . . . . . . . . . . 147
5.6.3 Completeness of PALADIN According to Evaluation Guidelines . . 148
5.6.4 Limitations in Automatic Logging of Interactions Parameters . . . 151
5.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
5.8 Parameters Used in PALADIN . . . . . . . . . . . . . . . . . . . . . . . . . 152
6 Modeling and Evaluating Mobile Quality of Experience 163
6.1 Introduction and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . 164
6.2 Context- and QoE-aware Interaction Analysis . . . . . . . . . . . . . . . . 166
6.2.1 Incorporating Context Information and User Ratings into Interaction Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
6.2.2 Arranging the Parameters for the Analysis of Mobile Experiences . 168
6.2.3 Using CARIM for QoE Assessment . . . . . . . . . . . . . . . . . . 169
Context Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
6.3.1 Quantifying the Surrounding Context . . . . . . . . . . . . . . . . 170
6.3.2 Arranging Context Parameters into CARIM . . . . . . . . . . . . . 173
User Perceived Quality Parameters . . . . . . . . . . . . . . . . . . . . . . 173
6.4.1 Measuring the Attractiveness of Interaction . . . . . . . . . . . . . 173
6.4.2 Measuring Users Emotional State and Attitude toward Technology
Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
6.5 Arranging User Parameters into CARIM . . . . . . . . . . . . . . . 177
CARIM Model Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
6.5.1 The Base Design: PALADIN . . . . . . . . . . . . . . . . . . . . . 177
6.5.2 The New Proposed Design: CARIM . . . . . . . . . . . . . . . . . 178
6.6 CARIM Model Implementation . . . . . . . . . . . . . . . . . . . . . . . . 181
6.7 Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
6.7.1 Participants and Material . . . . . . . . . . . . . . . . . . . . . . . 183
6.7.2 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
6.7.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
6.9 Comparing the Two Interaction Designs for UMU Lander 185
Validating the User Behavior Hypotheses . . . . . . . . . 186
Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
6.8.1 Modeling Mobile Interaction and QoE . . . . . . . . . . . . . . . . 188
6.8.3 CARIM Implementation and Experimental Validation . . . . . . . 190
CARIM Compared with Other Representative Approaches . . . . . 191
Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
7 Conclusions and Further Work
7.2 Conclusions of this PhD Thesis . . . . . . . . . . . . . . . . . . . . . . . . 196
7.1.2 Driving Forces of this PhD Thesis . . . . . . . . . . . . . . . . . . 196
Work and Research in User-System Interaction Assessment . . . . 197
7.1.3 Goals Achieved in this PhD Thesis . . . . . . . . . . . . . . . . . . 200
Future Lines of Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
Bibliography 205
A List of Acronyms 231
Figures - uploaded by
Pedro Luis Mateo NavarroAuthor contentAll figure content in this area was uploaded by Pedro Luis Mateo Navarro
Content may be subject to copyright.