Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten

Abstract

Präsentation zum World Usability Day in Osnabrück am 12.11.2020

Full text

Messung der Qualität des Benutzererlebnisses
bei der Verwendung von Sprachassistenten
Andreas M. Klein Prof. Dr. Maria Rauschenberger
Abteilung für Computersprachen und -systeme Fachbereich Technik
Universität Sevilla, Spanien Hochschule Emden/Leer, Deutschland
andreas.klein@iwt2.org maria.rauschenberger@hs-emden-leer.de
Last updated: 20-11-12
WUDOS 2020 ・12. November 2020

Department of Computer
Languages and Systems
University of Seville
Inhalt
2
1. Vorwort (Preface)
2. Einleitung (Introduction)
3. Methodik (Methodology)
4. Studie (Study for scale construction)
5. Ergebnisse und Diskussion (Results and discussion)
6. Fazit und Ausblick (Conclusion and future work)
7. Literaturhinweise (References)
WUDOS 2020 ・Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten ・Klein et al.

Department of Computer
Languages and Systems
University of Seville
Preface
3
Measuring User Experience Quality of Voice Assistants
2020 15th Iberian Conference on Information Systems and Technologies (CISTI)
Andreas M. Klein Martin Schrepp
University of Applied Sciences Emden/Leer, Germany SAP SE, Walldorf, Germany
Andreas Hinderks Jörg Thomaschewski
University of Seville, Spain University of Applied Sciences Emden/Leer, Germany
DOI: 10.23919/CISTI49556.2020.9140966 Publisher: IEEE
WUDOS 2020 ・Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten ・Klein et al.

Department of Computer
Languages and Systems
University of Seville
Preface
4
Construction of UEQ+ Scales for Voice Quality
2020 Mensch und Computer (MuC ’20)
Andreas M. Klein Martin Schrepp
University of Applied Sciences Emden/Leer, Germany SAP SE, Walldorf, Germany
Andreas Hinderks Jörg Thomaschewski
University of Seville, Spain University of Applied Sciences Emden/Leer, Germany
DOI: https://dl.acm.org/doi/10.1145/3404983.3410003 Publisher: ACM
WUDOS 2020 ・Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten ・Klein et al.

Department of Computer
Languages and Systems
University of Seville
Preface
5
Exploring Voice Assistant Risks and Potential with
Technology-based Users
2020 16th International Conference on Web Information Systems and
Technologies (WEBIST)
Andreas M. Klein Maria Rauschenberger
University of Seville, Spain Social Computing Systems, Max Planck Institute for
Software Systems, Saarbrücken, Germany
Andreas Hinderks Jörg Thomaschewski
University of Seville, Spain University of Applied Sciences Emden/Leer, Germany
Publisher: SCITEPRESS
https://www.researchgate.net/publication/345253241_Exploring_Voice_Assistant_Risks_and_Potential_with_Technology-based_Users
WUDOS 2020 ・Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten ・Klein et al.

Department of Computer
Languages and Systems
University of Seville
Introduction
6
•Voice assistants (VAs):
oGreat future prognoses [1]
oWide range of application [2]
oGeneral-purpose assistants
§‘Adaptive Voice (Vision) Assistants’ [3]
§Siri (Apple), Alexa (Amazon), Google
Assistant (Google), Cortana (Microsoft),
Bixby (Samsung),…
WUDOS 2020 ・Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten ・Klein et al.

Department of Computer
Languages and Systems
University of Seville
Introduction
•Voice User Interfaces (VUI)
“VUI is what a person interacts with when communicating with spoken
language application.” [4]
•User Experience (UX)
oISO 9241-210 [5]: Holistic concept includes all types of reactions
(emotional, cognitive, physical) before, during and after use of the
product.
oSet of distinct quality criteria [6] includes the classical usability
criteria (e.g. Efficiency) and non-goal directed criteria (e.g.
Stimulation).
7
WUDOS 2020 ・Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten ・Klein et al.

Department of Computer
Languages and Systems
University of Seville
Introduction
•Why measuring UX quality of VAs?
oGreat future prognoses, wide range of
application [1], [2]
oUseful tool for evaluating [7]
oExisting tools measure usability [8]
oConsideration of complete UX
oExploring improvements [9]
8
WUDOS 2020 ・Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten ・Klein et al.

Department of Computer
Languages and Systems
University of Seville
Methodolody
•UEQ+ Framework [10] (Modular questionnaire concept)
oContains 16 scales to measure different UX aspects.
oScales can be combined to create product-related questionnaire.
oScales type 1: User interaction with graphical user interfaces (GUI) [7]
•Research gap
oUEQ+ lacks scales for VUI.
•UX aspects of VUI differ significantly from those of GUI.
oHearing and voice function differently than the eyes.
•Construction of scales to measure UX aspects of VAs.
9
http://ueqplus.ueq-research.org
WUDOS 2020 ・Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten ・Klein et al.

Department of Computer
Languages and Systems
University of Seville
Methodolody
10
•UX aspects of VAs
Component
User (Expectations)
- Naturalness
-Confidentiality
System (Properties)
- Functionality
- Purpose
Context
- Environment
- Purpose of activity
Scale
Response behaviour
- According to social norms
- Human conversationalist
Response quality
-Current information
- Intention of the user fulfilled
Comprehensibility
- No special formulations (syntax)
- Intention of the user recognized
Components that influence UX in HCI [11] compared to derived scales for VUIs.
WUDOS 2020 ・Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten ・Klein et al.

Department of Computer
Languages and Systems
University of Seville
Study for scale construction
•Scale construction
oPool of candidate items for the three UX aspects for VAs.
oIntegration into the UEQ+ framework scale format.
•Study
oOnline questionnaire in German language.
oStudents and members of the University of Applied Sciences in
Emden/Leer (Germany).
o96 persons participated voluntarily.
oParticipant average age: 35 years (59 male, 35 female, 2 no answer)
11
WUDOS 2020 ・Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten ・Klein et al.

Department of Computer
Languages and Systems
University of Seville
Results and discussion
•Voice assistants rated by the participants:
12
0 5 10 15 20 25 30 35 40
Alexa
Siri
Google Assistant
Others
Number of participants
35
27
26
8
WUDOS 2020 ・Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten ・Klein et al.

Department of Computer
Languages and Systems
University of Seville
Results and discussion
•Factorial analysis of item sets (a total of 30 candidate items)
•Principal component analysis [13] (Varimax rotation)
•Assumption of three factors confirmed:
oResponse behavior loaded high on factor 2 (low on the others)
oResponse quality loaded high on factor 3 (low on the others)
oComprehensibility loaded high on factor 1 (low on the others)
•Details analysis is described in the research protocol [11].
13
WUDOS 2020 ・Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten ・Klein et al.

Department of Computer
Languages and Systems
University of Seville
Results and discussion
•Response behaviour:
oSet of candidate items and loadings on factor 2 (varimax rotation)
14
No. Items
(German – Original Version)
Items
(English Translation)
Loading
1technisch menschlich technical human 0.66
2künstlich natürlich artificial natural 0.80
3fremd vertraut unfamiliar familiar 0.66
4ungewöhnlich gewöhnlich unusual usual 0.25
5langsam schnell slow fast 0.48
6unangenehm angenehm
unpleasant
pleasant 0.75
7
unsympathisch
sympathisch
unlikeable likable 0.81
8unfreundlich freundlich unfriendly friendly 0.66
9langweilig
unterhaltsam
boring
entertaining
0.68
WUDOS 2020 ・Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten ・Klein et al.

Department of Computer
Languages and Systems
University of Seville
Results and discussion
•Response behaviour:
oItems with highest loadings (factor 2) and introducing sentence:
15
In my opinion the response behaviour of the voice assistant is
artificial
!
!
!
!
!
!
!
natural
unpleasant
!
!
!
!
!
!
!
pleasant
unlikable
!
!
!
!
!
!
!
likable
boring
!
!
!
!
!
!
!
entertaining
WUDOS 2020 ・Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten ・Klein et al.

Department of Computer
Languages and Systems
University of Seville
Results and discussion
•Response quality:
oItems with highest loadings (factor 3) and introducing sentence:
•Comprehensibility:
oItems with highest loadings (factor 1) and introducing sentence:
16
The answers and questions asked by the voice assistant are
inappropriate
!
!
!
!
!
!
!
suitable
useless
!
!
!
!
!
!
!
useful
not helpful
!
!
!
!
!
!
!
helpful
unintelligent
!
!
!
!
!
!
!
intelligent
In my opinion the voice assistant has understood my voice commands
complicated
!
!
!
!
!
!
!
simple
inaccurate
!
!
!
!
!
!
!
accurate
unambiguous
!
!
!
!
!
!
!
ambiguous
enigmatic
!
!
!
!
!
!
!
explainable
WUDOS 2020 ・Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten ・Klein et al.

Department of Computer
Languages and Systems
University of Seville
Results and discussion
•Product-related questionnaire
oSelect relevant scales
oProduct-specific UX aspects first [14]
oFurther criteria e.g. for marketing [14]
oMore information: UEQ+ Handbook (http://ueqplus.ueq-research.org)
17
Questionnaire example 1:
Users applying smart home
Questionnaire example 2
Voice assistant customer service
§Perspicuity
§Trust
§Trustworthiness of content
§Quality of content
§Efficiency
§Perspicuity
§Dependability
§Trust
§Response behaviour
§Response quality
§Comprehensibility
Voice quality scales
WUDOS 2020 ・Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten ・Klein et al.

Department of Computer
Languages and Systems
University of Seville
Conclusion and future work
•Conclusion:
oApproach based on UEQ+
oConstruction of VA scales
oStudy for scale construction
oThe factorial analysis confirmed:
§Response behaviour
§Response quality
§Comprehensibility
oTwo example using new VA scales.
•Future work:
oValidation of scales specific to VAs.
18
WUDOS 2020 ・Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten ・Klein et al.

Department of Computer
Languages and Systems
University of Seville
References
[1] Tuzovic, S. and Paluch, S. (2018). Conversational Commerce – A New Era for Service Business Development?, pages 81–100.
Springer Fachmedien Wiesbaden, Wiesbaden.
[2| Tractica (2020). Tractica. https://tractica.omdia.com/newsroom/press-releases/voice-and-speechrecognition-software-market-to-
reach-6-9-billionby-2025/.
[3] Knote, R., Janson, A., S¨ollner, M., and Leimeister, J. M. (2019). Classifying smart personal assistants: An empirical cluster
analysis. In Proceedings of the 52nd Hawaii International Conference on System Sciences.
[4] Cohen, M. H., Giangola, J. P., and Balogh, J. (2004). Voice User Interface Design. Addison Wesley Longman Publishing Co.,
Inc., USA.
[5] ISO9241-210, Ergonomics of human-system interaction - Part 210: Human-centred design for interactive systems (2010).
[6] Preece, J., Rogers, Y., & Sharp, H. (2015). Interaction Design: Beyond HCI, 4th edn. Wiley, Chichester, 2015.
[7] Klein, A. M., Hinderks, A., Schrepp, M., and Thomaschewski, J. (2020b). Construction of UEQ+ Scales for Voice Quality. In
Proceedings of the Conference on Mensch und Computer, MuC ’20, page 1–5, New York, NY, USA. ACM.
[8] Hone, K. S., & Graham, R. (2000). Towards a tool for the subjective assessment of speech system interfaces (SASSI). Natural
Language Engineering, 6(3-4), 287–303.
[9] Klein, A. M., Hinderks, A., Rauschenberger, M., & Thomaschewski, J., (2020a). Exploring Voice Assistant Risks and Potential
with Technology-based Users. In Proceedings of 16th International Conference on Web Information Systems and technology
(WEBIST), pages 1–8, SCITEPRESS.
[10] Schrepp, M. and Thomaschewski, J. (2019). Design and Validation of a Framework for the Creation of User Experience
Questionnaires. International Journal of Interactive Multimedia and Artificial Intelligence, 5(7):88–95.
[11] Hassenzahl, Tractinsky, User experience—a research agenda, Behaviour & Information Technology, Vol. 25, No. 2, March-April
2006, 91 –97.
[12] Klein, A. M., Hinderks, A., Schrepp, M., & Thomaschewski, J., (2020b). Protocol for Measuring User Experience Quality of Voice
Assistants. DOI: 10.13140/RG.2.2.12816.35848
[13] Revelle, W. (2018). Psych: Procedures for personality and psychological research, Northwestern University, Evanston, Illinois,
USA, https://CRAN.R-project.org/package=psych Version = 1.8.12.
[14] Winter, D., Hinderks, A., Schrepp, M. & Thomaschewski, J., (2017). Welche UX Faktoren sind für mein Produkt wichtig? In: S.
Hess & H. Fischer (Eds.), Mensch und Computer 2017—Usability Professionals. Gesellschaft für Informatik e.V. (pp. 191–200).
19
WUDOS 2020 ・Messung der Qualität des Benutzererlebnisses bei der Verwendung von Sprachassistenten ・Klein et al.