Technical ReportPDF Available
Faculty of Technology
University of Applied Sciences Emden/Leer
Protocol for Identifying User
Experience Aspects for Voice
User Interfaces with
Intensive Users
19th Mai 2022
Kristina Kölln
University of Applied Sciences Emden/Leer, Germany
Jana Deutschländer
Berliner Hochschule für Technik, Germany
Andreas M. Klein
University of Seville, Spain
Maria Rauschenberger
University of Applied Sciences Emden/Leer, Germany
Dominique Winter
University of Siegen, Germany
Page 2 of 22
Contents
Contents ...................................................................................................................................................................................... 2
Research Goal ......................................................................................................................................................................... 3
Interviews ................................................................................................................................................................................... 3
Guidelines ............................................................................................................................................................................. 3
Participants ......................................................................................................................................................................... 3
Survey ........................................................................................................................................................................................... 4
Qualitative Content Analysis ..................................................................................................................................... 4
Protocol ................................................................................................................................................................................. 4
Code System ..................................................................................................................................................................... 4
UX Aspects Full Table .......................................................................................................................................................6
Quotes .......................................................................................................................................................................................... 8
Summary ................................................................................................................................................................................... 11
Bibliography ............................................................................................................................................................................ 11
Appendix .................................................................................................................................................................................... 11
A1. German interview guideline ........................................................................................................................ 11
A2. English interview guideline ........................................................................................................................ 13
A3. German family interview guideline ..................................................................................................... 14
A4. Survey English Version ................................................................................................................................... 16
A5. Full List of Codes .................................................................................................................................................. 18
Page 3 of 22
Research Goal
We aimed to explore the User Experience (UX) aspects of voice user interfaces
(VUI) with intensive users. We did this with semi-structured Interviews and an
online survey.
First, this research report provides detailed information about the construction
and participants of the interviews. Second, it gives detailed information about the
construction of the surveys. Third, it lists the interview and survey data and results.
Interviews
Guidelines
We constructed three interview guidelines (see appendix A1 A3) as follows:
The original interview guideline was constructed in German and pretested within
the research team for its practicality. It was then translated into an English Version
for international interviewees. The third version contains supplements for a
participant with family and preschool-age children to take the differences
between the adult’s and child’s use of the VUI. Further information and a detailed
description are included in the publication.
Participants
ID
Transcript
Impairment
Gender
Age
Origin
Profession
Professional
use
Used VUIs
Application
context
Remarks
P1
InterviewD1
Low vision
female
27
Germany
Judicial Officer
none
Alexa
Accessibility,
Smart home
control
none
P2
InterviewK1
Blind
male
58
Germany
Director DZB,
Professor
Realization
Alexa skill for
Librarianship
Alexa, Siri
Accessibility,
Librarianship
tech-affine,
P3
InterviewK3
Blind
male
47
Ireland
Lecturer at
University
none
Siri, (Alexa)
Accessibility,
Smartphone
control
Accent, VUI
used by In-
laws
P4
InterviewJ1
Blind
male
69
?
none
Alexa, Siri
Accessibility
Accent
P5
InterviewJ2
Blind
female
38
Germany
Written
interpreter
Dragon
dictation
software for
written
interpretation
Siri
Accessibility,
Smartphone
control,
Interpreter
P6
InterviewK2
Motor (one
finger per
hand)
male
55
Germany
Representative
for persons with
disabilities;
District
government of
Cologne
Dragon
dictation and
emails
Alexa, Siri
Accessibility,
working tools
Smartphone
control,
Accessibility
tester,
tech-affine
P7
InterviewJDE1
none
male
?
India
Technical
consultant / AI &
IVR Engineer
VUI
development
Alexa, Siri
VUI developer
Not talkative
P8
InterviewJDE2
none
male
31
Germany
IT-Controller
none
Alexa, 'Hey
Audi' voice
assistant
Smart home
control
none
P9
InterviewJDE3
none
male
50
Germany
Software
developer
none
Google Voice
/ Google
Assistant
Set timer,
search queries
none
P10
InterviewJDE4
none
male
?
Germany
?
none
Alexa,
Smartphone
Radio
substitute,
(Fun) search
queries
Family with
children of
preschool
age
Page 4 of 22
Survey
We conducted a survey with German, English, and Spanish speaking participants
using Google Forms from April to June 2021. We constructed the survey in English
as the original language (see appendix A4) and translated it into German and
Spanish.
Qualitative Content Analysis
We carried out the qualitative content analysis [1] with MAXQDA Standard 2020
(Release 20.4.1) software. We present the coding system in the following.
We evaluated the interviews based on the number of mentions per participant,
only the participant who mentions a code once counts.
Protocol
We first did a full content analysis of the interviews. Two authors coded alternating
in the first round, except for one case in which one author coded two interviews in
a row. The second round was then done alternating in reverse order.
After the coding of the interviews, one author coded the qualitative parts of the
survey, following by a second author to revise the coding.
NO.
DATE
ANALYZED UNIT
CODING BY
1
30-05-2021
inteviewJ2.docx
Kristina Kölln
2
05-07-2021
interviewJDE2.docx
Jana Deutschländer
3
09-07-2021
inteviewJ1.docx
Kristina Kölln
4
18-07-2021
interviewJDE1.rtf
Jana Deutschländer
5
27-07-2021
inteviewD1.docx
Kristina Kölln
6
27-07-2021
interviewK2.docx
Jana Deutschländer
7
09-08-2021
inteviewJDE3.docx
Jana Deutschländer
8
29-08-2021
interviewK3.docx
Kristina Kölln
9
14-09-2021
inteviewJDE4.docx
Jana Deutschländer
10
20-09-2021
interviewK1.docx
Kristina Kölln
1 1
20-09-2021
inteviewJ2.docx
Jana Deutschländer
12
22-09-2021
interviewJDE2.docx
Kristina Kölln
13
22-09-2021
inteviewJ1.docx
Jana Deutschländer
14
23-09-2021
interviewJDE1.rtf
Kristina Kölln
15
24-09-2021
inteviewD1.docx
Jana Deutschländer
16
28-09-2021
interviewK2.docx
Kristina Kölln
17
28-09-2021
inteviewJDE3.docx
Kristina Kölln
18
29-09-2021
interviewK3.docx
Jana Deutschländer
19
29-09-2021
inteviewJDE4.docx
Kristina Kölln
20
01-10-2021
interviewK1.docx
Jana Deutschländer
21
11-2021
Survey Answers
Kristina Kölln
22
03-2022
Survey Answers
Jana Deutschländer
Code System
The resulting code system (see appendix A5) is distributed among the analysis
units as follows:
Page 5 of 22
25 individual codes were identified with the interviews. Each codes represents one
UX aspect. In total, these UX aspects were mentioned 127 times among 10
interviews (see figure 1). Seven additional codes were identified among the survey
participants. In total, 477 mentions of UX aspects were given by the survey
participants (see figure 2).
Figure 2 Code distribution in the survey
Figure 1 Code distribution among interviews
Page 6 of 22
UX Aspects Full Table
ID
Aspect
Definition
Example (Translated)
Mentioned
in
Interviews
(N=10)
Mentioned
in Survey
(N=52)
Sum
(N=62)
1
Comprehension
The VUI understands the user
correctly, even if they does not
speak extra clearly
It happens sometimes that it doesn't understand clearly
[Heuristiken für Voice User Interfaces (ResponsesEnglisch)
29/10]
10
37
47
2
Error-free
Both the result and the
operation do not give errors,
wrong answers and
misunderstandings.
i only expect that i'll get the answer or action that i expected
about 50% of the time. i'm not expecting much from VUI at
this point. i also try to limit my requests to things that are
easy and i know of expect the VUI can perform.
(Survey\Heuristiken für Voice User Interfaces
(ResponsesEnglisch): 17|16 - 17|16)
6
34
40
3
Aesthetic
The hardware of the VUI is
supposed to be minimalistic.
Visual feedback about the status
(listening, processing, disabled,...)
is positively received as long as it
is discreet.
Some light to communicate that they hear your voice.
Minimalistic (Survey\Heuristiken für Voice User Interfaces
(ResponsesEnglisch): 15|2 - 15|2)
Whether the Dot device is there or not... it has a purpose
other than looking good. This is my opinion.
3
35
38
4
Range of
functions
The VUI has as many functions
and application possibilities as
possible.
I shut down Cortana because I couldn't find a use for it.
Alexa seems to be far less capable than Siri and much more
prone to error, so I am losing interest in it.
(Survey\Heuristiken für Voice User Interfaces
(ResponsesEnglisch): 19|4 - 19|4)
4
29
33
5
Simplicity
The operation is easy to perform
and contains as few steps as
possible.
It should need a few words only. The less the better.
[interviewJDE1, 50]
8
23
31
6
Effectivity
The user reaches their goal.
"alexa play deutschen pop rock" -> alexa played deutschen
pop rock (Survey\Heuristiken für Voice User Interfaces
(ResponsesEnglisch): 9|24 - 9|24)
9
17
26
7
Support of the
user
The VUI helps users achieve their
goals.
I always have an up-to-date shopping list thanks to Alexa
and my meals don't burn so quickly thanks to the timer
function. (Survey\Heuristics for Voice User Interfaces
(ResponsesGerman): 9|21 - 9|21)
3
22
25
8
Humanity
The user has the feeling of
talking to a human being. They
can conduct a normal dialog, the
VUI persona responds with
humor and empathy, and the
voice sounds natural
So you realize you're making it a little more personal, right?
You act as if it's really a person you're communicating with.
(InterviewK1: 36)
4
21
25
9
Personal
fulfillment
The VUI allows the user to live
out their personality. They can
speak in their dialect and does
not have to disguise themself in
order to be better understood.
Sometimes, of course, they have issues recognizing my
voice, and I would really appreciate a multi-language
experience, it's hard to achieve currently. For example, I'm a
native Russian speaker, and my partner is a native Spanish
speaker, and we both talk in English sometimes. It would be
great to have a possibility to have a custom wake-up
keyword for each language, so that the assistant knows
what language to process next. (Survey\Heuristiken für
Voice User Interfaces (ResponsesEnglisch): 11|22 - 11|22)
3
21
24
10
Context-
sensitivity
The VUI knows its user,
understands the current
situation and can remember the
context of the conversation.
Maintaining (remembering) context, when user asks
followup questions about the same topic.
(Survey\Heuristiken für Voice User Interfaces
(ResponsesEnglisch): 13|29 - 13|29)
Emotion detection and playing things accordingly
(Survey\Heuristiken für Voice User Interfaces
(ResponsesEnglisch): 14|10 - 14|10)
4
20
24
11
Efficiency
The user reaches their goal
without detours.
I do not need fun. My goal is to buy a ticket, just get a
pleasant booking and traveling experience when I travel in
the bus or use the website. The experience should be fast,
the Interface should react, the website should not hang. The
booking process should work. (Interviews\interviewJDE1: 26)
7
17
24
12
Privacy
The VUI should not permanently
listen in, interrupt or even record
private conversations.
We didn't want to have a device at all, because somehow
we found the idea that a device always listens to all
6
16
22
Page 7 of 22
ID
Aspect
Definition
Example (Translated)
Mentioned
in
Interviews
(N=10)
Mentioned
in Survey
(N=52)
Sum
(N=62)
conversations in the house a bit spooky
(Interviews\interviewJDE4: 56)
13
Data security
If personal information must be
provided, it can be trusted that it
will not be shared and will be
handled ethically.
i don't want the audio recording and transcripts to be saved
and associated with me. (Survey\Heuristiken für Voice User
Interfaces (ResponsesEnglisch): 18|20 - 18|20)
7
15
22
14
Time-saving
The user does not need to fetch a
device or press a button and can
immediately start the usage
process. They receives the results
immediately after the request.
I was born with a physical disability, I don't have ten fingers
to type with. I can only type with one finger, so to speak, so I
can type with my right hand and with my left hand I can
basically only activate upper and lower case on the
keyboard. So it's a one-finger system, typing, and I just use it
because it's much, much faster. And by now it's also almost
error-free. Except for proper names or when you speak
unclearly, it's incredibly much faster. (Interview 2: 12)
6
14
20
15
Politeness
The VUI does not insult the user,
lets him finish them sentences
and does not activate without
being asked.
Not listening to the end of the command (it happens with
Google Assistant sometimes) (Survey\Heuristics for Voice
User Interfaces (ResponsesGerman): 12|15 - 12|15)
0
19
19
16
Linking with
third-party
products
Many third-party products
should be compatible. The VUI
can be easily connected with
them and there are no errors in
communication.
That's the next point of criticism, which actually concerns
the interfaces to the manufacturers, which I think could be
improved. Sometimes it works. Then another update comes
along, and then half of it doesn't work anymore.
(Interviews\interviewJDE2: 61)
8
11
19
17
Safety
The VUI gives the user a physical
and privacy security. For
example, by enabling operation
in the car without removing the
hands from the steering wheel,
or by protecting the data from
external access.
And Safety. I did not have to walk around with the phone in
my hand, because the watch is on my wrist.
(Interviews\InterviewK3: 24)
2
16
18
18
Capability to
learn
The VUI can learn new
commands, learns the
personality of its user, and can
exercise appropriate reactions.
Incorrectly learned commands
can be deleted again.
Self-learning systems are great, even if it sometimes seems
a bit scary (Survey\Heuristics for Voice User Interfaces
(ResponsesGerman): 10|18 - 10|18)
2
15
17
19
Intuitiveness
The user does not need to learn a
complicated vocabulary, but can
immediately communicate with
the VUI using their everyday
language. Setting up and
learning the VUI is possible
without additional help.
Uncomplicated set-up ' (Survey\Heuristics for Voice User
Interfaces (ResponsesGerman): 10|29 - 10|29)29)
And for someone who is not familiar with the technology
itself, everything is always very complex and complicated, I
think. (Interview D1: 73)
5
12
17
20
Practicality
The VUI helps the user with
everyday challenges.
Yes, you know how it is. You forget a lamp, and then when
you take your shoes off to go in, you have to take your shoes
off and walk over there. That makes it easier, yeah. That was
actually a ..., maybe it's not such a big highlight today. But
for me, it still adds some value. (InterviewsinterviewJDE2: 57)
7
10
17
21
Reliability
The VUI responds when it is
addressed, without false
activations. The results are
correct and verified. The quality
of the interaction should be
consistently high.
Uncomplicated, unmistakable and guaranteed activation
and deactivation possible (Survey\Heuristics for Voice User
Interfaces (ResponsesGerman): 13|20 - 13|20)
0
15
15
22
Help with Errors
If an error occurs, a way to fix it is
shown. There is a help function.
Comprehension problems, or comprehension problems
without indication of the cause of the problem
(Survey\Heuristics for Voice User Interfaces
(ResponsesGerman): 11|9 - 11|9)
3
11
14
23
Convenience
The user can use the VUI from
any situation without having to
make an effort. For example,
from the sofa, bed or desk.
So of course you're glad that you don't have to reach for
your cell phone or get up or anything. (Interview D1: 43)
7
7
14
Page 8 of 22
ID
Aspect
Definition
Example (Translated)
Mentioned
in
Interviews
(N=10)
Mentioned
in Survey
(N=52)
Sum
(N=62)
24
Fun
The VUI is fun to use, and its
humor is appropriate.
I particularly enjoy asking the device cheeky and silly
questions and seeing how politely it always responds. Of
course, that also contributes to the entertainment, when
you have a visitor, that you can make cheeky remarks and
be entertained with the reactions. (Interviews\interviewK1:
32)
3
8
11
25
Customizability
The persona of the VUI can be
set by the user according to their
preferences (gender, language,
humor, voice,...).
The ability to choose accents and gender in Siri was
unexpected but QUITE enjoyable (Survey\Heuristiken für
Voice User Interfaces (ResponsesEnglisch): 14|4 - 14|4)
0
8
8
26
Flexibility
The VUI can adapt to different
users and situations.
Many skills are still very simple and rudimentary. And then
they offer a list that is read out again and again, where I say,
no, I don't want the list, I want some other information and
there, of course, one wishes for more flexibility, that the
competencies respond more flexibly to one. (InterviewK1:
58)
4
4
8
27
Voice
The voice of the VUI is pleasant
and clearly understandable.
The assistant must have a pleasant voice and voice melody,
otherwise I don't like to use it for a prolonged period of time.
(Survey\Heuristics for Voice User Interfaces
(ResponsesGerman): 10|20 - 10|20)
0
7
7
28
Responsiveness
The VUI responds as soon at it is
adressed, but only when it is
adressed
VUI activated by commands in a television show and was
difficult to shut off. (Survey\Heuristiken für Voice User
Interfaces (ResponsesEnglisch): 11|4 - 11|4)
0
6
6
29
Independency
The user does not need any
assistance in using the VUI. It
allows additional independence
for users who would have
problems operating a GUI (For
example, the visually or motor
impaired, dyslexics, and
children).
Of course I don't need that in my life, but as a blind person I
can benefit a tremendous amount from smartphones and
such assistance systems, because otherwise I would have no
alternative to do certain other things. Or I can only do them
that way at all. Yes. And that's why it's even more important
that such systems play a big role for me in the sense that I
can participate in all kinds of processes. (Interview K1: 81)
4
2
6
30
Innovation
The VUI has new, modern and
unique features.
Since a certain update, there is an option to create tiles on
the far right of the screen. Tiles that contain certain
applications. And at this location, tiles are suggested that I
use in the evening at 18:00. That's not when I open my
Outlook app, that's when I might open my Facebook app.
(InterviewsJDE2: 89)
2
1
3
31
Add-Free
Advertising is not played or can
be turned off
Too much advertising (Survey\Heuristiken für Voice User
Interfaces (ResponsesEnglisch): 18|2 - 18|2)
0
2
2
32
Longevity
The VUI can be used for a long
time, does not break quickly and
does not need to be repeatedly
replaced with the latest model.
Only of the VUI would start lacking features of previously
had. (Survey\Heuristiken für Voice User Interfaces
(ResponsesEnglisch): 18|5 - 18|5)
0
2
2
Quotes
Below we show the quotes in the original form and language used as examples in
the publication and in the UX aspects table.
Original Version Article Quotes
Quote Used
Original Version
Source
Of course, I don’t need it in my life, but as a
blind person, I can benefit greatly from
smartphones and such assistance systems.
selbstverständlich brauche ich das nicht in
meinem Leben, aber ich als blinder Mensch
kann eben von Smartphone und von solchen
Assistenzsystemen unwahrscheinlich
profitieren
(interviewK1, Pos. 81)
as a blind person, typing on an Iphone is just
awkward.
as a blind person, “typing on an iphone is just
awkward”.
(InterviewK3, Pos. 12)
Page 9 of 22
Original Version UX-Aspects Quotes
Quote Used
Original Version
Source
It happens sometimes that it doesn't
understand clearly
It happens sometimes that it doesn't
understand clearly
[Heuristiken für Voice User Interfaces
(ResponsesEnglisch) 29/10]
i only expect that i'll get the answer or action
that i expected about 50% of the time. i'm not
expecting much from VUI at this point. i also
try to limit my requests to things that are easy
and i know of expect the VUI can perform.
i only expect that i'll get the answer or action
that i expected about 50% of the time. i'm not
expecting much from VUI at this point. i also
try to limit my requests to things that are easy
and i know of expect the VUI can perform.
(Survey\Heuristiken für Voice User Interfaces
(ResponsesEnglisch): 17|16 - 17|16)
Some light to communicate that they hear
your voice. Minimalistic
Whether the Dot device is there or not... it has
a purpose other than looking good. This is my
opinion.
Some light to communicate that they hear
your voice. Minimalistic
Also ob jetzt der Dot dasteht oder nicht
dasteht..., er hat einen anderen Zweck als gut
auszusehen. Ist meine Meinung.
(Survey\Heuristiken für Voice User Interfaces
(ResponsesEnglisch): 15|2 - 15|2)
I shut down Cortana because I couldn't find a
use for it. Alexa seems to be far less capable
than Siri and much more prone to error, so I
am losing interest in it.
I shut down Cortana because I couldn't find a
use for it. Alexa seems to be far less capable
than Siri and much more prone to error, so I
am losing interest in it.
(Survey\Heuristiken für Voice User Interfaces
(ResponsesEnglisch): 19|4 - 19|4)
It should need a few words only. The less the
better.
It should need a few words only. The less the
better.
[interviewJDE1, 50]
"alexa play deutschen pop rock" -> alexa
played deutschen pop rock
"alexa play deutschen pop rock" -> alexa
played deutschen pop rock
(Survey\Heuristiken für Voice User Interfaces
(ResponsesEnglisch): 9|24 - 9|24)
I always have an up-to-date shopping list
thanks to Alexa and my meals don't burn so
quickly thanks to the timer function.
Ich habe immer eine aktuelle Einkaufsliste
Dank Alexa und ich essen brennt Dank der
Timerfunktion nicht so schnell an.
(Survey\Heuristics for Voice User Interfaces
(ResponsesGerman): 9|21 - 9|21)
So you realize you're making it a little more
personal, right? You act as if it's really a person
you're communicating with.
Also, Sie merken schon, man personalisiert das
schon ein bisschen, ne? Man, dan tut so als ob
das halt ne wirklich ne ne ne ne Person ist mit
der man da kommuniziert.
(InterviewK1: 36)
Sometimes, of course, they have issues
recognizing my voice, and I would really
appreciate a multi-language experience, it's
hard to achieve currently. For example, I'm a
native Russian speaker, and my partner is a
native Spanish speaker, and we both talk in
English sometimes. It would be great to have
a possibility to have a custom wake-up
keyword for each language, so that the
assistant knows what language to process
next.
Sometimes, of course, they have issues
recognizing my voice, and I would really
appreciate a multi-language experience, it's
hard to achieve currently. For example, I'm a
native Russian speaker, and my partner is a
native Spanish speaker, and we both talk in
English sometimes. It would be great to have a
possibility to have a custom wake-up keyword
for each language, so that the assistant knows
what language to process next.
(Survey\Heuristiken für Voice User Interfaces
(ResponsesEnglisch): 11|22 - 11|22)
Maintaining (remembering) context, when
user asks followup questions about the same
topic.
Emotion detection and playing things
accordingly
Maintaining (remembering) context, when
user asks followup questions about the same
topic.
Emotion detection and playing things
accordingly
(Survey\Heuristiken für Voice User Interfaces
(ResponsesEnglisch): 13|29 - 13|29)
(Survey\Heuristiken für Voice User Interfaces
(ResponsesEnglisch): 14|10 - 14|10)
I do not need fun. My goal is to buy a ticket,
just get a pleasant booking and traveling
experience when I travel in the bus or use the
website. The experience should be fast, the
Interface should react, the website should not
hang. The booking process should work.
I do not need fun. My goal is to buy a ticket,
just get a pleasant booking and traveling
experience when I travel in the bus or use the
website. The experience should be fast, the
Interface should react, the website should not
hang. The booking process should work.
(Interviews\interviewJDE1: 26)
We didn't want to have a device at all, because
somehow we found the idea that a device
always listens to all conversations in the house
a bit spooky
Wir wollten gar kein Gerät haben, weil, ja,
irgendwie die Vorstellung, dass äh ein Gerät
äh alle Gespräche im Haus immer abhört,
fanden wir so ein bisschen spuky.
(Interviews\interviewJDE4: 56)
i don't want the audio recording and
transcripts to be saved and associated with
me.
i don't want the audio recording and
transcripts to be saved and associated with
me.
(Survey\Heuristiken für Voice User Interfaces
(ResponsesEnglisch): 18|20 - 18|20)
I was born with a physical disability, I don't
have ten fingers to type with. I can only type
with one finger, so to speak, so I can type with
my right hand and with my left hand I can
basically only activate upper and lower case on
ich bin mit einer ähm Körperbehinderung
geboren worden, [...] ich habe keine zehn
Finger zum Tippen, ich kann nur mit einem
Finger sozusagen tippen [...] also mit der
rechten Hand kann ich tippen und mit der
(InterviewK2: 12)
Page 10 of 22
the keyboard. So it's a one-finger system,
typing, and I just use it because it's much,
much faster. And by now it's also almost error-
free. Except for proper names or when you
speak unclearly, it's incredibly much faster.
linken Hand kann ich bei der Tastatur im
Grunde nur so die Groß-, Kleinschreibung
aktivieren. Ähm ist dann also ein
Einfingersystem, tippen, und ähm ich nutze es
einfach, weil es viel, viel schneller geht. Und äh
mittlerweile auch fast fehlerfrei. Ähm außer
bei Eigennamen oder wenn man undeutlich
spricht, [...] ist es unglaublich viel schneller.
Not listening to the end of the command (it
happens with Google Assistant sometimes)
Nicht bis zum Ende des Befehls zuhören
(passiert bei Google Assistant manchmal)
(Survey\Heuristics for Voice User Interfaces
(ResponsesGerman): 12|15 - 12|15)
That's the next point of criticism, which
actually concerns the interfaces to the
manufacturers, which I think could be
improved. Sometimes it works. Then another
update comes along, and then half of it
doesn't work anymore.
Das ist dann nämlich auch schon der nächste
Kritikpunkt, also tatsächlich die Schnittstellen
zu den Herstellern, die sind ähm ja
ausbaufähig, sage ich mal. Mal funktioniert es.
Dann kommt wieder ein Update rein, dann
funktioniert wieder die Hälfte nicht.
(Interviews\interviewJDE2: 61)
And Safety. I did not have to walk around with
the phone in my hand, because the watch is
on my wrist.
And Safety. I did not have to walk around with
the phone in my hand, because the watch is
on my wrist. (Interviews\InterviewK3: 24)
(Interviews\InterviewK3: 24)
Self-learning systems are great, even if it
sometimes seems a bit scary
selbst lernende Systeme sind klasse, auch
wenn’s manchmal etwas unheimlich anmutet
(Survey\Heuristics for Voice User Interfaces
(ResponsesGerman): 10|18 - 10|18)
Uncomplicated set-up '
And for someone who is not familiar with the
technology itself, everything is always very
complex and complicated, I think.
Unkomplizierte Inbetriebnahme '
Und für jemanden, der sich mit Technik an
sich nicht auskennt, ist das, finde ich, alles
immer sehr komplex und kompliziert.
(Survey\Heuristics for Voice User Interfaces
(ResponsesGerman): 10|29 - 10|29)29)
(Interview D1: 73)
Yes, you know how it is. You forget a lamp, and
then when you take your shoes off to go in,
you have to take your shoes off and walk over
there. That makes it easier, yeah. That was
actually a ..., maybe it's not such a big highlight
today. But for me, it still adds some value.
Ja, man kennt das ja. Man vergisst eine Lampe,
muss dann, wenn man die Schuhe auszieht,
um reinzulaufen, die Schuhe ausziehen und
dort hinlaufen. Das macht's halt leichter, ja.
Das war tatsächlich ein ..., ist heute vielleicht
nicht mehr so das große Highlight. Aber für
mich doch immer noch ein gewisser
Mehrwert.
(InterviewsinterviewJDE2: 57)
Uncomplicated, unmistakable and
guaranteed activation and deactivation
possible
Unkompliziertes, eindeutiges und garantiertes
Aktivieren und Deaktivieren möglich
(Survey\Heuristics for Voice User Interfaces
(ResponsesGerman): 13|20 - 13|20)
Comprehension problems, or comprehension
problems without indication of the cause of
the problem
Verständigungsprobleme, bzw.
Verständnisprobleme ohne Hinweis auf die
Ursache des Problems
(Survey\Heuristics for Voice User Interfaces
(ResponsesGerman): 11|9 - 11|9)
So of course you're glad that you don't have to
reach for your cell phone or get up or
anything.
Also klar freut man sich, dass man dann nicht
extra zum Handy greifen muss oder aufstehen
muss oder sonst was.
(Interview D1: 43)
I particularly enjoy asking the device cheeky
and silly questions and seeing how politely it
always responds. Of course, that also
contributes to the entertainment, when you
have a visitor, that you can make cheeky
remarks and be entertained with the
reactions.
Äh, besondere Freude bereitet mir ähm, dem
Gerät freche und dumme Fragen zu stellen
und zu gucken, äh wie höflich das dann
immer antworter. Das trägt natürlich auch
immer zur Unterhaltung bei, wenn man mal
besuch hat, dass man mal ähm eben freche
Bemerkungen macht und äh sich da
unterhalten lässt mit dem äh den Reaktionen.
(Interviews\interviewK1: 32)
The ability to choose accents and gender in
Siri was unexpected but QUITE enjoyable
The ability to choose accents and gender in
Siri was unexpected but QUITE enjoyable
(Survey\Heuristiken für Voice User Interfaces
(ResponsesEnglisch): 14|4 - 14|4)
Many skills are still very simple and
rudimentary. And then they offer a list that is
read out again and again, where I say, no, I
don't want the list, I want some other
information and there, of course, one wishes
for more flexibility, that the competencies
respond more flexibly to one.
Äh viele Skills sind da noch sehr sehr einfach
und rudimentär. Und bieten dann ne Liste an,
die dann immer wieder vorgelesen wird, wo
ich sag', ne die will ich nicht die Liste, ich will
andere Informationen und [...] da wünscht
man sich natürlich mehr Flexibilität, dass [ ...]
die Skills da schon flexibler auf auch einen
eingehen.
(InterviewK1: 58)
The assistant must have a pleasant voice and
voice melody, otherwise I don't like to use it for
a prolonged period of time.
Der Assistent muss eine angenehme Stimme
und Stimmmelodie haben, sonst mag ich ihn
nicht längere Zeit nutzen.
(Survey\Heuristics for Voice User Interfaces
(ResponsesGerman): 10|20 - 10|20)
VUI activated by commands in a television
show and was difficult to shut off.
VUI activated by commands in a television
show and was difficult to shut off.
(Survey\Heuristiken für Voice User Interfaces
(ResponsesEnglisch): 11|4 - 11|4)
Page 11 of 22
Of course I don't need that in my life, but as a
blind person I can benefit a tremendous
amount from smartphones and such
assistance systems, because otherwise I would
have no alternative to do certain other things.
Or I can only do them that way at all. Yes. And
that's why it's even more important that such
systems play a big role for me in the sense that
I can participate in all kinds of processes.
Äh selbstverständlich brauche ich das nicht in
meinem Leben, aber ich als blinder Mensch
kann eben von Smartphone und von solchen
Assistenzsystemen unwahrscheinlich
profitieren, weil ich bestimmte andere Dinge
sonst keine Alternative hätte sie zu tun. Oder
ich sie erst mal auf diese Art und Weise
überhaupt tun kann. Ja. Und und das machts
natürlich um so mehr wichtig warum solche
Systeme ähm eben ne große Rolle für mich
spielen im Sinne dessen, dass ich teilhaben
kann an allen möglichen Prozessen.
(Interview K1: 81)
Since a certain update, there is an option to
create tiles on the far right of the screen. Tiles
that contain certain applications. And at this
location, tiles are suggested that I use in the
evening at 18:00. That's not when I open my
Outlook app, that's when I might open my
Facebook app.
Da gibt es seit irgendeinem Update gibt es auf
dem ganz rechten Bildschirm die Möglichkeit,
da werden Kacheln gebildet. Kacheln, die
bestimmte Anwendungen beinhalten. Und an
dieser Stelle werden Kacheln vorgeschlagen,
die ich Abends um 18:00 Uhr nutze. Da werde
ich nicht meine Outlook App öffnen, da werde
ich vielleicht meine Facebook App öffnen.
(InterviewsJDE2: 89)
Too much advertising
Too much advertising
(Survey\Heuristiken für Voice User Interfaces
(ResponsesEnglisch): 18|2 - 18|2)
Only of the VUI would start lacking features of
previously had.
Only of the VUI would start lacking features of
previously had.
(Survey\Heuristiken für Voice User Interfaces
(ResponsesEnglisch): 18|5 - 18|5)
Summary
This research protocol gives more details about the research process regarding UX
aspects for VUIs with intensive users. It gives additional information about the
interview and survey process. The detailed results of the content analysis are
presented among the original versions of all quotes used in this protocol and the
associated article. The protocol will be published alongside the article.
Bibliography
[1] P. Mayring, Qualitative content analysis: theoretical foundation, basic
procedures and software solution. Klagefurt, Germany: Creative Commons -
Namensnennung, Nicht kommerz., Keine Bearbeitung, 2014.
Appendix
A1. German interview guideline
The following pages show the used interview guideline in German.
Interviewleitfaden: Nutzung von VUIs
Einleitungsfragen:
I: Vielen Dank, dass du dich dazu bereit erklärt hast, an diesem Interview teilzunehmen.
Wie besprochen werde ich unser Gespräch zu Dokumentationszwecken aufzeichnen. Ist
das für dich in Ordnung?
B: <<Antwort>>
I: Wir möchten für unsere Studie eine Möglichkeit zur Bewertung von Sprachinterfaces
hinsichtlich ihrer User Experience schaffen. Als Sprachinterface zählt für uns jedes
Interface, dass durch Sprache aktiviert und bedient werden kann. Es kann wiederum mit
Sprache antworten aber auch mit Text oder bestimmten Aktionen. Ein Beispiel wäre Alexa
Page 12 of 22
von Amazon oder auch ein sprachgesteuertes Navigationssystem im Auto. Fallen dir noch
weitere Sprachinterfaces ein, auf die diese Definition zutrifft?
B: <<Antwort>>
I: Würdest du dich als regelmäßigen Nutzer in einem dieser Systeme bezeichnen?
B: <<Antwort>>
I: Und welche(s) wäre(n) das?
B: <<Antwort>>
I: Wie lange nutzt du Sprachinterfaces bereits? Eine ungefähre Schätzung reicht.
B: <<Antwort>>
I: Aus welchem Grund nutzt du allgemein Sprachinterfaces?
B: <<Antwort>>
Spezifischere Fragen zur Entscheidung FÜR ein Produkt:
I: Kannst du mir einmal ein typisches Nutzungsszenario beschreiben? Also beginnend bei
der Aktivierung bis zum Abschluss der Bedienung?
B: <<Antwort>>
I: Gibt es hierbei etwas, dass dir besonders viel Spaß macht oder Freude bereitet?
B: <<Antwort>>
I: Warum denkst du ist das so?
B: <<Antwort>>
I: Gibt es Gründe, warum du dich speziell für <<Füge hier Produkt aus vorheriger Antwort>>
entschieden hast?
B: <<Antwort>>
I: Gab es ähnliche Produkte, gegen die du dich damals aktiv entschieden hast?
B: <<Antwort>>
Wenn Ja: I: Was hat dich dazu bewegt lieber Produkt A zu nehmen?
B: <<Antwort>>
I: Kannst du mir einmal deine positivste Erfahrung mit einem Sprachinterface
beschreiben?
B: <<Antwort>>
I: Was hat dich daran so begeistert?
B: <<Antwort>>
Spezifischere Fragen GEGEN ein Produkt:
I: Hast du im Vergleich dazu auch schon einmal negative Erfahrungen mit einem
Sprachinterface gemacht?
B: <<Antwort>>
I: Was hat dich daran besonders gestört?
B: <<Antwort>>
I: Gibt es etwas, dass du allgemein an Sprachinterfaces kritisieren würdest?
B: <<Antwort>>
I: Sorgt etwas davon dafür, dass du Sprachinterfaces weniger nutzt, als du gerne würdest?
B: <<Antwort>>
I: Hattest du schon einmal den Fall, dass du die Nutzung eines spezifischen Produktes
vollständig aufgegeben hast, weil dich die Nutzung nur noch frustriert hat?
B: <<Antwort>>
Wenn ja: I: Was war die Ursache?
B: <<Antwort>>
Fragen nach Wünschen zur Verbesserung:
I: Was würdest du dir wünschen, damit du Sprachinterfaces (wieder) mehr nutzen würdest
oder du mehr Freude an der Nutzung hättest?
B: <<Antwort>>
I: Was sollte das ideale Sprachinterface können? Hast du eine bestimmte Eigenschaft oder
Funktion vor Augen?
B: <<Antwort>>
I: Gibt es eine Entwicklung bei Sprachinterfaces, die dir Sorgen bereitet?
Page 13 of 22
B: <<Antwort>>
I: Vielen Dank, das war es auch schon. Möchtest du noch irgendetwas loswerden oder
erzählen?
B: <<Antwort>>
A2. English interview guideline
The following pages show the used interview guideline in English.
Interview guideline: VUI Usage
Questions for Introduction:
I: Thank you, for doing this interview with me. As we talked about earlier, I will record our
conversation for documentation purposes. Do you agree with this?
B: <<Answer>>
I: In this study, we want to develop an option to evaluate the user experience of voice user
Interfaces. We define a voice user interface as an interface, which can be activated and
controlled with speech. It may answer with speech, but also with text or actions. One
example would be Alexa from Amazon or a speech-controlled navigation system in an
average car. Do you know any more interfaces, that can be defined this way?
B: <<Answer>>
I: Would you say, that you are a regular user of these kind of interfaces?
B: <<Answer>>
I: Which ones would that be?
B: <<Answer>>
I: How long are you using voice interfaces in general?
B: <<Answer>>
I: What is the reason for you using voice user interfaces?
B: <<Answer>>
Specific questions in favor of one product:
I: Can you describe a typical scenario when you are using your voice interface? Like from
the very start of activation to the end of the operation?
B: <<Answer>>
I: Is there anything in this scenario, that gives you extra pleasure or fun?
B: <<Answer>>
I: Why do you think is it this pleasing?
B: <<Answer>>
I: What were the reasons for you to decide to get <<product of the interviewed person
here>>?
B: <<Answer>>
I: And were there any other similar products, that you ended up not choosing?
B: <<Answer>>
When yes: I: Why did you end up choosing product A?
B: <<Answer>>
I: Could you describe your favorite moment so far with a voice interface?
B: <<Answer>>
I: What made it so special?
B: <<Answer>>
Specific questions against a product:
I: On the other hand: Did you had a negative experience with a voice interface so far?
Could you please describe it for me?
B: <<Answer>>
I: What was especially bothering in this situation?
B: <<Answer>>
Page 14 of 22
I: Is there anything you would criticize about voice interfaces in general?
B: <<Answer>>
I: Are you using voice interfaces less than you would like because of one of these?
B: <<Answer>>
I: Did you have a moment where you stopped using a specific product completely because
the usage was just too frustrating?
B: <<Answer>>
When yes: I: What was the reason?
B: <<Answer>>
Questions about future wishes:
I: If you could make any wish, regardless on already existing technology and features, what
would you choose so you could expand your usage of voice interfaces or so you could have
more fun with the usage?
B: <<Answer>>
I: Thinking about the ideal voice interface, is there any function or characteristic it should
have?
B: <<Answer>>
I: On the other hand, is there a development regarding voice interfaces that worries you?
B: <<Answer>>
I: Thank you very much, that would be all from my part. Is there anything you would like to
add?
B: <<Answer>>
A3. German family interview guideline
The following pages show the used family interview guideline in German.
Interviewleitfaden: Nutzung von VUIs
Einleitungsfragen:
I: Vielen Dank, dass du dich dazu bereit erklärt hast, an diesem Interview teilzunehmen.
Wie besprochen werde ich unser Gespräch zu Dokumentationszwecken aufzeichnen. Ist
das für dich in Ordnung?
B: <<Antwort>>
I: Wir möchten für unsere Studie eine Möglichkeit zur Bewertung von Sprachinterfaces
hinsichtlich ihrer User Experience schaffen. Als Sprachinterface zählt für uns jedes
Interface, dass durch Sprache aktiviert und bedient werden kann. Es kann wiederum mit
Sprache antworten aber auch mit Text oder bestimmten Aktionen. Ein Beispiel wäre Alexa
von Amazon oder auch ein sprachgesteuertes Navigationssystem im Auto. Fallen dir noch
weitere Sprachinterfaces ein, auf die diese Definition zutrifft?
B: <<Antwort>>
I: Habt Ihr eines dieser Systeme im Haushalt?
B: <<Antwort>>
I: Und welche(s) wäre(n) das?
B: <<Antwort>>
I: Wer nutzt dieses Gerät in eurem Haushalt am häufigsten?
B: <<Antwort>>
I: Wie lange nutzt Ihr Sprachinterfaces bzw. Alexa bereits? Eine ungefähre Schätzung
reicht.
B: <<Antwort>>
I: Aus welchem Grund nutzt Ihr Sprachinterfaces allgemein?
B: <<Antwort>>
Spezifischere Fragen zur Entscheidung FÜR ein Produkt:
Page 15 of 22
I: Kannst Du mir einmal ein typisches Nutzungsszenario beschreiben? Also beginnend bei
der Aktivierung bis zum Abschluss der Bedienung?
B: <<Antwort>>
I: Gibt es hierbei große Unterschiede, wenn Ihr oder eure Kinder das Gerät nutzen?
B: <<Antwort>>
I: Könntest Du mir diese genauer erklären?
B: <<Antwort>>
I: Gibt es an der Nutzung etwas, dass Euch besonders viel Spaß macht oder Freude
bereitet?
B: <<Antwort>>
I: Gibt es hier Unterschiede zwischen Euch und euren Kindern?
B: <<Antwort>>
I: Gibt es Gründe, warum Ihr euch speziell für <<Füge hier Produkt aus vorheriger
Antwort>> entschieden habt?
B: <<Antwort>>
I: Gab es ähnliche Produkte, gegen die Ihr euch damals aktiv entschieden habt?
B: <<Antwort>>
Wenn Ja: I: Was hat Dich dazu bewegt lieber Produkt A zu nehmen?
B: <<Antwort>>
I: Kannst du mir einmal eure positivste Erfahrung mit einem Sprachinterface
beschreiben?
B: <<Antwort>>
I: Was hat Euch daran so begeistert?
B: <<Antwort>>
Spezifischere Fragen GEGEN ein Produkt:
I: Habt Ihr im Vergleich dazu auch schon einmal negative Erfahrungen mit einem
Sprachinterface gemacht?
B: <<Antwort>>
I: Was hat Euch daran besonders gestört?
B: <<Antwort>>
I: Gibt es eine Art Kindersicherung? Oder können die Kinder alle möglichen Funktionen
von Alexa theoretisch mitnutzen?
B: <<Antwort>>
I: Wie hast Du deinen Kindern Alexa nähergebracht bzw. erklärt?
B: <<Antwort>>
I: Worauf hast Du dabei besonders viel wert gelegt?
B: <<Antwort>>
I: Welche Funktionen verwenden deine Kinder besonders häufig?
B: <<Antwort>>
I: Wie gehen deine Kinder damit um, wenn Alexa sie nicht richtig versteht?
B: <<Antwort>>
I: Gibt es etwas, dass Du allgemein an Sprachinterfaces kritisieren würdest?
B: <<Antwort>>
I: Sorgt etwas davon dafür, dass Ihr Sprachinterfaces weniger nutzt, als Ihr gerne würdet?
B: <<Antwort>>
I: Hattet Ihr schon einmal den Fall, dass einer von Euch die Nutzung vollständig
aufgegeben hat, weil die Nutzung nur noch frustriert hat?
B: <<Antwort>>
Wenn ja: I: Was war die Ursache?
B: <<Antwort>>
Fragen nach Wünschen zur Verbesserung:
I: Wenn Du Dir irgendetwas wünschen könntest, egal ob es nach aktuellem Stand der
Technik realistisch ist oder nicht, was würdest Du Dir für eine Eigenschaft oder Funktion
für Sprachinterfaces wünschen?
Page 16 of 22
B: <<Antwort>>
I: Glaubst Du, dass deine Kinder einen anderen Wunsch hätten? Was würden sie sagen?
B: <<Antwort>>
I: Gibt es eine Entwicklung bei Sprachinterfaces, die Dir Sorgen bereitet?
B: <<Antwort>>
I: Vielen Dank, das war es auch schon. Möchtest Du noch irgendetwas loswerden oder
erzählen?
B: <<Antwort>>
A4. Survey English Version
Page 1
Your Experiences with Voice User Interfaces
This is a survey to gain detailed insights about the usage of voice user interfaces (or VUI).
We will ask you about your experiences and expectations with VUIs, positive and negative.
Based on the results we will develop a set of heuristics for the evaluation of the usability
and user experience for VUIs
You don't have to be an expert on heuristics or VUI. The only requirement for participating
is having routine in the usage of a VUI.
You may answer in english or german.
Thank you for considering participation in this study.
Legal Informations
Voluntary Participation
Participation in this study is voluntary. You may decline to answer any questions that you
do not wish to answer and you can withdraw your participation at any time by not
submitting your responses. A few demographic questions at the start are however
mandatory for us to consider your answers.
Confidentiality
You need to know that your identity will be confidential. All of the data will be summarized
for publication, and no individual could be identified from these summarized results. The
data collected from this study will be maintained on a password-protected computer
database in a restricted access area of the university.
For all questions, contact [anonymous] de or the supervisor of the study [anonymous]
Page 2
About You
Age
Short answer text (Numerical)
Gender
Page 17 of 22
Multiple choice (Male, Female, Divers, Prefer not to say)
Location (Country)
Short-answer text
Profession
Short-answer text
Page 3
Voice User Interfaces
Many different definitions and wordings for voice user interfaces exist. Our definition when
talking about voice user interfaces (or VUI) in this questionnaire is:
A Voice User Interface is any kind of interface or device that is controlled by voice. It can
answer with voice, actions or text.
Examples of VUIs are intelligent personal assistants like "Siri" or "Alexa", but also less
intelligent interfaces like your average car navigation system which can only follow a few
defined commands or dictation software like "Dragon Naturally speaking".
Page 4
Your experience with VUIs
How often do you use VUIs?
Multiple choice (more than 5h a day, a few hours a day, less than an hour a day, a
few times a week, a few times a month, I don’t use VUIs regularly, other…)
Why do you use VUIs?
Checkboxes (fun, comfort, professional reasons (as a tool at work), professional
reasons (scientific research), motorical handicapped, visually handicapped, other…)
Which VUIs did you use so far? If there are many, name those you use the most.
Long-answer text
Page 5
Features of VUIs
Please think about the VUIs you have already used and the characteristics they had or
lacked.
You can give one or more answers to every question.
If you can't answer a specific question leave it blank.
We want to especially consider accessibility.
Feel free to add unique experiences, should you have any kind of handicap.
Please take your time answering these questions. We would appreciate detailed answers.
Page 18 of 22
Could you describe a positive experience with a VUI you had?
Long-answer text
What is the first positive characteristic a VUI should have, that comes to your mind?
Long-answer text
Could you describe a negative experience with a VUI you had?
Long-answer text
What is the first negative characteristic a VUI should avoid, that comes to your mind?
Long-answer text
What are other important characteristics of a VUI in your opinion?
Long-answer text
What positive feature of a VUI surprised you the most?
Long-answer text
Are you expecting additional visual feedback from your VUI? (e.g. light, graphics, text)
Long-answer text
What obstacles are you expecting when using a VUI?
Long-answer text
Was there any problem with using a VUI you did not expect?
Long-answer text
Is there a characteristic of a VUI that would make you stop using it?
Long-answer text
Was anything missing with the VUIs you used so far?
Long-answer text
Page 6
Thank you for participating!
Do you want to leave any comment?
Long Answer text
A5. Full List of Codes
List of Codes
Frequency
Codesystem
832
UX-Aspects New
0
Politeness
19
Reliability
15
UnwanteActivating without being askedd activation
1
Page 19 of 22
Reliability
5
Customizability
8
Voice
7
Gender of the voice
0
Responsiveness
6
Add-Free
2
Longevity
1
Lacking previous features in future versions
1
UX-Aspects
0
Comprehension
77
Pro spec. VUI: good voice recognition
2
Contra spec. VUI: Missunderstanding, not understanding
input
8
neg. Experience: VUI does not understand user
10
neg. Experience: Pronunciation must be very clear
2
neg. Experience: Siri doesn’t understand accent
(Irisch) that well
1
neg. Experience: corrections needed
5
Aesthetic
35
Suggestion for improvement: More discreet hardware
1
Criticism: instead of echo "chunk" prefer hidden function
1
Criticism: Alexa Echo visual feedback through ring instead
of verbal
2
Criticism: Alexa activation beep sometimes too quiet
1
Error-free
40
Against use: error-proneness dependence on VUI
2
Contra: Siri calls 911 by mistake
1
Contra spec. VUI: unexpected inappropriate answers
2
Contra spec. VUI: Manual control necessary for data entry
2
Desired function: Be able to transcribe fluent speech
1
neg. Experience: Unpleasant consequences due to VUI
errors
2
Criticism: Siri too high error rate when dictating
2
Range of functions
29
Pro/Contra spec. VUI: Range of functions
2
Extended range of functions
2
Siri desired function timetable information
1
Desired function: Borrow/Return library items
1
Desired function: Voice input for TV streaming
1
Simplicity
43
Evaluation criterion: habit
1
Pro / Contra spec. VUI: Number of necessary I/O actions
4
neg. Experience: Complicated set-up
2
Page 20 of 22
posit. Experience: Listen to music that is difficult to play on
iPad
1
Support of the user
24
Pro spec. VUI: Wellbeing/Health
1
Personal Fulfilment
21
Pro Usage: Broadening the horizon
1
Target function: Accent recognition
2
posit. Experience: Podcasts/music suggestions expand
horizons
2
Humaneness
36
Evaluation Criteria: Humor
2
posit. Experience: Humorous answers to humorous
questions
3
Conversation skills
2
Suggestion for improvement: Empathy
1
Evaluation criterion: free sentences usable
2
Criticism: too command-driven, more like conversation
1
Context-sensitivity
20
Desired function: Better focus on the user
1
context-sensitive intelligence for input evaluation
2
Suggestion for improvement: Empathy
1
Criticism: error-prone with unusual spelling
1
posit. Experience: Context-sensitive answers
1
Effectivity
23
neg. Experience: The desired goal is not achieved
5
posit. Experience: first working timer operation
1
posit. Experience: Goal achieved
2
Efficiency
28
neg. Experience: None or delated reaction
2
posit. Experience: Search query with short exact answer
2
Privacy
16
Contra Usage: Environment listens in
4
Privacy: approaches with audio and image go too far
1
Privacy: Passwords are read out by Siri
2
Privacy: Eavesdrop
1
Target function: (de)activation function
1
Criticism: Device listens to all conversations
1
neg. Experience: Environment catches everything
2
Safety
20
Pro usage: teaching physical safety
1
neg. Experience: Unpleasant consequences due to VUI
errors
2
Data Security
35
Page 21 of 22
Pro/Contra spec. VUI: Implement data security
trustworthiness
3
Data protection: Option not to use data
1
Data protection: Only allow ethical data use
1
Data protection: Payment option instead of data
usage
1
Data protection: Recording the conversations
1
Data protection: Usability of data only with active
consent
5
Data protection: Without consent data only for
machine learning
1
Data protection: Audio fingerprint
1
Data protection: Too much power with individual
corporations
3
Data protection: Right to be forgotten
1
Capability to learn
23
Pro spec. VUI: Learning ability Trainability
2
Time-saving
26
Pro Usage: Fastness
3
Intuitiveness
25
Evaluation Criterion: Operating request
0
Operating request: Accentuation
1
Operating request: Soundscape / other speakers
4
Operating request: Intuitive control
1
Operating request: Concentration
1
Operating request: Pace
1
Operating request: Vocabulary known
2
Help with errors
11
neg. Experience: Lack of assistance in case of error
1
Target function: Feedback in case of errors
1
Target function: Help function
3
Linking with third-party products
13
Pro spec. VUI: Product already used, operable through VUI
7
Screen reader compatibility
2
Pro spec. VUI: Siri since already iPhone users
1
Pro spec. VUI: Link with Amazon Prime
1
Contra spec. VUI: Siri instead of Google Voice, as
Apple users
2
neg. Experience: Faulty Alexa Skills for external providers
4
posit. Experience: Apple Watch with Siri combined
1
Practical
43
Evaluation Criterion: Facilitation of tasks
10
Target function: simplify repetitive, tedious
processes
0
Control washing machine
1
Page 22 of 22
Pro Usage: Simplification in everyday life
1
Pro Usage: repetitive, facilitate processes
2
Pro spec. VUI: Simplification
3
Sollfunktion: Smarter -> simplify complex tasks
3
Pro spec. VUI: Operation possible despite distance from
the device
4
posit. Experience: first switching off of all lamps from front
door
1
posit. Experience: Be able to listen to news at any time
1
Fun
15
Pro spec. VUI: Fun
3
Convenience
26
posit. Experience: Additional movement becomes
unnecessary
3
posit. Experience: Simple and practical
3
Flexibility
11
Contra: only exact command works, no concretize
2
Desired functions: Different languages selectable
2
Criticism: Oversensitive
1
Criticism: Dragon only one language for speech
recognition adjustable
1
Independency
12
Evaluation Criterion: Independence for young children
1
Understands childrens language
2
Target function: Child lock
2
Accessibility: Participation through voice control
1
Criticism: Interpreting -data online or much local. Memory
needed
1
neg. Experience: Strong dependence on internet
connection
1
neg. Experience: Third party software/product required
2
Innovation
1
Pro Usage: Curiosity
1
Criticism: lack of intelligence, complexity very limited
3
posit. Experience: more possibilities, open for new things
1
Pro/Contra spez. VUI: in der Ausbildung empfohlen
1
Pro spez. VUI: Freunde haben so ein Gerät bereits
1
Pro spez. VUI: am weitesten Verbreitet
4
... Afterwards, we translated the interview guidelines into English to include international participants and made an additional version to interview a user whose children also use the VUI. The interview guidelines are available in the original language German and English translation in the research protocol (Kölln et al., 2022). ...
... Two interviews had to be documented with a memory log because the recording failed. All transcriptions and memory logs are available in the research protocol in their original language (Kölln et al., 2022). Afterwards, the collected data was analyzed with the qualitative content analysis (Mayring, 1994). ...
... Our final questionnaire contains 19 questions about the experiences and expectations of the VUI users regarding the VUIs. The complete questionnaire can be found in the research protocol in English, German, and Spanish versions (Kölln et al., 2022). The survey was then shared on the social media platforms LinkedIn, Facebook, and Twitter as well as through the personal networks of the authors. ...
Conference Paper
Full-text available
Voice User Interfaces (VUIs) are becoming increasingly available while users raise, e.g., concerns about privacy issues. User Experience (UX) helps in the design and evaluation of VUIs with focus on the user. Knowledge of the relevant UX aspects for VUIs is needed to understand the user’s point of view when developing such systems. Known UX aspects are derived, e.g., from graphical user interfaces or expert-driven research. The user’s opinion on UX aspects for VUIs, however, has thus far been missing. Hence, we conducted a qualitative and quantitative user study to determine which aspects users take into account when evaluating VUIs. We generated a list of 32 UX aspects that intensive users consider for VUIs. These overlap with, but are not limited to, aspects from established literature. For example, while Efficiency and Effectivity are already well known, Simplicity and Politeness are inherent to known VUI UX aspects but are not necessarily focused. Furthermore, Independency and Context-sensitivity are some new UX aspects for VUIs.
ResearchGate has not been able to resolve any references for this publication.