PreprintPDF Available
Preprints and early-stage research may not have been peer reviewed yet.

Abstract

In interaction or user-centered design practices, it is common to employ interviews and think-aloud techniques to gather data about user behavior. These techniques enable researchers to learn about how users think and use technologies during the design and user testing process. However, such techniques involve accessing audio feedback, which may require workarounds if the researcher identifies as deaf or hard of hearing (DHH). We report on a project led by a DHH researcher in which workarounds to audio access resulted in methodological changes. We discuss the implications of these adjustments.
Interview and Think Aloud Accessibility for Deaf and Hard of
Hearing Participants in Design Research
Becca Dingman
bad6955@g.rit.edu
School of Information
Rochester Institute of Technology
Rochester, NY, USA
Garreth W. Tigwell
garreth.w.tigwell@rit.edu
School of Information
Rochester Institute of Technology
Rochester, NY, USA
Kristen Shinohara
kristen.shinohara@rit.edu
School of Information
Rochester Institute of Technology
Rochester, NY, USA
ABSTRACT
In interaction or user-centered design practices, it is common to
employ interviews and think-aloud techniques to gather data about
user behavior. These techniques enable researchers to learn about
how users think and use technologies during the design and user
testing process. However, such techniques involve accessing audio
feedback, which may require workarounds if the researcher identi-
es as deaf or hard of hearing (DHH). We report on a project led by
a DHH researcher in which workarounds to audio access resulted
in methodological changes. We discuss the implications of these
adjustments.
CCS CONCEPTS
Human-centered computing Accessibility.
KEYWORDS
accessibility, research methods, design methods
ACM Reference Format:
Becca Dingman, Garreth W. Tigwell, and Kristen Shinohara. 2021. Interview
and Think Aloud Accessibility for Deaf and Hard of Hearing Participants in
Design Research. In The 23rd International ACM SIGACCESS Conference on
Computers and Accessibility (ASSETS ’21), October 18–22, 2021, Virtual Event,
USA. ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/3441852.
3476526
1 INTRODUCTION
In user-centered design, interview and think aloud methods are
popular ways to learn about users’ thoughts and preferences when
using technologies [
1
,
17
]. Gathering this information from users
can help designers and researchers to know how users respond
to and use technologies, which in turn can help designers and
researchers to envision new and improved designs. Using these
methods, we investigated the experiences of DHH podcast users [
4
].
Our aims were to understand personal interest, accessibility and
future design considerations. Although podcasts are a popular way
for people to get information about a specic topic (the average
American podcast listener averages 8 podcasts per week) [
13
], few
podcasts provide transcripts, and podcast-hosting platforms are not
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for prot or commercial advantage and that copies bear this notice and the full citation
on the rst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’21, October 18–22, 2021, Virtual Event, USA
©2021 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-8306-6/21/10.
https://doi.org/10.1145/3441852.3476526
designed in a way that allows users to access transcripts. Through-
out our work, we made methodological adjustments to account
for the inaccessibility of our research process. Specically, DHH
participants who’s primary language is ASL adjust to think-aloud
protocols by describing their actions rst, then completing the ac-
tion, rather than simultaneously speaking and doing. Additionally,
DHH participants vary widely in communication preferences, re-
quiring nuanced adjustments for interview technique. We reect
and report on these experiences in the context of completing a de-
sign investigation into the accessibility of podcast technologies for
DHH users. We provide suggestions for conducting similar work
and for future research.
2 ADAPTING RESEARCH METHODS
We make use of a multitude of research methods in HCI, and must
select the most suitable method to answer our research questions [
9
].
For example, conducting interviews when we want to understand
why people react in a particular way to new technologies [
19
], de-
ploying diary studies to capture data in-the-wild and overcome the
associated memory biases with interviews and questionnaires [
8
],
or running controlled lab-based studies to accurately compare in-
teraction methods [
6
]. In HCI research, the think aloud method has
provided design researchers with a way to capture what technology
users are thinking as they test designs [
2
,
11
,
12
,
21
], illuminating
insights into why people do what they do when using technology.
Although eort is directed toward selecting the best research
methods, we must consider whether those methods are appropriate
for the participant sample. For example, little work has focused
on adapting design process methods and tools for cultural dier-
ences [
3
], despite data indicating cultural background can alter
response behaviors in specic usability settings [
10
,
20
], and the
accessibility implications of this are unknown [18].
Although the think aloud protocol is useful for gaining insight
into users’ thinking in-the-moment [
2
], it is yet unclear how such
a protocol ought to be adjusted (if at all) for users who do not
employ verbal communication. For example, DHH people have
their own culture and language system (e.g., ASL in the US) [
5
,
14
],
which has generated interest in adapting traditional methods we
used in HCI. Previous work investigated the feasibility of think
aloud methods with DHH people nding the method appropriate,
and providing guidance for working with interpreters during think
aloud sessions [
15
,
16
]. However, such work did not evaluate the
eectiveness of the method, or investigate adjustments for DHH
researchers. In other work, researchers successfully translated the
System Usability Scale into ASL nding it maintained criterion
validity and internal reliability during an evaluation study [
7
]. Thus,
ASSETS ’21, October 18–22, 2021, Virtual Event, USA Dingman et al.
there are opportunities to further reect on whether we need to
dene new processes for other data collection methods for DHH
participants.
3 METHODS
Our investigation into accessible podcast design involved user as-
sessment interviews and feedback on prototypes [
4
]. The rst au-
thor, who identies as DHH, conducted the semi-structured inter-
views and prototype feedback sessions to understand the experi-
ences DHH users have with podcasts and their desires for features
to include on podcast platforms. Participants were recruited for the
interviews, and then invited back to give feedback on the proto-
types designed from the interview ndings. A second prototype
session with a dierent group of participants elicited feedback on a
nal design. See [
4
] for more details on the DHH podcast platform.
3.1 Interviews
In the interviews, participants were asked questions about their
experience with podcasts, their desires to change the current plat-
form they use, their ideas on how transcripts should appear on
the platform, and other features that could enhance their experi-
ence while listening to podcasts [
4
]. We discussed using automatic
speech recognition (ASR) to auto-transcribe podcasts, including a
threshold for potential errors.
Interview participants were recruited through community social
media groups and email lists, including the “NTID Community”
listserv. The “RIT Cross-Registered Community” are individuals
who are DHH students supported by NTID but are taking RIT
classes. NTID support students receive access support including:
captioning services, ASL interpreters, and note-taking services.
We recruited participants who were at least 18 years of age, who
identied as deaf or hard of hearing, and who followed a podcast,
or wanted to follow a podcast. Interviews were 20 to 45 minutes
and were conducted on a video-conferencing software, Zoom, to
limit in-person interactions due to the COVID-19 global pandemic.
The participants were entered into a $25 rae as compensation.
All interviews were conducted in dierent communication meth-
ods based on participants’ preferences. For example, interview
preferences included simultaneous communication of Spoken Eng-
lish and American Sign Language (ASL), also known as “simcom,
through a real-time captionist, with Google’s ASR transcription
app, Live Transcribe, as well as in ASL. The interviews were video
recorded with participant consent on Zoom.
Seven participants were interviewed. Their ages ranged between
21 to 27. Four identied as male, one as female, and two as non-
binary. All except for one participant had some experience with
listening to a podcast. We show participant hearing status, lan-
guages used daily, and preferred communication method for inter-
views in Table 1. While some participants preferred Simcom, some
participants preferred to use Signed Exact English, or SEE.
3.2 Think Aloud With Prototypes
Initial low-delity prototypes were created using Figma based on
the information gathered from the interviews. The prototypes were
then shared remotely with four of the participants from the original
interviews who identied as DHH. Due to the small number of
participants for this phase of the project, we refrain from more
specically identifying preferred communication methods.
Feedback sessions were 20 to 30 minutes each and were con-
ducted on Zoom. Participants were asked to interact with the low-
delity prototype through the link provided on Figma, and to share
their screens and click through the prototype using their mouse
or trackpad. Participants were given one of seven scenarios at
a time and asked to work through the prototype thinking aloud
[
2
,
11
,
12
,
21
]. Some participants conducted the think aloud in ASL
while some did so in spoken English. Those who used ASL signed
what they planned to do then interacted with the prototype.
Based on participant feedback on the low-delity prototypes,
high-delity prototypes were created. To assess these high-delity
prototypes, a new group of participants was recruited to give feed-
back. Four new participants were recruited who had not previ-
ously been interviewed and had not previously interacted with the
low-delity prototype. All participants identied as DHH, three
identied as female, and one as male. Again, we refrain from pro-
viding specic participant communication preferences due to the
low number of participants.
The high-delity prototype feedback sessions ranged from 20 to
30 minutes each and were conducted on Zoom. During the feedback
session, participants were asked to rst explore the prototype, and
then to complete a set of tasks using a Figma rendition of the
high-delity prototype. Participants completed the same seven
tasks as in the low-delity feedback session. During the feedback
sessions, participants were asked to use the think aloud protocol.
Specically, they were instructed to think aloud in ASL or English
when completing the tasks. Again, we observed that participants
who used ASL signed what they planned to do before interacting
with the prototype.
4 REFLECTIONS AND DISCUSSION
We highlight that we encountered some unexpected interactions
when conducting this work. First, was that participants preferred
a variety of dierent approaches to the interview process. Second,
we observed that some participants conducted their think aloud in
ASL by signing what they planned to do before interacting with
the prototype.
4.1 Variation in Interview Preferences
As this work was led by a DHH researcher, they were able to rec-
ognize dierent communication preferences of individual partici-
pants and respond accordingly. Thus, we were able to collect addi-
tional information about participant communication preferences.
We highlight that the variation in communication preferences per-
haps should be taken into account in such interviews, such as the
DHH researcher did in this case. In addition, because the inter-
views were about podcast accessibility—and in some cases, about
the clarity of podcast speakers—knowing communication prefer-
ences could help the researcher better understand the participant
preferences when dealing with podcast audio.
Interview and Think Aloud Accessibility for Deaf and Hard of Hearing Participants in Design Research ASSETS ’21, October 18–22, 2021, Virtual Event, USA
Table 1: Interview Participant Hearing Status and Preferred Communication
ID Hearing Status Languages Used Daily Preferred Communication
P1 hard of hearing Simcom, English, ASL Simcom, Spoken English
P2 deaf Simcom, English, SEE Simcom
P3 hard of hearing English, Hindi, Japanese Spoken English
P4 deaf/hard of hearing English Spoken English, ASL
P5 deaf English, Thai Written English
P6 hard of hearing Simcom, English, ASL Spoken English
P7 hard of hearing Simcom, English, ASL Spoken English
4.2 ASL Signers and the Think Aloud Protocol
Whereas prior work found gestural think aloud protocols for DHH
participants comparatively feasible to sessions conducted with hear-
ing participants in terms of comment quantity [
15
,
16
], we observed
in both feedback sessions that some participants signed their think-
aloud portion of the task using ASL before commencing with the
task. Although we did not observe issues as a result throughout
the feedback sessions (we did not observe participants changing
their “think-aloud” statements once they completed their tasks),
we consider that the sequential order of steps slightly alters the
“in-the-moment” benet of the think aloud protocol. Specically,
this adjustment enabled us to gain insight into what participants
planned to do, but it did not allow us to take advantage of the
think aloud protocol to its full potential because on-the-y actions
or thoughts may not be captured. Larger studies, or studies that
engage more complex user tasks may disadvantage ASL signers
using think-aloud by not suciently capturing their in-the-moment
reactions. In addition, there may be wide variation in how it might
be addressed. For example, participants in our study signed rst,
and then completed the tasks. However, for more complex tasks,
participants might pause mid-task to add their thoughts, or add
more thoughts post-task. Finally, whereas other work also provided
recommendations for working with interpreters [
16
], such as to
address prompts or comments that may be lost in translation, this
work was led by a DHH researcher, ameliorating at least some
potential communication issues.
5 CONCLUSION
We conducted a user assessment and prototyping study with DHH
participants about the accessibility of podcasts. In doing the work,
we observed some slight adjustments from audio/verbal interview-
ing and think-aloud protocols to better suit the needs of DHH
participants with varying communication preferences. For our lim-
ited study, we did not observe that these adjustments impacted data
collection or analysis. However, we reect that larger studies or
more complex usability testing may warrant additional consider-
ation to ensure these procedures are accessible and inclusive. We
provide here some examples of how these adjustments were made
and accounted for in our work. Future work may investigate other
ways to ensure accessibility of these and similar procedures when
including DHH researchers and participants.
REFERENCES
[1]
Chadia Abras, Diane Maloney-Krichmar, Jenny Preece, et al
.
2004. User-centered
design. Bainbridge, W. Encyclopedia of Human-Computer Interaction. Thousand
Oaks: Sage Publications 37, 4 (2004), 445–456.
[2]
Ted Boren and Judith Ramey. 2000. Thinking aloud: Reconciling theory and
practice. IEEE transactions on professional communication 43, 3 (2000), 261–278.
[3]
Apala Lahiri Chavan. 2005. Another culture, another method. In Proceedings of
the 11th International Conference on Human-Computer Interaction, Vol. 21.
[4]
Becca Dingman, Garreth W. Tigwell, and Kristen Shinohara. 2021. Designing a
Podcast Platform for Deaf and Hard of Hearing Users. In The 23rd International
ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, USA)
(ASSETS ’21). Association for Computing Machinery, New York, NY, USA. https:
//doi.org/10.1145/3441852.3476523
[5]
Karen Emmorey, Stephen M Kosslyn, and Ursula Bellugi. 1993. Visual imagery
and visual-spatial language: Enhanced imagery abilities in deaf and hearing ASL
signers. Cognition 46, 2 (1993), 139–181.
[6]
Tiago João Vieira Guerreiro, Hugo Nicolau, Joaquim Jorge, and Daniel Gonçalves.
2010. Assessing Mobile Touch Interfaces for Tetraplegics.In Proceedings of the 12th
International Conference on Human Computer Interaction with Mobile Devices and
Services (Lisbon, Portugal) (MobileHCI ’10). Association for Computing Machinery,
New York, NY, USA, 31–34. https://doi.org/10.1145/1851600.1851608
[7]
Matt Huenerfauth, Kasmira Patel, and Larwan Berke. 2017. Design and Psy-
chometric Evaluation of an American Sign Language Translation of the Sys-
tem Usability Scale. In Proceedings of the 19th International ACM SIGACCESS
Conference on Computers and Accessibility (Baltimore, Maryland, USA) (AS-
SETS ’17). Association for Computing Machinery, New York, NY, USA, 175–184.
https://doi.org/10.1145/3132525.3132540
[8]
Reed Larson and Mihaly Csikszentmihalyi. 2014. The experience sampling
method. In Flow and the foundations of positive psychology. Springer, 21–34.
[9]
Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser.2017. Research methods
in human-computer interaction. Morgan Kaufmann.
[10]
Jung-Joo Lee and Kun-Pyo Lee. 2007. Cultural Dierences and Design Methods
for User Experience Research: Dutch and Korean Participants Compared. In
Proceedings of the 2007 Conference on Designing Pleasurable Products and Interfaces
(Helsinki, Finland) (DPPI ’07). Association for Computing Machinery, New York,
NY, USA, 21–34. https://doi.org/10.1145/1314161.1314164
[11]
Clayton Lewis. 1982. Using the" thinking-aloud" method in cognitive interface
design. IBM TJ Watson Research Center Yorktown Heights, NY.
[12]
Clayton Lewis, John Rieman, and Task-Centered User Interface Design. 1993. A
Practical Introduction. University of Colorado, Boulder, Department of Computer
Science (1993).
[13]
Ang Li, Alice Wang, Zahra Nazari, Praveen Chandar, and Benjamin Carterette.
2020. Do podcasts and music compete with one another? Understanding users’
audio streaming habits. In Proceedings of The Web Conference 2020. 1920–1931.
[14]
Carol Padden, Tom Humphries, and Carol Padden. 2009. Inside deaf culture.
Harvard University Press.
[15]
Vera Roberts and Deborah Fels. 2002. Methods for inclusion: employing think
aloud protocol with individuals who are deaf. In International Conference on
Computers for Handicapped Persons. Springer, 284–291.
[16]
Vera Louise Roberts and Deborah I Fels. 2006. Methods for inclusion: Employing
think aloud protocols in software usability studies with individuals who are deaf.
International Journal of Human-Computer Studies 64, 6 (2006), 489–501.
[17]
Yvonne Rogers, Helen Sharp, and Jenny Preece. 2011. Interaction design: beyond
human-computer interaction. John Wiley & Sons.
[18]
Garreth W. Tigwell, Kristen Shinohara, and Laleh Nourian. 2021. Accessibility
Across Borders. In CHI ’21 Workshop: Decolonizing HCI Across Borders (CHI
Workshop ’21). 1–4. https://arxiv.org/abs/2105.01488
[19]
Sarah J. Tracy. 2013. Qualitative Research Methods: Collecting Evidence, Crafting
Analysis, Communicating Impact. Wiley-Blackwell.
[20]
Helma Van Rijn, Yoonnyong Bahk, Pieter Jan Stappers, and Kun-Pyo Lee. 2006.
Three factors for contextmapping in East Asia: Trust, control and nunchi. CoDe-
sign 2, 3 (2006), 157–177. https://doi.org/10.1080/15710880600900561
[21]
MW Van Someren, YF Barnard, and JAC Sandberg. 1994. The think aloud method:
a practical approach to modelling cognitive. London: AcademicPress (1994).
... (RQ3) How do DHH users prefer to receive podcast transcripts? See [3] for a reflection on using interview and think-aloud methods with DHH participants. ...
Preprint
Full-text available
Listening to podcasts is a popular way for people to spend their time. However, little focus has been given to how accessible pod-cast platforms are for Deaf and Hard-of-Hearing (DHH) people. We present a DHH-centered accessible podcast platform prototype developed with user-centered design. Our proposed design was constructed through semi-structured interviews (n=7) and prototype design feedback sessions (n=8) with DHH users. We encourage podcast platform designers to adopt our design recommendations to make podcasts more inclusive for DHH people and recommend how podcast hosts can make their shows more accessible. CCS CONCEPTS • Human-centered computing → Accessibility.
Preprint
Full-text available
Listening to podcasts is a popular way for people to spend their time. However, little focus has been given to how accessible pod-cast platforms are for Deaf and Hard-of-Hearing (DHH) people. We present a DHH-centered accessible podcast platform prototype developed with user-centered design. Our proposed design was constructed through semi-structured interviews (n=7) and prototype design feedback sessions (n=8) with DHH users. We encourage podcast platform designers to adopt our design recommendations to make podcasts more inclusive for DHH people and recommend how podcast hosts can make their shows more accessible. CCS CONCEPTS • Human-centered computing → Accessibility.
Book
Research Methods in Human-Computer Interaction is a comprehensive guide to performing research and is essential reading for both quantitative and qualitative methods. Since the first edition was published in 2009, the book has been adopted for use at leading universities around the world, including Harvard University, Carnegie-Mellon University, the University of Washington, the University of Toronto, HiOA (Norway), KTH (Sweden), Tel Aviv University (Israel), and many others. Chapters cover a broad range of topics relevant to the collection and analysis of HCI data, going beyond experimental design and surveys, to cover ethnography, diaries, physiological measurements, case studies, crowdsourcing, and other essential elements in the well-informed HCI researcher's toolkit. Continual technological evolution has led to an explosion of new techniques and a need for this updated 2nd edition, to reflect the most recent research in the field and newer trends in research methodology. This Research Methods in HCI revision contains updates throughout, including more detail on statistical tests, coding qualitative data, and data collection via mobile devices and sensors. Other new material covers performing research with children, older adults, and people with cognitive impairments.
Conference Paper
In usability studies, designers and researchers frequently use subjective questions to evaluate participants' impression of the usability of some product. The System Usability Scale (SUS) is a popular standardized questionnaire consisting of ten English statements about the usability of a product, to which participants indicate their agreement on a five-point scale. Many deaf adults in the U.S. have lower levels of English reading literacy, but there are currently no standardized questionnaires similar to SUS for Deaf and Hard-of-Hearing (DHH) users who are fluent in American Sign Language (ASL). To facilitate the inclusion of such users in studies, we created an ASL translation of SUS following accepted methods of survey translation: using a bilingual team including native ASL signers who are members of the Deaf community, along with back-translation evaluation to determine whether the meaning of the original was preserved. To validate whether key psychometric properties were preserved during translation, we deployed the ASL instrument in a study with 30 DHH participants. By comparing the results to users? responses to another measurement instrument, along with scores from 10 additional DHH participants responding to the original English SUS, we verified the criterion validity and internal reliability of the new "ASL-SUS." We are disseminating the translated instrument to promote the inclusion of DHH users in HCI research studies or in usability testing of consumer products.
Article
Contextmapping techniques have been tried and tuned for participants in Western cultures and are known to provide inspiration in the conceptual phase of design. Because these techniques rely heavily on activities such as expressing feelings in public and discussing in groups, they are less attuned to participants from more ‘reserved’ cultures, e.g. East Asia. In this project we adapted the techniques for use with East Asian participants. Our findings indicate that, when conducted in appropriate forms, contextmapping techniques can work in East Asia. However, more than in the West, a well-demarcated script is needed. By ‘script’ we mean a construct that frames the roles of the participants and the researcher, and provides a clear stage on which the participant plays the role of ‘expert of his experience’ and outside of which he/she is free from the burden of expressing him- or herself. The importance of a script in East Asia led us to review the value of scripting and staging as design parameter for the techniques in general.