Available via license: CC BY-NC 4.0
Content may be subject to copyright.
1
Supplementary information
I Persona descriptions
1. Margo: A mother and professional with multiple demands on her time.
Margo is a middle-aged professional who is beginning to realize that the demands of her day take a
greater toll on her than they had when she was younger. Specifically, the ability to follow conversations
declines as the day progresses, and her overall alertness at work is limited by the fatigue that sets in
after multiple meetings. Given her underlying mild hearing loss she could be an ideal candidate for a
hearing instrument which helps her overcome these difficulties, a few barriers lie in the way: 1) the root
cause of fatigue or communication errors is not widely associated with hearing problems, 2) the
potential benefit of better hearing is not easily quantified, and 3) there is a public stigma associated with
wearing ear-level devices.
Moving Margo from simply understanding the problem to a grasping the value of assistance is the
primary goal of the suggested innovative path. One thought is that the stigma that comes from wearing
a device that projects her age and disability can be outweighed by better education and changes in
marketing of acceptable accessories (i.e. fashion trends in hearing devices). The first step is educating
Margo on the issues she faces in her work day and how they are related to her auditory environment.
Future developments could provide an advanced acoustic monitoring ear-level device that is capable of
cataloging the scenes she encounters, marking the moments when communication breaks down, and
recording other biophysical measures. The power in providing the user with individualized data is that it
supports autonomous decisions on intervention (e.g. step count activity monitors). Moving beyond
monitoring capabilities, the same device, equipped with a receiver, could provide need-based
amplification as well as more sophisticated audio processing when Margo is in a noisy or reverberant
environment. With time, and assuming more general acceptance of ear-worn devices, this team
anticipates that all will benefit from a device that acts as one’s personal “master knowledge interface.”
Such capabilities would include: instant foreign-language translation, a mind-body monitor and
conscience, enhanced social awareness and acute guidance.
Though the primary focus of these technologies was to address potential “pain points” in Margo’s day,
the team also addressed the potential of enhancing one of Margo’s “gain points” – specifically, the
moment when she spoke to her college-aged daughter by phone before the workday began. We suspect
that communication over long distances must become more immersive and ultimately provide shared
audio (and/or visual) space.
2. Paul: Normally hearing young male who is very busy with lots of things going on in various
areas across his life.
Paul is a normal hearing 25 year-old who is looking for ways to simplify his busy, distractible life. He
works in the technology sector and is always excited to try out new gadgets that can improve his
experience in the world. He does not currently have any hearing problems and his primary concerns are
tuning out unnecessary information in his life so he can really focus. Paul gets anxious, stressed and
overloaded with all of the projects he has to do, is constantly forgetting about events that he has in his
calendar and is generally easily distracted. Paul believes that it would be helpful to have a device that
2
could direct his attention to only necessary information in the acoustic environment. He wants to be
able to focus on his work, but remain engaged in the world around him and not put himself in physical
danger if he misses alerting sounds. He also wants a way to integrate his schedule in this device so that
he never forgets what he needs to do. Ideally, these would be non-intrusive notifications that gently
remind him of his responsibilities. Paul sees technology as a way to improve his life and a way to help
him cope with the frenetic experience of modern day life—he wants to tune out unnecessary
distractions and tune in to his work to be productive and focused.
Paul is looking for technological advances to improve his life, but also becomes very nervous about using
applications and programs that do not have specific privacy policies that ensure that his data will not be
shared with third parties. This is an anxiety that he shares with many others in his industry. He only
downloads apps that have clear privacy settings and that will not share his data without his consent.
Many of the other working groups focused on improving the quality of life for hearing-impaired
individuals, but Paul’s concerns and desires as a normal hearing individual should also direct the future
of research in auditory science.
3. Bruce: Violinist with increasing hearing impairment.
Bruce is a violinist who developed a significant unilateral hearing impairment from failure to wear
hearing protection. Bruce is losing pleasure in music and in social interactions and conversation
(particularly political conversations). His teachers didn’t reinforce the importance of the use of hearing
protection so Bruce is keen to raise awareness of this very important issue.
Focusing on the needs of Bruce, this working group was inspired by one of the presentation rants
(Stankovic: https://www.listeninginto2030.org/future-focus) and felt that biologic intervention is a real
possibility in the future. However, medical interventions, as we learned with cochlear implants, require
realistic expectations regarding subsequent function and dedication to extensive aural rehabilitation.
Cochlear hair cell regeneration may be a real possibility in the near future, but there could still be some
deterioration in the auditory nerve or elsewhere along the auditory pathway. Thus, technology such as
hearing aids or FM systems may still be needed for those undergoing medical interventions in the
future. Plus, prevention of hair cell loss or use of hearing protection and awareness of noise exposure
limits will continue to remain important in the future, especially if the new normal is everyone wearing a
hearable device. This topic that was brought up many times throughout the workshop.
4. Om: A 60 year-old farmer in India with growing hearing impairment.
Om’s persona came to life out of thinking about how science and technology might impact people in the
developing world. Keeping in mind the different daily challenges faced in this community, we developed
Om, an older, middle-class Indian farmer living in a remote village, who is increasingly concerned about
his hearing loss and the impact this has on his daily life. On a growing basis, Om finds the tasks of his
daily life increasingly frustrating, which in turn, has a significant impact on his health. Going to the post
office in a nearby town to pay his electricity bill, Om finds that interacting with everyone along the way
is a challenge. When he finally returns home from his day, he is exhausted and feels increasingly isolated
from his family. The daily challenges that Om face are intensified by his developing hearing-loss because
it affects his ability to interact in the world as well as his overall sense of well-being. Living in a remote
town in India also means that to get help, Om will need to travel a long distance to see a Doctor.
3
The challenge for Om is to develop a reliable, cost-effective, wearable device that focuses on his well-
being and adapts along with his evolving needs – a doctor in Om’s ear. Meeting this challenge requires
technology that not only provides effective hearing compensation, but also meaningful bio-markers to
help track his overall well-being and which communicates remotely with clinicians to monitor his overall
health.
5. Nancy: A busy mother with a large family and a partner with some hearing disability leading
to disengagement.
Social isolation is interfering with Nancy’s interactions with people around the her. There is a need to
increase her capacity to communicate. Nancy is by all means a generic character, but therein lies her
appeal. She is the person that everyone knows: the friend, the co-worker, the neighbor. A middle-class
housewife from Anytown, USA.
In their youth, when Nancy and her husband were avid musicians, she chose to be proactive about using
hearing protection, but her husband shunned that as an inconvenience. She is now paying the price for
his imprudence. Miscommunication with a husband who staunchly refuses to admit that his hearing is
failing is quite stressful. He eschews the fact that he did not hear his wife tell him to pick up their
teenage daughter, a blunder that led their daughter to abscond with her boyfriend, leaving Nancy
distraught. While standard hearing aids may have prevented this disaster from happening, our team
imagined a more sophisticated technology capable of closing the communication loop. Not only must
the listener hear what the speaker says, but the speaker must know they have been understood. If
Nancy could have a message receipt, so-to-speak, confirming there had or had not been semantic
processing of her words, she could have judged better whether or not to reiterate her message. When
an email bounces, we often receive a reply informing us of such, and what if our hearing technology
could do the same? In-ear EEG devices are an active area of development now and it was not
inconceivable that a device capable of tracking and confirming human speech processing could be
created by 2030.
Going further, what if such devices could be used to communicate in a multilayer, multichannel mode,
regardless of proximity; communicating not only the bare information, but multiple layers of supporting
cues? Along with his hearing loss, Nancy's husband's memory had begun to slip as well. When Nancy
rings him up on his wearable, it sounds like his wife is standing right in front of him. She is reminding him
to pick up the kids. He can feel through his device the sternness his wife is trying to communicate by
using her bio-sensing technology. Reflecting on the events of the night before, he quickly switches over
to the kids’ channels and checks in on each one. Feeling with his device the flippancy in his daughter’s
response, he knows he better hurry to get her. And that boy's voice in the background seemed a bit too
close for comfort, anyway.
6. Jane: Teacher with normal hearing
Jane, a teacher in her mid-thirties, evolved from a discussion of the ways in which social and behavioral
issues could be addressed within the context of a world comfortable with auditory augmentation.
Frustrated with a classroom full of distracted students and a general sense of diminishing control, Jane is
a normal hearing individual who would be able to benefit from many helpful features of acoustic
technology. For example, in a world where all students are equipped with wearable devices, Jane would
4
be able to use biofeedback measures such as attention-tracking and EEG to determine whether or not a
given activity was successful or unsuccessful at engaging students. A student-by-student data
breakdown after class would provide her with information about individuals, highlighting points in the
lesson plan that proved more or less effective and allowing her to identify students who remained
disengaged throughout the presentation of the material. These data could be used to predict test and
homework scores and identify students who need extra help earlier rather than later. Similarly, a room
full of students using wearable devices would allow Jane the opportunity to design and control an
immersive classroom environment that is conducive to learning, in which salient auditory features are
enhanced and unwanted sounds are filtered out. Jane could dynamically modify input to individual
students’ devices according to what might help them learn best; for example, a student with trouble
concentrating may perform better with soothing background noise.
7. Jessica and Fernando: Relationship building and reinforcement
In contrast to the other single person personas, our working group conceived of a married couple as the
persona at the center of our journey maps. By focusing on the couple, we could emphasize that
communication is a two party activity. Communication difficulties in one person create negative
spillover effects in both parties. Fernando has service related traumatic brain injury (TBI) that has
caused a spectrum of challenges to his ability to communicate with his wife, the hardworking mother,
Jessica. In addition to hearing impairment, Fernando also has mild cognitive impairment and
depression. Jessica is frustrated that she has become socially isolated because Fernando will not go out
salsa dancing. She is also frustrated with the day to day living with Fernando because Fernando does
not hear her requests (e.g., pick up Nando, their son, from school). Fernando is ashamed to go out into
stressful (eg. complex auditory) environments such as clubs and restaurants, and is depressed because
he cannot be the man he was for Jessica. Both Jessica and Fernando struggle with how best to
communicate their feelings and limitations to each other.
II Meta-comments on the session format for other researchers hosting similar
workshops (ECR view points)
For other research disciplines seeking to host a similar workshop, we can offer the following guidelines
and evaluation of the process (e.g., our lessons learned). Overall it was positive. The link between the
rants (short talks by a few of the researchers on selected topics) and the actual brainstorming sessions
could have benefited from discussion of the rants by the group and/or within the context of the
personas.
Some of the senior researchers felt that gearing discussion towards problems in audition, whether these
be issues of attention processing or perception, put the focus on near term issues. The advantage of
this approach is that the themes and motifs become grounded in situations that are readily apparent,
and embodied by the personas. On the other hand, a contrasting opinion was that Apple’s knowledge
navigator might never have arisen from such a discussion, nor other innovations such as the internet.
These visionary ideas were not conceived to solve a problem that was apparent so much as to solve a
problem that people didn’t know they had (e.g. sharing information through handheld devices across
the world). But experience form this workshop suggests that solutions to persona-based problems
5
benefit from a broader view of “problem” to mean identifying issues that the personas themselves
wouldn’t actually list as problems.
While not explicitly stated, we believe that the confluence of many different researcher disciplines was
excellent for gaining perspectives on problems as viewed from those fields. For example, how would an
audiologist view a universal listening device that is easy to use and does not require fitting and tuning?
Such a device takes away a most gratifying component of their job and likely renders the current
business model financially unsustainable.
Diversity among discussants at each table probably did reduce the focus of their positions based on a
“lowest common denominator” effect when proposing technical research directions. To use an analogy,
while themes were a fertile soil for context in which to find seeds of research ideas, the seeds
themselves were inchoate and sparser than one might have desired. While the themes identified are
crucial as guides for research directions, more specifics are needed before specific proposals could be
built. Certainly, technical definitions of these seeds were sharpened by senior researchers whose
valuable experience helped to warn of bottlenecks, timelines, and areas of risk, but this should not
detract from the variety of interesting ideas that arose from wider consideration of the potential users
of communication devices.
It would have been nice to have had a video such as the knowledge navigator that predicts the next
innovative device(s) of 2030 and, perhaps this workshop will challenge another group of diverse and
influential auditory researchers to exercise their innovative thinking towards consumer needs. Thus, one
of the most important outcomes of this meeting may be that the 40+ attendees, as they move forward
with their research questions, may adapt their goals in a way that connects more closely to user needs
in wearable technology.
III Meta-comments on the session format for other researchers hosting similar
workshops (participant view points)
Participant reactions were along three lines. First, appreciation for the bold use of design thinking to
envision futuristic solutions for speech and hearing problems. Second, some expressed discomfort at the
change of perspective that is required by design thinking’s emphasis on the end user’s experience. Third,
some themes or issues deemed important by the participant were not covered in the workshop.
Reactions along these three directions are summarized in turn next.
Several participants noted that the workshop was effective at identifying themes to guide future
technology developments and associated basic and applied research that will produce products in the
next couple of decades to meet user need. One participant said “… these themes provide a nice
roadmap to how researchers can participate in the ongoing information processing revolution being
driven by the amazing computing power that is available in less and less expensive packages …”.
While participants appreciated the importance of the design thinking approach, some found it an
unsettling experience to use it in a workshop for envisioning basic and applied research fifteen years
hence. One source of the discomfort was that the approach “required us to change our perspective and
to leave behind our areas of expertise and instead imagine scenarios involving a fictitious person … only
at the very end were we able to start applying what we brought to the workshop …”. Another wondered
6
if the approach was “more useful for product design more than for the basic science requirement that is
needed to make this radical new product”. On the other hand, one participant felt that the value of the
design thinking approach was insufficiently realized – “the conclusions reached by the group were
heavily biased towards our own areas of study … found that to be ironic since the mediated process was
designed to encourage participants to consider the subject’s (i.e., the person with hearing loss)
perspective”. This was perhaps because, as one participant noted, “most of us were not ready to think
along those lines (the perspective of the user of a hearing aid)”.
Some participants noted that perhaps the format and participant makeup of the workshop contributed
to missing an opportunity to highlight specific issues that will be important in coming decades. One
noted “missing … are solutions that focus on how the members of society will need to change to use
these technologies … learning how people react to these complex predictive and responsive information
arrays and how we can improve the abilities of humans to perceive and understand may be even more
important than helping to develop technologies that can do the perceiving and understanding for us”.
Another felt that the workshop output was suffused with a sense of techno-panacea and insufficient
recognition of what will remain a significant problem in two decades – “… there will undoubtedly be
people with hearing loss in 2030 whose hearing is degraded to the point that they are suffering from
social isolation … our solutions were of the sort that we would “solve” hearing thus the psychosocial
issue would subsequently be solved … the psychosocial issues shall remain, and those issues deserve
and demand attention”.
From the organizers’ perspective, it was not unexpected that many participants were discomfited by the
exercise. This would have been especially true for the academic and basic scientists for whom design
thinking is unfamiliar. This discomfort could be seen as positive if it led participants to think in
unaccustomed and novel ways. Such ferment might perhaps have been better utilized if we had
recruited more and a greater diversity of participants from industry. Achieving such diversity might be
something that future organizers of such a workshop might prioritize.
Table 1 List of workshop participants. * indicates early career researcher.
Participants
Affiliation
Madalyn
Bates
Ammunition Group LLC
Josh
Bernstein
Walter-Reed Medical Center
Simon
Carlile
Starkey Hearing Technologies
Maria
Chait
University College London
Edward
Chang
University of California, San Francisco
Greg
Ciccarelli*
Massachusetts Institute of Technology
Jane
Cockburn
Kairos Now
Anna
Diedesch*
Oregon Health Sciences University
Brent
Edwards
Earlens
Carol
Espy-Wilson
University of Maryland
Megan
Finnegan*
University of Illinois, Urbana-Champaign
Jack
Gallant
University of California, Berkeley
Frederick
Gallun
National Center for Rehabilitative Auditory Research
7
Ray
Goldsworthy
University of Southern California
Jennifer
Groh
Duke University
Ervin
Hafter
University of California, Berkeley
Simon
Henin*
New York University Medical Center
Ingrid
Johnsrude
Western University
Sridhar
Kalluri
Starkey Hearing Technologies
Alexander
Kell*
Massachusetts Institute of Technology
Edmund
Lalor
University of Rochester
Zach
London
Ammunition Group LLC
Richard
Lyon
Google
Josh
Miele
Smith Kettlewell Eye Research Institute
Lee
Miller
University of California, Davis
Peggy
Nelson
University of Minnesota
Erol
Ozmeral*
University of South Florida
Jochem
Rieger
University of Oldenburg
Casey
Roark*
Carnegie Mellon University
Matt
Rolandson
Ammunition Group LLC
Jessica
Sagers*
Harvard University
Shihab
Shamma
University of Maryland
Barbara
Shinn-Cunningham
Boston University
Malcolm
Slaney
Google
Doug
Solomon
IDEO
Konstantina
Stankovic
Harvard Medical School
Chris
Stecker
Vanderbilt University
Richard
Stern
Carnegie Mellon University
Frederic
Theunissen
University of California, Berkeley
Tim
Trine
Starkey Hearing Technologies (now EarGo)
David
Whitney
University of California, Berkeley
Eric
Young
Johns Hopkins University
Fan-gang
Zeng
University of California, Irvine