Content uploaded by Garreth W. Tigwell
Author content
All content in this area was uploaded by Garreth W. Tigwell on Feb 08, 2024
Content may be subject to copyright.
Party Face Congratulations! Exploring Design Ideas to Help
Sighted Users with Emoji Accessibility when Messaging with
Screen Reader Users
CARLA F. GRIGGIO, Aarhus Universiy, Denmark and Aalborg Universiy, Denmark
BENJAMIN M. GORMAN, Bournemouth University, United Kingdom
GARRETH W. TIGWELL, School of Information, Rochester Institute of Technology, USA
Emoji are a popular, expressive form of non-verbal communication. However, people often use emoji in ways
that result in confusing or cumbersome screen reader output. We created two accessibility support designs: (1)
Preview, which displays a basic text transcript of a message with emoji that a screen reader would narrate,
and (2) Alert, which summarises potential accessibility issues caused by emoji within a message. We explored
our designs using an online survey and provided participants with the option to edit messages that contained
emoji, should they choose to do so. We collected 1508 modied messages from 116 sighted participants and
conducted a qualitative analysis of the data to identify the strategies participants used when asked to edit a
message for accessibility issues and their appreciation of each design. We found that participants preferred
the Preview design over Alert since it allows for subjective interpretations of what constitutes an accessible
message. We report sighted users’ rewriting strategies (e.g., editing the message to move the emoji to the end)
and incorrect assumptions about screen readers that would lead to using textual markers that are incompatible
with screen readers. We discuss the design implications for future systems for accessible messaging.
CCS Concepts: • Human-centered computing
→
Empirical studies in collaborative and social com-
puting; Empirical studies in accessibility.
Additional Key Words and Phrases: Accessibility, Computer-mediated communication, Emoji, Messaging,
Screen Readers
ACM Reference Format:
Carla F. Griggio, Benjamin M. Gorman, and Garreth W. Tigwell. 2024. Party Face Congratulations! Exploring
Design Ideas to Help Sighted Users with Emoji Accessibility when Messaging with Screen Reader Users. Proc.
ACM Hum.-Comput. Interact. 8, CSCW1, Article 175 (April 2024), 31 pages. https://doi.org/10.1145/3641014
1 INTRODUCTION
Emoji are widely used as a visual means of expression in daily communication. In the context
of messaging (e.g., SMS, WhatsApp, WeChat), people often use emoji to convey emotions and
conversational tone (e.g., “Thank you ”) or even as simple decorations to their messages (e.g.,
“Making tacos ”) [
7
]. People have also appropriated emoji as a means to express their identities [
49
,
58
] and intimacy within close relationships [
18
,
19
]. However, little work has investigated sighted
people’s awareness of how the use of emoji can create inaccessible messages when vocalised by a
screen reader, which is a device often used by blind and low vision people.
While emoji can be powerfully expressive, prior work found that the way in which emoji are
used can be inaccessible to screen reader users due to sighted people focusing on the visual aspects
Authors’ addresses: Carla F. Griggio, cfg@cs.aau.dk, Aarhus Universiy, Aarhus, Denmark and Aalborg Universiy, Copen-
hagen, Denmark; Benjamin M. Gorman, bgorman@bournemouth.ac.uk, Bournemouth University, United Kingdom; Garreth
W. Tigwell, School of Information, Rochester Institute of Technology, Rochester, NY, USA, garreth.w.tigwell@rit.edu.
This work is licensed under a Creative Commons Attribution-ShareAlike International 4.0 License.
© 2024 Copyright held by the owner/author(s).
ACM 2573-0142/2024/4-ART175
https://doi.org/10.1145/3641014
Proc. ACM Hum.-Comput. Interact., Vol. 8, No. CSCW1, Article 175. Publication date: April 2024.
175:2 Carla F. Griggio, Benjamin M. Gorman, & Garreth W. Tigwell
of emoji and not considering how emoji are interpreted non-visually [
51
]. The Unicode Consortium,
a non-prot organisation, determines both the rough visual appearance and textual descriptions
of each emoji [
52
]. We will refer to the textual description for an individual emoji in our paper as
an emoji descriptor, which a screen reader vocalises when describing an emoji in a message. For
example, the emoji descriptor for is face with tears of joy. When emoji are included as part of a
sentence, such as “That’s great ”, the content of the vocalised screen reader output would be,
“That’s great face with tears of joy”.
If sighted individuals are unaware of how screen readers vocalise emoji descriptors, their use
of emoji may lead to miscommunication or an uncomfortable experience for screen reader users.
For example, in an interview study with blind and low-vision people, Tigwell et al. [
51
] document
how sighted users’ lack of awareness of emoji descriptors may cause grammatical errors when
replacing words with emoji (e.g., "It is today" would be voiced as "It is sun today", not "sunny")
or produce excessively long messages due to the overuse of decorative emoji. Furthermore, emoji
can often be interpreted or re-appropriated based on their visual appearance rather than their
descriptors [
21
,
50
,
55
], resulting in voiced messages that fail to convey the meaning intended by
the sender. For example, a sighted user may use to illustrate a "high ve" gesture, but screen
readers will voice it as "pray" or "hands pressed together" [51].
These issues make emoji an interesting case study for exploring ways to enhance text-based
communication between sighted and blind or low vision users. Only creating accessibility tools
for blind and low vision users might not be enough to improve the accessibility of conversations
with sighted users, as this excludes sighted users from understanding how their messages are
interpreted at the receiving end. Therefore, our research aims to understand how to reduce the
burden associated with these accessibility challenges for screen reader users by designing tools
that assist sighted users in adjusting their messages with emoji to be more compatible with screen
readers. To guide our design process, we had three key research questions:
RQ1. In what ways do sighted users adjust their use of emoji in text messages when they are
aware that the receiver is using a screen reader?
RQ2. What kind of assumptions and misconceptions about how screen readers interpret
emoji hinder sighted users’ eorts to improve emoji accessibility?
RQ3. How can we promote the adoption of best practices for using emoji in ways that are
more compatible with screen readers?
We conducted an online survey with 116 sighted users of messaging apps to collect examples of
how they would rephrase a given message with emoji (e.g., "I got an A in the exam ") to make
it more accessible to a conversation partner using a screen reader. Our participants were exposed
to three dierent designs: (A) No Support—the message is shown in a typical messaging platform
with no guidance, (B) Preview Support—the message is augmented with a text transcription of how
it would be voiced by a screen reader (e.g., “I got an A in the exam smiling face with sunglasses
smiling face with sunglasses smiling face with sunglasses”), and (C) Alert Support—the message
is augmented with an alert about potential accessibility problems (e.g., “ emoji appears three
times—emoji descriptors may be voiced consecutively”). The elicited rewritten messages allowed
us to assess sighted users’ awareness of dierent types of potential emoji accessibility issues, their
intuitions on using emoji when messaging with a screen reader user, tensions between favouring
accessibility or personal expression, and their preferences between the design ideas.
Our participants used a variety of message rewriting strategies, ranging from removing emoji
(e.g., rethinking the message from scratch, replacing the emoji with additional text) to adapting
the use of emoji (e.g., adding verbal clarications of the intended meaning or tone of the emoji).
Through analysing these strategies, we identied incorrect assumptions and misunderstandings
Proc. ACM Hum.-Comput. Interact., Vol. 8, No. CSCW1, Article 175. Publication date: April 2024.
Party Face Congratulations! 175:3
about how screen readers work, potentially introducing more accessibility issues instead of reducing
them (e.g., replacing emoji with punctuation-based emoticons that lack screen reader support),
even among participants who reported prior accessibility knowledge. Finally, we found that most
of the participants preferred Preview over Alert, mainly because Preview was viewed as less
constraining, allowing participants to assess accessibility issues on their own.
Our paper makes three contributions: First, we propose two designs to support sighted users in
considering the accessibility of emoji they use when messaging with screen reader users. Second,
we identify a range of strategies employed by sighted users to enhance the accessibility of messages
that include emoji, which uncover assumptions and common misconceptions about screen readers
that often result in additional accessibility issues. And third, we oer design recommendations for
tools dedicated to help sighted users message in more accessible ways.
2 BACKGROUND AND RELATED WORK
2.1 Importance of Emoji in Computer-Mediated Communication
Emoji are graphic illustrations used within text-based communication that have gained immense
popularity [
3
], in part because of the vast range of dierent concepts, ideas, and things that can be
represented in visual short-hand with emoji [
1
]. For example, there are sets of emoji for emotions
, people , animals , food , ags , objects , symbols , etc., and new sets are released
each year [
4
]. While emoji are often viewed as fun and playful [
33
], the signicance of their role
within current communication should not be underestimated [57].
Researchers from elds such as CSCW, HCI, and linguistics have been studying the ways in
which emoji have become a part of how we communicate, and it is clear that emoji have become a
complex language in their own right. People may use emoji to clarify or enhance the intent behind
a message [
7
], and emoji can be repurposed to signify jokes between friends, present stories in
picture form, and to support relationships [
7
,
18
,
19
,
44
,
50
,
55
,
58
]. Moreover, the inclusion of
modiers for emoji such as skin tone modiers supports improved self-representation by users [
42
].
Workforce leadership using emoji in email can be viewed as more likeable, although there is
a complex relationship in how this manifests depending on sender and receiver genders and
workplace culture [
41
], and there are hesitations from new employees in using emoji in virtual
workspaces until interpersonal bonds have been adequately formed and co-workers have been
observed to see how they are using non-textual responses [46].
The versatility of emoji is both its strength and weakness since there can often be misunder-
standings due to design, people’s dierent interpretations of what the emoji are, or unclear intent
in how some people use emoji [
34
–
36
,
43
,
50
], as well as considering the inuence of individual
factors such as cultural and personality on emoji use [
22
,
27
,
30
]. With this in mind, we need to
design systems to help mitigate emoji issues and guide best practices for using emoji.
2.2 Accessibility support within collaboration and communication contexts
There are approximately 43 million people worldwide who are blind [
48
] and estimates for the
number of people worldwide who have a vision impairment rises to 2.2 billion [
54
]. Therefore, it is
important that communication technology and related digital services remain accessible.
One piece of assistive technology that is often used by blind and low vision people is a screen
reader. A screen reader is software that will read aloud the contents on a computer display [
23
],
for example, the items listed in a web page menu for the user to determine what they would
like to access. However, for a screen reader to be able to do this, it requires the designers and
developers of digital technology and services to follow certain implementation steps to ensure
that the screen reader understands how to interpret the structure and features of an interface. For
Proc. ACM Hum.-Comput. Interact., Vol. 8, No. CSCW1, Article 175. Publication date: April 2024.
175:4 Carla F. Griggio, Benjamin M. Gorman, & Garreth W. Tigwell
example, it is possible for a screen reader to read aloud what is in an image but only if the app
or website creator has dened alternative text (image descriptions) within the underlying HTML
when posting the image. Often digital systems and digital content are inaccessible to screen readers
due to poor implementation [
23
,
26
,
39
], and this has also been observed within CSCW settings
such as collaboration and communication.
First, regarding collaborative settings, research has highlighted the challenges people with vision
impairments encounter during collaborative writing [
8
], and has explored design solutions to
improve the accessibility of collaborative document editing for screen readers [9, 10, 24].
Second, regarding communication, research has demonstrated the widely used QWERTY key-
board layout is not optimised for use with a screen reader and the alternative layout of the keyboard
letters can provide improved typing during text messaging [
29
]. In addition to keyboards, the way
in which people type out their words can also be inaccessible to screen reader users. Moreover,
Lee and Ashok [
25
] conducted a study that revealed how blind participants faced signicant cogni-
tive and interaction challenges when their screen reader encountered out-of-vocabulary words
because the screen reader does not narrate the content adequately, and, as a result, the screen
reader user has to access the content multiple times and in various ways to comprehend these
unfamiliar terms. Therefore, casual language with elements like abbreviations (e.g., I love you vs
ily), misspellings, wordplay, slang, etc. can have a detrimental impact on the comprehension and
social media experience of screen reader users.
Finally, the structure of social media platforms and how people use them has generally created
access issues for screen reader users [
13
,
15
,
37
,
51
]. For example, Twitter became less accessible
when the platform allowed images to be posted [
37
], and more recent work studied the extent
of this issue where very few users include embedded image descriptions for screen readers to
access [
13
]. Furthermore, research has found a lack of alternative text for common visual media
people share both privately and publicly such as GIFs [16] and memes [17].
These ndings are a snapshot of a much larger issue surrounding sighted people’s general lack
of awareness and understanding of producing accessible content for screen readers. In summary,
screen reader accessibility relies on both the creation of accessible digital systems and for people to
create content in such a way that is accessible to screen reader users. In our work, we are interested
in exploring the challenges surrounding emoji use in messages that will be read by a screen reader.
2.3 Emoji Accessibility
Some emoji designs are unambiguous for sighted users, such as the “thumbs up” emoji , but there
are examples where sets of emoji can be visually similar while having dierent meanings. The
issues surrounding this are notable when the person receiving the wrong emoji is using a screen
reader since the screen reader will read a specic emoji descriptor aloud. For example, the “face
with big pleading eyes” emoji and the “smiling face holding back tears” emoji are extremely
similar in design, and a sighted person could send the wrong one, but the emoji descriptors “face
with big pleading eyes” and “smiling face holding back tears” are dierent.
Tigwell et al. [
51
] published one of the rst pieces of work on emoji accessibility highlighting
challenges screen reader users face, such as searching and selecting emoji, and receiving messages
that are cumbersome or unclear due to how emoji are used (e.g., repetitive use of emoji, placement
of emoji, replacing words with emoji and the emoji descriptor causing grammatical errors).
Selecting emoji is typically a visual search task, and it can be dicult for most users due to the
growing number of emoji available. There has been some prior work to improve emoji selection
interaction within mobile interfaces [
40
], although the research did not consider screen reader
users. More recently, Zhang et al. [
56
] explored emoji search for people with vision impairments,
Proc. ACM Hum.-Comput. Interact., Vol. 8, No. CSCW1, Article 175. Publication date: April 2024.
Party Face Congratulations! 175:5
but rather than rely on touch interaction, their solution utilised voice commands. Audio cues have
also been used to facilitate picking emoji [
38
] and to enhance the Facebook emoji react feature [
5
].
Tigwell et al. [
51
] did provide recommendations for using emoji in more accessible ways, such
as avoiding long repetitions of emoji (e.g., So funny! ” vs. “So funny! ), placing decorative
emoji at the end of sentences (e.g., It is today.” vs. “It is sunny today ”), and being explicit
about the intended meaning of the emoji in the text in case the emoji descriptor diers from how
the sender interprets its visual design (e.g., Oh? vs Oh, I’m intrigued ). However, Tigwell et
al.’s paper did not collect data from sighted users about their awareness of emoji accessibility issues
and they did not explore how best to provide sighted users with accessibility recommendations
when writing messages with emoji.
3 QUESTIONNAIRE - METHOD
To understand how to provide accessibility writing support to sighted users when using emoji, we
ran an online questionnaire, with people who did not self-identify as having a vision impairment.
An online questionnaire allowed us to collect a much larger dataset from a more diverse group of
participants than would have been likely with an in-person lab study on a university campus.
3.1 Materials
We divided our questionnaire into three parts. The initial section focused on general demographics,
including age, gender, and emoji usage. The second part involved a design evaluation task, where
participants were presented with example messages in three dierent designs (see Figure 1).
Participants were given the option to edit each message they read before sending. If participants
chose to edit, then they were asked to type out the edited message. Finally, we concluded the
questionnaire with closing questions that focused on participants’ preferences regarding the
accessibility support designs.
Fig. 1. Examples of the images shown within the design probe for each prompt for message 1_A_2:
A) No Support, B) Preview, and C) Alert. The screenshots are based on WhatsApp and use iOS emoji.
Proc. ACM Hum.-Comput. Interact., Vol. 8, No. CSCW1, Article 175. Publication date: April 2024.
175:6 Carla F. Griggio, Benjamin M. Gorman, & Garreth W. Tigwell
3.1.1 Accessibility Support Designs. We used Sketch (sketch.com) to recreate a WhatsApp interface
(whatsapp.com). We chose WhatsApp as the basis for our message platform because it is a widely
used application for communication across dierent contexts. WhatsApp has its own set of emoji
designs, which are used on both iOS and other platforms. However, since the WhatsApp designs
are based on iOS emoji, we decided to use iOS emoji in our message prompts for simplicity. We
designed three probes for our study: No Support, Preview, Alert support (details below)
1
. Within
each design, the current message thread would be blurred to draw attention to the message being
“typed” and to provide context that it was part of an ongoing message thread. We also made clear
that the message was being sent to Taylor, the persona in our scenario who uses a screen reader.
3.1.2 Design Rationale.
A)
No Support: This design provided no specic accessibility support and resembled the
default appearance of WhatsApp. We included this to understand sighted people’s initial
assumptions and attitudes towards editing messages when considering their conversational
partner is a screen reader user.
B)
Preview: This design provided a basic text transcript a screen reader would relay displayed
above the message entry box. We chose to represent the transcription in the simplest form,
without any additional edits by the screen reader software (e.g., some screen readers may
add the word emoji or simplify repeated emoji — three face with tears of joy). An example is
shown in Figure 1.B, with a message that has the “Face With Tears of Joy” emoji repeated
three times, resulting in the full message reading as “That’s great! Face with Tears of Joy
Face with Tears of Joy Face with Tears of Joy”.
Before being presented with the message prompts with this design, participants were pre-
sented the following: “The next seven questions will show a ‘Preview’ box above the message
draft for Taylor. This Preview box will display each message in the way that Taylor will hear
it from their screen reader. Similar to the previous questions, it is up to you to send each
message as it is or to edit it before sending it to Taylor.”
C)
Alert: The design highlighted potential accessibility challenges within the messages using
an alert box above the message entry box, following the guidelines outlined by Tigwell
et al. [
51
]. For instance, in Figure 1.C, since the Face with Tears of Joy is presented three
times, the alert shows “Alert emoji appears three times - emoji descriptors may be voiced
consecutively”.
Before being presented with the message prompts with this design, participants were pre-
sented with the following: “The next seven questions will show an ‘Alert’ box above the
message draft for Taylor. This Alert box will warn you about potential accessibility issues.
Similar to the previous questions, it is up to you to send each message as it is or to edit it
before sending it to Taylor.”
3.1.3 Emoji Message Prompt Sets. We collaboratively developed the messages for the prompts.
We started with three categories of issues caused by emoji in messages, which are based on the
guidelines introduced by Tigwell et al. [
51
]. Each of the categories have sub-issues, and we had
three messages for each sub-issue. This resulted in three sets of messages. Our nal categories
were:
1
We based our designs on how VoiceOver—a popular screen reader—spoke aloud at the time. Since completing our work,
VoiceOver has received an update to narrate emoji in a concatenated way, e.g., saying “three grinning face”, however, older
versions of VoiceOver still say “grinning face, grinning face, grinning face.”
Proc. ACM Hum.-Comput. Interact., Vol. 8, No. CSCW1, Article 175. Publication date: April 2024.
Party Face Congratulations! 175:7
Table 1. Our final prompt message sets organized by emoji sub-issue. The three categories of issues caused
by emoji are based on the guidelines introduced by Tigwell et al. [
51
]. The three categories are: 1) Number of
emoji, 2) Placement of emoji, and 3) Purpose of emoji.
Emoji sub-issue Prompt Set 1 Prompt Set 2 Prompt Set 3
1.a) Three duplicate emoji
I got an A in the exam
I don’t think I can do this
That’s great!
1.b) Two emoji pairs
Congratulations! School starts tomorrow
Looking good
1.c) Three related emoji I miss you too
Starting university next
week
Out of oce is on
2.a) Emoji within sentence What a sunny day!
It’s her birthday today I booked a dentist ap-
pointment
2.b) Emoji at beginning of
sentence What happened?
Congratulations on
your 10th anniversary!
I’d really appreciate
that
3.a) Potentially ambiguous
emoji It’s so cute I know right
How did last night go?
3.b) Unintended long emoji
descriptor Good luck
I can’t believe they did
that We’ll be there!
1) Number of emoji - The number of emoji within a message leads to long or repetitive spoken
output by a screen reader: a) Repeating one emoji, b) Repeating a sequence of emoji, and c) A
sequence of dierent but related emoji.
2) Placement of emoji - The placement of emoji within the message such as within a sentence
or where it disrupts a coherent voiced output by a screen reader: a) Within a sentence, and b) At
the beginning of the sentence.
3) Purpose of emoji - When the descriptor of the emoji may not match the intended purpose:
a) Ambiguous emoji that can be interpreted dierently between individuals or cultures, and b)
Unintended long emoji descriptors, especially in the cases of emoji with modiers (e.g., gender,
skin tone), where the base emoji is often simple (e.g., “Woman Facepalming” ), but users may
be unaware of the extra descriptors that will be voiced when customising the emoji or how those
additional details are placed within the full description (e.g., “Woman Facepalming: Light Skin
Tone” , where it may be more natural for someone to voice "light-skinned woman facepalming").
The content of the messages were then further inspired by some of the examples discussed by
Cramer et al. [
7
], Tigwell et al. [
51
], Weissman and Tanner [
53
], and Wiseman and Gould [
55
]. In
total, we had three message prompts for each sub-issue, and these are presented in Table 1.
3.2 Procedure
3.2.1 Design Evaluation Task. Participants were provided the following that explained the scenario:
“Imagine you’re chatting with Taylor, a friend of yours who is blind. Taylor uses a screen reader to
listen to a spoken version of the messages you exchange in your favourite messaging app. Next, we will
show you a set of messages with emoji and ask about whether and how you would edit them before
sending them to Taylor. You may or may not want to use emoji in your edited messages, this is totally
up to you. In case you do, and are currently using a computer, you can open an emoji menu with the
following commands while you’re typing on a text eld:
Mac: Control + Command + Space, Windows: Windows key + ; (semi-colon) or Windows key + . (period).
If you’re using a mobile device, you can access emoji from your mobile keyboard.”
Proc. ACM Hum.-Comput. Interact., Vol. 8, No. CSCW1, Article 175. Publication date: April 2024.
175:8 Carla F. Griggio, Benjamin M. Gorman, & Garreth W. Tigwell
As a form of quality control, we also asked participants to demonstrate that they were able to
type emoji on the device they were completing the survey from. The question showed an image of a
message that read “I can type emoji ” with the text: “Please re-type the following text (including
the thumbs up emoji) to make sure that you can type emoji:”.
Our evaluation task had participants evaluate 21 total message prompts. For each of the three
designs, each participant evaluated a set of seven message prompts. All participants started with
Prompt Set 1 (Table 1) with no support. Since both Preview and Alert could potentially teach
participants about accessibility problems, we counterbalanced their order across participants using
a Latin square. Here, we want to emphasise that the study is not an experiment but a qualitative
survey. However, the survey asks participants to describe how each design helped them write more
accessible messages as well as which one they preferred, so we presented them in a balanced order
across participants to minimise the potential impact of recency eects. To generate a more diverse
range of responses and opinions, we also alternate the combination of Preview and Alert with
Prompt Set 2 and Prompt Set 3. The prompts can be seen with their support format in Table 2, Table
3, and Table 4 where 1_A_1, refers to category 1, issue A and prompt set 1.
For each message prompt participants were shown the following message: “Imagine you have
drafted this message to your vision impaired friend, Taylor. Here is what it might look like on
your device”, and then a screenshot for that message/design was displayed as shown in Figure 1.
Participants were then asked: “Considering the recipient is your vision impaired friend, Taylor,
which of the following would you like to do?” and given the options to either "Send the message
as it is", “Edit the message before sending it” or could respond with “I’m not sure”. If participants
chose to edit, they were asked to re-write the message with the changes that they thought were
necessary to make it more accessible to a friend with a vision impairment.
We chose not to ask participants why they edited a message as we anticipated that probing
into the reasons for their edits might be perceived as an extra burden by participants, potentially
considerably extending the duration of the survey and aecting the overall quality of their responses.
Our primary interest was in understanding participant strategies employed when attempting to
make the message more accessible to their friend with a vision impairment rather than going into
the specic reasons behind employing those strategies.
The nal message sets and text are provided in Appendix A (Tables 2, 3, and 4).
3.2.2 Closing estions. On completing all message prompts, participants were asked to rank
each design option in order of preference that they felt informed them the most about the potential
issues that Taylor may experience when receiving the messages. Participants were then asked to
list specic aspects of what helped and what didn’t help about each of the design options.
We then captured closing demographics relating to prior experience with accessibility, assistive
technologies, and whether they had messaged someone before who had a signicant vision im-
pairment (i.e., low vision, blind). Finally, we asked participants if they consciously adapted their
messaging approach based on their knowledge of that person’s vision impairments and were asked
to explain how with some examples. We included the questions regarding accessibility knowledge at
the end to minimise the eect of participants who might be tempted to do more reading about screen
readers before completing the design evaluation task (i.e., we wanted to capture their approach to
the study with their current knowledge).
3.3 Participants
Ethical approval was obtained from our IRB. We distributed the survey through social media
(e.g., Facebook, Instagram, Twitter), Reddit (r/SampleSize), and university mailing lists. Admin
Proc. ACM Hum.-Comput. Interact., Vol. 8, No. CSCW1, Article 175. Publication date: April 2024.
Party Face Congratulations! 175:9
permission was sought in all cases where we were outside of a group space. The survey was open
from 07/2022 and our last response was collected on 30/01/2023.
In total, 200 participants completed our online questionnaire. Eighty-four participants were
removed from our analysis—reasons for removal were: 32 participants consented but then did not
continue, 11 participants did not demonstrate they could enter emoji, and a nal 41 participants
were excluded because they responded to a few message prompts and then quit.
We had 116 participants remaining. Participants completed the online questionnaire on a variety
of desktop and mobile devices. All participants reported using emoji within the week before taking
part in our questionnaire. None of our participants had a signicant vision impairment.
3.3.1 Age & Gender. Our participants were aged between 18-60 years-old (Mode=24; Mean=30.46;
SD=8.80). Our participants’ gender distribution
2
was: Man (97 participants), Woman (59), Non-
binary (15), Prefer not to disclose (1), Prefer to self describe (4), and No response (5). Text responses
included: Me (1), Genderuid (1), Gender nonconforming (1), Genderqueer (1).
3.3.2 Platform Usage. We asked participants to indicate which messaging apps they use (multiple
responses were allowed): Whatsapp (67 participants), Facebook Messenger (65), Instagram (DMs)
(59), iMessage (iOS SMS) (45), Discord (42), Messages (Android SMS) (41), Slack (35), Signal (20),
Telegram (18), Snapchat (17), Line (11), Twitch (9), TikTok (DMs) (9), and WeChat (2). Twenty-six
participants (22.41%) also indicated other responses, which included email clients, mobile apps,
video conferencing/communication platforms, and social websites.
3.3.3 Geographic Range. Our participants resided in the following countries: USA (37 participants),
UK (25), Japan (9), France (6), Switzerland (5), Canada (4), Australia (3), Germany (3), Czech Republic
(2), Finland (2), Spain (2), Argentina (1), Denmark (1), Hungary (1), Ireland (1), Israel (1), Mexico (1),
Netherlands (1), New Zealand (1), Portugal (1), Russian Federation (1), South Korea (1), and Turkey
(1). Six participants did not respond.
3.3.4 Educational Experience. The educational experience of our participants was varied. Responses
for the highest completed education were: University (Graduate or Postgraduate) (56 participants),
University (Undergraduate) (31), High school or equivalent (13), Community college or preparatory
school (pre-university) (8), Apprenticeship (3), Associate’s degree (1), Professional degree (1). Three
participants provided other responses that included: Some college (P35), Currently enrolled in
undergraduate (P61), PhD (P85).
We asked participants to indicate if their education included any technical subjects because we
wanted to gauge whether they might have learned about accessibility. All participants responded
to the question and 70 participants (60.34%) indicated yes. Responses for this were categories as
follows: Computer Science (52), Engineering & Math (12), and Other (14).
Please note there are participants that were grouped under two categories based on their re-
sponses. Furthermore, some of the responses refer to full degrees, while other times, responses
indicate a small part of a degree or class within a course. Only one person explicitly mentioned
accessibility, while other people listed programs/courses that might cover accessibility (e.g., HCI).
We asked our participants some further demographic questions after they completed the study:
3.3.5 Accessibility & Assistive Technology Experience. Although many participants indicated that
their educational background was not related to accessibility, we had 94 participants (81.03%) who
indicated that they had prior experience with accessibility, while 18 participants (15.52%) had no
prior knowledge of accessibility, and four (3.45%) did not respond to the question. Participants could
2We allowed multiple responses and followed the recommendations of Scheuerman et al. [45].
Proc. ACM Hum.-Comput. Interact., Vol. 8, No. CSCW1, Article 175. Publication date: April 2024.
175:10 Carla F. Griggio, Benjamin M. Gorman, & Garreth W. Tigwell
provide information in an open text eld to explain more, and 81 participants chose to provide
details. We looked at whether the participants provided specic details on their prior experience
that would be most relevant to our study (e.g., indications of knowledge on emoji accessibility
issues, knowledge about screen readers, and learnt about access needs by communicating with
people with vision impairments). Nineteen participants shared knowledge and/or experiences
relevant to our study, 54 participants were less specic, and we cannot tell how much relevant
knowledge and/or experiences they had (e.g., P4 said “My job is in digital accessibility”), and eight
participants made it clear they had prior experience with accessibility but in a dierent subdomain
(e.g., P113 said “I knew about physical accessibility”).
We also asked participants the assistive technologies and features they were aware of (multiple
responses allowed), and found: Text-to-speech (107 participants), Hearing aid/devices (105), Screen
readers (102), Closed captioning (98), Magniers (94), High contrast mode (90), Braille input devices
(69), Reduced motion blur (44), and Switch access (14). Four participants provided no response.
Fourteen participants included other responses: Voice control (3 participants), Speech recognition
software (2), Braille output device (1), Colour blind lter (1), Home button (1), One handed mode
(1), Question unclear but knew of them (1), Reduced animation mode (P31), and one tags (1).
Finally, we asked our participants if they had messaged someone with a signicant vision
impairment. There were 29 participants (25.00%) who had messaged someone with a signicant
vision impairment, while the majority of our participants responded with no (62 participants;
53.45%), 21 participants were unsure, and four participants did not respond. Out of the 29 participants
who had messaged someone with a signicant vision impairment, 24 participants provided further
information. We found that 19 participants would adapt their messaging style, while two participants
did not, and three participants mentioned the person with a vision impairment they spoke to did
not require them to adapt their messaging style. The adaptions reported included aiming for clear
messages that were shorter and could be broken into multiple messages. There was a tendency to
reduce or remove visual media such as emoji and GIFs, avoid poor emoji placement or multiple use
of emoji, and describe images if they are going to be used or avoid images. Some participants would
aim for more punctuation (e.g., P60 said “yes, I have never used emojis when messaging the visually
impaired student I work with. I use correct punctuation more often in case it aects the screen reader’s
tone.”) or using ASCII emoticons in place of emoji, and to avoid creative messaging practice (e.g.,
P76 said “At the time keysmashes (i.e. hsgjskdjdndhdj) were very popular, and that person specically
said that they didn’t want people to do that a lot because of their screenreader. Mostly we just followed
whatever they requested.”).
3.4 Analysis
Positionality and epistemological stance. We are three sighted researchers. Two of us have ex-
perience conducting research at the intersection between HCI and Accessibility, while the other
specialises in HCI and Social Computing. We designed this study with a qualitative mindset and
conducted the analysis with a constructivist perspective, acknowledging that the reported results
are the product of our interpretation and that the knowledge we contribute is a construction, not
a discovery [
12
]. This is reected in our analysis process in a number of ways: First, multiple
coders annotated the data set to enrich the analysis with diverse viewpoints, and not as a means to
increase the reliability of our interpretations, since we acknowledge the subjectivity that we bring
into the research without considering it a ‘bias’ that needs to be controlled [
6
]. Second, we treated
coding disagreements as an opportunity to enrich the interpretation of a response, generating
notes about surprising insights, reconsidering the scope of a code or creating new ones; we did not
keep track of our consensus and did not report an IRR since these practices do not align with the
epistemological stance in which we positioned ourselves [
32
]. Third, we do not report counts or
Proc. ACM Hum.-Comput. Interact., Vol. 8, No. CSCW1, Article 175. Publication date: April 2024.
Party Face Congratulations! 175:11
quantitative measures since we designed and deployed the survey with the purpose of generating
diverse responses (e.g., from participants with dierent cultural and socio-economic backgrounds,
in response to dierent prompts) rather than measuring or quantifying behaviour. In line with
this, our analysis approach prioritised the identication of a range of insightful patterns in the
ways that sighted users attempted to make messages with emoji more accessible rather than the
exhaustive computing of frequencies for these patterns. For example, for each edited message, we
assigned up to two codes describing how participants were attempting to make the message more
accessible, prioritising the most representative codes and discarding less salient ones.
Data cleaning. We excluded 19 entries where participants described what they would change
instead of rewriting the message (e.g., “I think I’d reduce the emoji to a single one, otherwise it’s going
to be annoying for them”, P32); 46 entries where participants indicated that they would correct the
message but left the correction eld blank; six entries where the rewritten message was exactly the
same as the prompt; and one entry where the participant explained that they did not understand
the meaning of the prompt message ("Out of oce is on ").
Coding process. We conducted a qualitative, open-coding analysis on all valid rewritten messages
(1508). The codes described changes in the syntax and typographical features of the rewritten
messages compared to the original prompts without considering the potential intention behind
such changes. For example, codes included “Remove duplicates”, “Remove all emoji”, “Move emoji
to the end”, “Replace emoji with punctuation marks”, and “Change order of emoji”.
One author initially coded the rst two-thirds of the data set and developed a set of initial codes.
The other two authors then utilised these codes to analyse the remaining two-thirds of the data
set individually, ensuring that each entry was coded by two authors. During the coding process,
we also marked surprising or complex responses as “interesting” to later discuss with the entire
team. We then conducted a series of meetings where we revisited the data set together, using
code disagreements and "interesting" responses as shortcuts to discussing the most surprising
patterns in the data, the denition and scope of each code, the creation of new codes, mistakes and
contradictions across responses, and initial thoughts on implications for design.
During these discussions, we wrote down analysis notes on a shared document, which we used to
inform and structure the main results of the study. One author grouped codes into larger categories
of rewriting strategies for making a message more accessible to a screen reader user, which the
team discussed and agreed on in the last meeting of this series. All notes about mistakes and
contradictions across responses (e.g., cases where the rewritten message was less accessible to a
screen reader user) were also grouped into three larger categories and discussed in a meeting.
Last, one author read all free-text responses to the question, “Please list specic aspects about
what helped and what didn’t help about each of the three designs”, noting surprising or insightful
opinions with accompanying quotes about what was perceived as helpful or confusing about
each probe. These notes were discussed by the team, reecting on how participants’ explanations
for their preferences, in addition to the rewriting strategies and misunderstandings about screen
readers, pointed to implications for the design of tools for helping sighted users compose more
accessible messages in conversations with a screen reader user.
4 RESULTS
We present the results of our analysis based on three main aspects: 1) Strategies to make messages
more accessible, 2) Misunderstandings and incorrect assumptions about screen readers, and 3)
Preview vs. Alert: what aspects of the designs were perceived as helpful to make messages more
accessible?
Proc. ACM Hum.-Comput. Interact., Vol. 8, No. CSCW1, Article 175. Publication date: April 2024.
175:12 Carla F. Griggio, Benjamin M. Gorman, & Garreth W. Tigwell
Through detailed exploration using participant examples and quotes, we provide a comprehensive
understanding of the strategies employed, misconceptions held, and preferences expressed by
participants. We provide examples of dierent types of rewritten messages that our participants
submitted, without emoji (Appendix B) and with emoji (Appendix C). We present the rewritten
messages, and the participant quotes verbatim from the typed responses to the survey.
4.1 Sighted users’ strategies for aempting to make messages more accessible
Participants applied diverse strategies attempting to enhance the accessibility of the given messages,
which uncover patterns in their assumptions about how screen readers interpret emoji. We grouped
these strategies into two overarching categories: those that avoid the use of emoji completely and
those that attempt to use emoji in ways that may be more compatible with a screen reader.
4.1.1 Avoiding the use of emoji. Some of the most common strategies relied on rewriting
messages so that they would not include any emoji. Although we lack data about the reasoning
behind each message rewrite, we speculate that these strategies may feel more reliable, where
sighted users take control over the words that the screen reader will pronounce. These strategies
may also imply a belief that sending emoji to a screen reader user is problematic or meaningless.
We identied four ways in which participants avoided the use of emoji:
Removing all emoji. The most common pattern among rewritten messages was to leave the
original message as it was and simply remove all emoji (see Appendix B, Table 5 for examples). We
speculate that this strategy was often adopted as a quick x, however, it sacrices the non-verbal
aspect of the message conveyed by the emoji.
Rethinking the message from scratch. This strategy suggests an intention to convey the prompt’s
meaning through text alone, compensating for the non-verbal aspect of the removed emoji in
writing (see Appendix B, Table 6 for examples).
In the case of decorative emoji, which are often used to “produce a larger eect on the recip-
ient” [
7
], some participants “translated” them into phrasings that emphasised an aspect of the
message. P57 rewrote “What a sunny day! ” into “What a beautiful sunny day!”, emphasising
a positive tone. In the case of the prompt “We’ll be there! ”, P76 changed it to “We’ll both be
there!” and P51 to “We’re coming for sure”, translating the redundancy of the decorative emoji into
a phrasing that conveys extra certainty.
In the case of emoji that conveyed emotions [
7
], the rewritten messages often included ono-
matopoeia or explicit descriptions of the emotion. For example, for “It’s so cute ”, P39 wrote “Its
so cute im going to cry! Squee”, and P70 “Awww so cute”. Another resource was to use special word
spellings to manipulate the “attitude” or tone of the sentence (which we revisit in Section 4.2.2).
For example, P111 rewrote “I know right ” as “I know righttt”, and P112 changed “I got an A in
the exam ” into “So, guess what... Ya boy aced the exam”, expressing some anticipation and
a show-o attitude.
Some participants completely changed the message into what would seem a more personal way
of expressing the same idea. For example, P51 changed “Congratulations! ” into a short
and simple “woot!”, and P8 into “Congratz”.
Replacing the emoji with additional text. A large proportion of participants replaced the emoji in
the prompt for text (e.g., an extra phrase, punctuation marks, and other textual markers), keeping
the main content of the message the same as if they assumed the role of an “emoji translator” for
the screen reader (see Appendix B, Table 7 for examples). Many replacements were extra phrases
describing the feelings and actions evoked by an emoji (e.g., P89 replaced for “Keeping my
ngers crossed for you”) or verbalising emotional reactions (e.g., P63 replaced with “Oh no!”).
Proc. ACM Hum.-Comput. Interact., Vol. 8, No. CSCW1, Article 175. Publication date: April 2024.
Party Face Congratulations! 175:13
Others seemed to more explicitly replace an emoji for a descriptor, typically indicating it as a
separate part of the message in the form of a parenthesis or label, such as with dierent variations
of brackets, the hash symbol, asterisks or dashes. Some descriptors featured wordings close to
what a screen reader would voice, e.g., P87’s “#birthdaycake” for or P26’s “(facepalm)” for .
However, descriptors were rarely the ocial ones; participants came up with their own. For example,
P5 changed “I know right ” into “I know right? (Kinky emoji)”, and P53 into “I know right!
<sassy hair ip emoji>”. An interesting quality of these self-made descriptors is that they focus on
describing the participants’ interpretation of the emoji’s intent (e.g., a “sassy” tone), or even a new,
preferred intent (e.g., a “kinky” tone) rather than a strict description of how they look like.
A similar approach was translating emoji into actions of a similar tone, expressing them in the
form of self-described “autonomous staged directions” [
59
]. When doing this, participants switched
from their rst-person perspective as message senders to describe their actions in third-person,
bounded by asterisks. For example, P56 rewrote “I got an A in the exam ” as “I got an A in
the exam *drops mic*”, inviting the receiver to imagine them dropping a mic, a popular gesture
signaling that one has just said or done something particularly impressive. P49 replaced
for “*throws confetti*”. These stage directions are popular in Internet culture as text-based, non-
verbal cues to describe the situational context of the message. While these stage directions typically
appear within bounding asterisks, some messaging platforms oer dedicated functionality for
third-person action descriptions (e.g., IRC platforms and Discord oer users a “\me” command that
they can type before a message to italicise it and indicate it as an autonomous stage direction). P26’s
rewriting of “I know right ” as “I know right /sass” reminds us of such commands, indicating
/sass as an invitation to imagine the sender saying the message with a “sassy” intonation.
Another way of conveying the tone expressed by emoji with just text was to use punctuation
marks instead. Some participants used repeated question marks and exclamation marks as a resource
to emphasise the surprise, worry, excitement or curiosity intended in a message. For example,
P53 rewrote “ What happened?” as “What happened????”, and P82 changed “Congratulations!
” into “Congratulations!!!”.
Last, a few participants replaced the emoji for their “ASCII emoticon” equivalent, e.g., P87 replaced
with “:-O”, and P79 replaced with “:D”. Here, we nd it interesting that participants may
assume that just because these expressions are composed of text characters they are screen-reader
friendly. Unfortunately, this greatly depends on the screen reader settings of the receiver, and in
most cases it may fail to convey what the sender means.
Most of these examples show very interesting attempts at compensating the non-verbal infor-
mation of the emoji in text, but unfortunately, using punctuation marks to indicate intonation or
to denote a parenthesis, a staged direction, or an emoticon is generally not suitable for making
messages more accessible to a screen reader user. We analyse this in more detail in Section 4.2.2.
Replacing the emoji with alternative media. Only one participant used this strategy, but it
shows an interesting approach to conveying non-verbal cues in the modality of the receiver
(audio) rather than the sender (text) (see Appendix B, Table 8 for the example). P15 changed
“Congratulations! ” into “Congratulations! https://youtu.be/nYwAHURPQ-s”, and “
Congratulations on your 10th anniversary!” into “Congratulations on your 10th anniversary!
https://youtu.be/nYwAHURPQ-s”, pointing to a 1-second video that plays the sound of a party
horn. When encountering a URL, A screen reader would typically say “link” and then vocalise the
URL, it might read out the title of the link but it depends on how it was inserted into the message.
4.1.2 Adapting the use of emoji. These are strategies where participants chose to either keep
the original emoji in the message or include others, and modify other aspects of the message to
Proc. ACM Hum.-Comput. Interact., Vol. 8, No. CSCW1, Article 175. Publication date: April 2024.
175:14 Carla F. Griggio, Benjamin M. Gorman, & Garreth W. Tigwell
make them more accessible to a screen reader user (e.g., changing their placement in the message,
changing the text so that the emoji descriptor ts better).
Shortening the overall length of the voiced message. Most of the examples in this strategy preserved
the verbal content of the message and tried to reduce the words dedicated to emoji descriptors (see
Appendix C, Table 9 for examples). For example, a common pattern was to remove duplicate emoji:
P2 changed “Congratulations! ” into “Congratulations! ”, and P13 changed “I don’t
think I can do this ” for “I don’t think I can do this ”. While repeating emoji usually
emphasises an emotion, when considering a screen reader user as the receiver, these participants
chose to avoid that redundancy.
Some also chose to reduce the number of emoji to one. In the cases of prompts that had dierent
emoji in a row, we speculate that participants chose the emoji that best represented the emotions
or situational meaning conveyed by the set. For example, P30 rewrote “School starts tomorrow
” as “School starts tomorrow ”, and P3 rewrote “Starting university next week
” as “Starting university next week ”. When reducing the number of emoji, some chose
dierent emoji from the ones in the prompt message as a personal approximation to the meaning
that the whole sent expressed (e.g., P22 changed “I miss you too ” into “I miss you too ”).
Overall, this strategy illustrates an eort to preserve the expressive quality of emoji as non-verbal
cues without overwhelming the screen reader user with voiced descriptors. Besides reducing the
number of emoji, some added text (similar to Section 4.1.1) to reinforce the meaning of the emoji in
their message. For example, P5 rewrote “I got an A in the exam ” as “I got an A in the exam
how cool is that?!”. As an opposite example of adding text, P74 decided to rewrite “I can’t believe
they did that ” by removing all text and simply responding a emoji. Last, others changed an
emoji with a skin tone modication to the same emoji in its default, yellow version, resulting in a
shorter descriptor.
Clarifying the meaning or tone of emoji. This strategy represents cases where participants tried
to “help the screen reader user” by clarifying the intended meaning of the emoji with text (see
Appendix C, Table 10 for examples). For example, a few participants added their own descriptors
next to the emoji: P18 rewrote “Congratulations! ” as “Congratulations! (claps
& streamers)”, and P19 changed “I got an A in the exam ” into “I got an A in the exam
[person with sunglasses emoji]”. The clarifying text might suggest that the participants did not
know that emoji are voiced by screen readers, or perhaps they wanted to make sure that their
intended meaning was explicit regardless of how the emoji was pronounced. In either case, the
screen reader will voice both the emoji and the clarifying parenthesis, potentially making the
message even longer, redundant, or complex.
Others added punctuation marks, likely in an attempt to clarify the meaning of the message by
manipulating its intonation (see Section 4.2.2), or added text preceding the emoji, expecting the
screen reader to “complete the sentence”: P110 appropriated “I got an A in the exam ” into
“I got an A in the exam, I’m feeling like a ”.
Last, as a dierent approach to clarifying the intended meaning of an emoji, some participants
chose to swap the prompted emoji for another one with a less ambiguous descriptor. For example,
in the case of an ambiguous emoji such as in “I know right ”, P18 changed it for , P22 for ,
and P116 for , where each adopted their own interpretation of “the right” message to send.
Isolating the emoji to distinguish it from the main message content. For message prompts with
emoji in the beginning or middle of a sentence, many participants moved emoji to the end, in line
with prior recommendations [
51
] (see Appendix C, Table 11 for examples). By repositioning the
emoji, participants prioritised a clear understanding of the main message, which also contextualises
Proc. ACM Hum.-Comput. Interact., Vol. 8, No. CSCW1, Article 175. Publication date: April 2024.
Party Face Congratulations! 175:15
the meaning of the emoji. Three participants thought of more explicit ways of marking a separation
between the verbal part of the message and the emoji: P13 proposed to split “ What happened?”
into two messages: one for , and the second for the text. And P11 and P19 explicitly “announced”
the emoji at the end of the sentence, e.g., “I can’t believe they did that [Emoji] ”.
4.2 Misunderstandings and incorrect assumptions about screen readers
Participants’ rewritten messages often introduced new accessibility issues, signaling misunderstand-
ings and incorrect assumptions about how screen readers work. Two key issues were identied:
participants lacked knowledge about accurate emoji descriptions, and used textual markers that
were incompatible with screen reader functionality.
4.2.1 Unknown emoji descriptors.
Selecting similar-looking emoji. Some rewritten messages suggested accidental selections of
dierent emoji from the ones in the original prompt. For example, P2 rewrote "What a sunny
day! ” as "What a sunny day! ", replacing the sun emoji with the “sun with face” emoji . We
cannot be certain about whether these dierences were deliberate or not, but they helped us reect
on scenarios where a sighted user may not consider or notice a subtle dierence in the design
of two emoji, unaware that their descriptors may convey dierent emotions when voiced. For
example, in the case of the prompt "School starts tomorrow ", P18 rewrote it as "School
starts tomorrow ”. The emoji look very similar, where and both have closed eyes and a
downward-curved mouth, and and both have blue foreheads, though the rst one also has
downward-curved eyebrows and sweat. The original message uses emoji to convey a feeling of
disappointment and anxiety, while the rewritten one conveys distraught and fear.
We believe it is important to support the deliberate and conscious selection of emoji and to avoid
situations of accidentally choosing a visually similar emoji, since voiced descriptors may leave
less room for interpretation than the visual representation of the emoji (e.g., "Fearful face with
blue forehead"). Similar examples include P7’s "School starts tomorrow ", swapping for the
crying emoji , and P16’s "School starts tomorrow ", swapping for the “sad pensive” emoji
and for the “downcast face with sweat” emoji .
Replacing emoji for others with longer descriptions. In Section 4.1.2, we illustrate how some
participants changed the emoji in the prompt to clarify the intent of the message (according to
their interpretation). We observed a few examples where, in doing so, they selected emoji with
very long descriptors. For example, P50 changed “I know right ” to “I know right ” using
an emoji that VoiceOver, for example, reads as “Nail Polish Being Applied To Fingers with Light
Skin Tone”. Most of these cases were related to the use of skin tone modiers. For example, P36
changed “Congratulations! ” to “Congratulations! ” (“Clapping Hands With Light
Skin Tone”), and P26 changed “ Looking good ” to “Looking good ” (“Ok Hand With
Light Skin Tone”). Emoji are often used as a way of expressing identity [
7
,
18
,
49
], however, it can
feel overly repetitive to voice a skin tone modier for every emoji that has one. We speculate that
some participants had customised their emoji keyboard to apply a particular skin tone modier to
all emoji that aorded it, or that they chose some of the emoji from their recently used list. While
it is up to each user to decide whether this is an issue for them or not, we believe it is important
that sighted users are aware of the extra words added by modiers (e.g., gender, skin tone) so they
can choose to do this deliberately rather than by accident.
4.2.2 Textual markers that lack screen reader support. Some strategies replaced emoji for
textual markers to convey the tone or emotional expression of the original message, such as
exclamation marks (e.g., changing “It’s so cute ” to “It’s so cute!” (P29)) or capitalised words
Proc. ACM Hum.-Comput. Interact., Vol. 8, No. CSCW1, Article 175. Publication date: April 2024.
175:16 Carla F. Griggio, Benjamin M. Gorman, & Garreth W. Tigwell
(e.g., “TEARS OF JOY!” (P51)). As described in Section 4.1.1, we believe that these strategies aim at
translating visual, non-verbal cues (i.e., the emoji) to purely verbal expressions. However, some
examples suggest that participants were often unaware that they used text in visual, non-verbal
ways that screen readers either ignore or misinterpret. For example, while sighted users may
interpret the all-caps and exclamation mark in “TEARS OF JOY!” (P51) as a scream of excitement,
VoiceOver voices it in a plain way, the same as “tears of joy” in lower case with no exclamation
mark. Other misunderstandings regarding how textual markers are (or are not) voiced included:
Repeating question and exclamation marks. Screen readers typically have a at intonation. For
example, VoiceOver uses the same type of falling intonation to pronounce “I know right.” and “I
know right!”, and adds an extremely subtle rising intonation for “I know right?”. However, many
participants used exaggerated punctuation as a way of indicating an exaggerated intonation in the
text, which usually makes no dierence to the screen reader. If a sighted user means to convey
non-verbal information in a message about its tone, including an emoji, e.g., “I know right ”
would be a more accessible approach to repeating punctuation marks, e.g., “I know right??” (P64).
Using ASCII emoticons. A few participants replaced emoji for ASCII emoticons, however, not all
screen readers interpret emoticons as such, or may depend on customisations to indicate how to
pronounce them. In general, punctuation marks are ignored. For example, the default conguration
of VoiceOver would read the happy emoticon “:D” as the name of the character “D” (“dee”), and the
smile emoticon “:)” may not be pronounced at all.
Elongating words. Some participants attempted to convey the intended intonation of a message by
“stretching” a word, e.g., typing “noooo” instead of “no” (P77) for a dramatic eect. However, screen
readers do not interpret word elongations as a tone indication. In the case of “noooo”, VoiceOver
pronounces it as “noo” (as in “noon“). Similarly, P112’s example “Soooo... how did last night goooo?”
is voiced as “Soo [pause] how did last night goo?”.
Separating the main text from text representing non-verbal cues with punctuation. Many rewritings
were based on replacing emoji for a descriptor invented by the participant, e.g., “It’s so cute (googly
eyes)” (P51), or for an autonomous stage direction describing an action performed by the sender, e.g.,
“I got an A in the exam *drops mic*” (P56). We nd it interesting that the text replacing the emoji is
typically separated from the rest by punctuation marks such as dierent types of brackets, asterisks,
etc., to mark a dierence between “the main message” and its “non-verbal complement”. However,
these markers will most likely not be pronounced by the screen reader or pronounced quite literally,
which may not match the eect the sender intends. For example, VoiceOver pronounces “I can’t
believe they did that (facepalm)” (P26) as “I can’t believe they did that facepalm”, and in the case of
“I got an A in the exam *drops mic*”, it pronounces “I got an A in the exam star drops mic star”.
4.3 Preview vs. Alert: what aspects of the designs were perceived as helpful to make
messages more accessible?
When asked to rank each design option in order of preference, most participants ranked the
Preview design rst (88 participants). Only 12 participants ranked the Alert design rst, 11
preferred having no support, and ve participants did not respond. Next, we provide some insights
into those preferences.
4.3.1 Seeing the emoji descriptors. The design aspect that helped participants the most was being
able to read the emoji descriptors. Preview always displays the descriptors, but Alert only did
it for selected prompts. We had expected participants to have some degree of familiarity with
emoji descriptors, for example, from emoji shortcodes [
11
] in apps like Discord, Teams or Slack
Proc. ACM Hum.-Comput. Interact., Vol. 8, No. CSCW1, Article 175. Publication date: April 2024.
Party Face Congratulations! 175:17
(e.g., “:heart:” or “(heart)” for ) or searching with keyboards in an emoji menu. However, many
seemed to “have no idea what the emojis are actually ‘named”’ (P91). P92 said: “ For the rst and
second set of questions, I had no idea how emojis were described. So only used fairly simply happy /
sad expressions so as not to confuse Taylor.” One participant even thought that the emoji descriptors
shown in each prompt were our own invention, disagreeing with the meanings we had chosen:
“The preview was the most literal, but most of them were wrong in identifying the meaning of the emoji.
You should put more eorts in being accurate regarding the most common and average meaning.”(P54).
We discuss more examples of this in Section 4.2.1. Especially when prompted with a message using
Preview, most participants felt that they knew “exactly” how it would be voiced by the screen
reader. Participants often explained the advantage of Preview as a tool that let them know “exactly
what it will sound like” (P84) or “exactly how Taylor will hear it” (P107), “which made spotting
potential issues very straightforward” (P30). P11 explained: “In preview, one can really see the impact
of the way we type and insert emojis. It’s a sort of WYSIWYG [What You See Is What You Get] mode.”
We believe that seeing the emoji descriptors in context, visualising how they modied the entire
message without having to imagine it, is the main factor that made Preview the favourite choice.
However, we want to emphasise that the perceived accuracy in the translation of a message with
emoji into “screen reader language” is not really as it seems, and may not help sighted users to
realise that the voice they hear when reading the Preview message is simply how they read it “in
their heads”, leading to mistakes as those discussed in Section 4.2.2. For example: “The third was
very helpful reading exactly as Taylor would hear the message, this encouraged me to avoid emojis all
together and use punctuation instead” (P92).
Last, participants with experience in emoji accessibility and screen readers also appreciated
that they could read the descriptors, noting that they sometimes forget or ignore the descriptors
of particular emoji. Even experts may miss the dierence between an emoji with a skin tone or
gender modier and its default version. For example, P30 said: “I forgot that some screenreaders will
take an exceptionally long time to read out emoji with skin tone modiers until I got reminded by the
preview method later on.” Moreover, even though experts would know how to “test” their message
before sending it, seeing the descriptors in the context of the typed message can promote a more
immediate messaging experience:
“For the most part I understand how screen readers work and the need to limit emojis for
access, so the most helpful thing for me is to see how each one will actually be read out to
save me from needing to test it myself with the screen reader option on my phone or looking
it up. I don’t personally need advice on other stu like limiting unnecessary emojis, I just
need to be able to make sure the emoji name matches intended meaning (P14)”
4.3.2 Recognising potential problems by oneself instead of “being told what to do”. A strong pattern
among the participants that preferred Preview was that they felt Preview was letting them
“empathise” with the screen reader user and understand what would make a message more clear
or less “annoying”. For example, P51 explained: “ communicating the exact experience (rather than
having it described) seems a stronger way to build empathy”.
We nd it particularly interesting that some participants felt very strongly about having complete
control over their expression and their choice of whether a message needed to be improved for
accessibility, even though the task was to rewrite messages that were not originally their own:
“I liked the alerts less, because I feel like it took away from my judgement as to what I was actually
trying to communicate” (P53). P38 showed this by framing Alert and Preview as opposites in this
regard: “[Preview] also invites the user to decide for themselves if the message is accessible, whereas the
alert could be seen as telling the user what to do.” The accessibility problems that Alert indicated
were often perceived as a form of reprimand instead of a helpful pointer, which triggered defensive,
Proc. ACM Hum.-Comput. Interact., Vol. 8, No. CSCW1, Article 175. Publication date: April 2024.
175:18 Carla F. Griggio, Benjamin M. Gorman, & Garreth W. Tigwell
negative attitudes among some: “The alert has an annoying hectoring quality: I would avoid using
emoji, or avoid texting Taylor altogether” (P17).
“Alert is unclear and confrontational. It feels like you’re being told o. (...) I get bored reading
it and I feel detached and personally more likely not to change my message in deance. ‘Don’t
tell me what to do’ vibes.” (P112)
Ultimately, being able to judge accessibility problems by themselves and how to address them
points to a more subjective understanding of accessibility in the context of messaging with a
screen reader user. For example, P1 characterised Alert as "babying", as “as people who use a screen
reader can still understand the deeper meaning of emoji”. Other participants liked repurposing the
accessibility issues presented in Tigwell et al. [
51
] as opportunities for sending funny messages,
also suggesting more subjective perspectives on what is accessible and what is not:
“It also allows some creativity as sometimes infringing the rule can be done on purpose to
use emojis creatively to convey meaning. So I might choose to type 2 emojis consecutively to
create a fun voice eect.” (P11)
“I use emojis for satire, so I removed emoji uses that were serious. I kept some of them because I
found their text to speech versions even funnier than the regular emoji ones (eg the
one is already satirical and funny because nobody actually writes like this, but it is much
funnier when you hear a robot voice taking 15s to read it out).” (P24)
4.3.3 Considering a dierent perspective on what can be experienced as an accessibility issue. On the
other end of the participants who saw Alert as confrontational or hectoring, others highlighted
its value in warning about potential accessibility issues that seemed less trivial to them:
“The alert method is useful, too, but is maybe a bit too abstract. It may not be immediately
obvious whether a message is going to be truly bothersome to Taylor just from the description
alone. However, it did alert me to problems I didn’t consciously consider before, such as emoji
immediately at the start of the message being confusing.” (P30)
“The preview gave a special insight into what Taylor would be receiving that the alert did not
but the alert agged up problems I would have never considered even with the preview (i.e
the description that was too long) so they both have their benets.” (P69)
In line with taking a more subjective perspective on emoji accessibility, some reected on
potential individual dierences across messaging users. For example, P33 pointed out that “(Both
sighted and blind) people have dierent preferences based on the chat context, the alert might not
always be proper”, challenging the idea of having general guidelines about how to use emoji with
screen reader users. P53 went as far as imagining customisable designs where a sighted user and
blind friends could dene their own “alerts”:
“I also wonder though if people texting with this plugin would have the chance for their
blind friends to input their preferences for dierent emoji behaviors (in the form of a quick
questionnaire or something) and then when texting individuals, it could give you the heads
up like "remember that Anne hates emoji so like seriously stop it" or "Raj likes the impact of
emoji, re away in most cases, just keep them after the message because they’re confusing before
or in the middle of sentences"” (P53)
5 DISCUSSION AND IMPLICATIONS FOR DESIGN
Our work focused on understanding how to help sighted users consider the accessibility of emoji
when messaging a screen reader user, and we approached this by exploring support designs that
could promote the adoption of Tigwell et al.’s recommendations [51].
Proc. ACM Hum.-Comput. Interact., Vol. 8, No. CSCW1, Article 175. Publication date: April 2024.
Party Face Congratulations! 175:19
5.1 Summary of Findings
Our study addressed three research questions related to the awareness, adjustments, and as-
sumptions of sighted users regarding emoji accessibility for screen reader users. We found that
participants employed various strategies when rewriting messages, ranging from removing emoji
altogether to adapting their use by clarifying meaning or isolating emoji within the message. De-
spite some prior awareness of accessibility, participants held misconceptions about screen readers,
as evidenced by their use of ineective descriptors and use of unsupported textual markers.
Furthermore, our ndings demonstrate a preference for Preview over Alert, as it allowed
participants to independently assess the accessibility of each message. Our ndings provide insights
into the importance of raising awareness among sighted users, promoting adjustments in messaging
practices, and dispelling misconceptions to enhance emoji accessibility for screen reader users.
5.2 Sighted Users’ Messaging Practices
Tigwell et al.’s guidelines [
51
] were grounded in survey and interview data with a focus on screen
reader users and proposed that people who use emoji should be mindful of the number of emoji in a
row, the placement of emoji in sentences, and that emoji descriptors may not match sender’s intent.
Our analysis of how sighted users try to make messages with emoji more accessible extends and
challenges these recommendations (as well as similar resources such as [
28
,
47
]) in the context of
messaging, providing insight into how to design tools that can enhance sighted users’ understanding
of how their messages are received by a screen reader user.
First, some of the rewriting strategies were aligned with Tigwell et al.’s recommendations
especially those focused on shortening the overall voiced message, such as removing duplicates,
reducing the number of emoji, and isolating the emoji from the rest of the message so that they
are voiced at the end. Replacing an emoji for another one in cases of messages with ambiguous
emoji also aligns with such recommendations. However, most participants explained that without
knowing the emoji descriptors it is challenging to understand whether a message has accessibility
issues or how it should be adjusted to be compatible with a screen reader. This was the case even
for expert users who know about how emoji can interfere with screen readers. Based on these
results, we argue that sighted users could greatly benet from situated and interactive support
for writing messages that are compatible with screen users, and that it is crucial to uncover emoji
descriptors for promoting the adoption of any guidelines for authoring content (e.g., writing a
message) with non-verbal cues intended for screen reader users.
Second, we found that the most common approach to improving the accessibility of messages
with emoji was simply to remove the emoji entirely. This strategy suggests a prevailing perception
that sending emoji to a screen reader user might be unhelpful. Unfortunately, the result is a message
that lacks any form of non-verbal communication, and we do not want to encourage the avoidance
of using emoji. In some cases, participants chose to omit emoji from their messages but cleverly
rephrased the content to retain the emotional, situational, or emphasising function that the emoji
was conveying, but, in doing so, this approach led to the use of other non-verbal, text-based
expressions that could be considered inaccessible. These included elongating words, using all caps,
repeating questions and exclamation marks, and denoting emotions or actions with parenthesis
or bounding asterisks. These insights suggest that messaging recommendations for sighted users
should be careful not to discourage the use of emoji, while also oering guidance on alternative
non-verbal means of communication. It is important for sighted users to be aware that even though
these alternatives rely on text, they are still visual in nature and, most often, incompatible with
screen readers.
Proc. ACM Hum.-Comput. Interact., Vol. 8, No. CSCW1, Article 175. Publication date: April 2024.
175:20 Carla F. Griggio, Benjamin M. Gorman, & Garreth W. Tigwell
Third, our results illustrate a subjective view on whether and how a message is more accessible to
a screen reader user. This challenges the notion of promoting guidelines as generalised principles to
follow, since the concept of accessibility may be informed by individual perspectives. For example,
some of the emoji accessibility issues identied by Tigwell et al., such as repeating the same emoji
multiple times, can be appropriated as a source of humour or teasing among users. This indicates
that what may be considered inaccessible in one context could be perceived as a dedicated joke in
another. Moreover, even when provided with explicit warnings about accessibility, sighted users
may still “disagree” with them. Given the diverse strategies we found for making a message with
emoji more accessible without losing its tone, situational meaning, or emotional connotations,
we believe that this subjective perspective on accessibility may stem from prioritising personal
expression over adhering to generalised guidelines. Many participants expressed a preference for
the Preview approach over Alert, especially because Preview’s proactive nature allowed them
to assess accessibility issues on their own. In some cases, this assessment was based on previous
interactions with blind friends. The exibility of Preview avoided imposing restrictions on personal
expression. In contrast, Alert was seen as constraining, prescribing issues out of context and
potentially making the sender feel limited in expressing themselves freely. This calls for adaptable
approaches that consider individual perspectives when guiding sighted users about accessibility in
messaging practices.
5.3 Implications for Supporting Messaging Accessibility
These insights point to implications for the design of tools to support sighted people when messaging
screen reader users. First and foremost, emoji descriptors should be visible during emoji selection.
For instance, descriptors could be displayed in emoji menus or as part of keyboard autocomplete
suggestions. This approach would assist sighted users while composing messages, enabling them to
ensure that their intended message is accurately conveyed since prior work has found people tend
to replace words with emoji, and this can result in grammatical errors [
51
]. We believe that this
preemptive visibility of descriptors could be more user-friendly than simply pointing out potential
accessibility issues after the message is written. By having access to descriptors during selection,
users can make more deliberate choices regarding whether a long descriptor is appropriate for a