PreprintPDF Available

Emoji Accessibility for Visually Impaired People

Authors:
Preprints and early-stage research may not have been peer reviewed yet.

Abstract and Figures

Emoji are graphical symbols that appear in many aspects of our lives. Worldwide, around 36 million people are blind and 217 million have a moderate to severe visual impairment. This portion of the population may use and encounter emoji, yet it is unclear what accessibility challenges emoji introduce. We first conducted an online survey with 58 visually impaired participants to understand how they use and encounter emoji online, and the challenges they experience. We then conducted 11 interviews with screen reader users to understand more about the challenges reported in our survey findings. Our interview findings demonstrate that technology is both an enabler and a barrier, emoji descriptors can hinder communication, and therefore the use of emoji impacts social interaction. Using our findings from both studies, we propose best practice when using emoji and recommendations to improve the future accessibility of emoji for visually impaired people.
Content may be subject to copyright.
Emoji Accessibility for Visually Impaired People
Garreth W. Tigwell Benjamin M. Gorman Rachel Menzies
Rochester Institute of Technology Bournemouth University University of Dundee
Rochester, NY, USA Bournemouth, England, UK Dundee, Scotland, UK
garreth.w.tigwell@rit.edu bgorman@bournemouth.ac.uk r.menzies@dundee.ac.uk
ABSTRACT
Emoji are graphical symbols that appear in many aspects of
our lives. Worldwide, around 36 million people are blind and
217 million have a moderate to severe visual impairment. This
portion of the population may use and encounter emoji, yet it is
unclear what accessibility challenges emoji introduce. We first
conducted an online survey with 58 visually impaired partici-
pants to understand how they use and encounter emoji online,
and the challenges they experience. We then conducted 11
interviews with screen reader users to understand more about
the challenges reported in our survey findings. Our interview
findings demonstrate that technology is both an enabler and
a barrier, emoji descriptors can hinder communication, and
therefore the use of emoji impacts social interaction. Using
our findings from both studies, we propose best practice when
using emoji and recommendations to improve the future ac-
cessibility of emoji for visually impaired people.
Author Keywords
Emoji; CMC; Accessibility; Visual Impairments.
CCS Concepts
Human-centered computing Accessibility;
INTRODUCTION
There are currently over 3,000 emoji in the Unicode stan-
dard [27], and emoji are widely adopted in daily commu-
nication. In particular, emoji are prominent across social
media: On Twitter over 25.4 billion tweets contain emoji
(emojitracker.com), 5 billion emoji are sent daily on Face-
book messenger [18], and in 2015 half of the comments and
captions on Instagram contained emoji [19].
Emoji are also prevalent across many other aspects of society.
Companies and marketers engage with audiences using emoji,
some even paying up to 1M USD for custom emoji hashtags
on Twitter [21]. In 2015, the ‘Face With Tears of Joy’ emoji
was selected as word of the year [60] and Domino’s allowed
customers to text or tweet the ‘Slice of Pizza’ emoji to
place an order (anyware.dominos.com). Emoji are evident in
educational settings (e.g., learning management systems [12]),
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a
fee. Request permissions from permissions@acm.org.
CHI ’20, April 25–30, 2020, Honolulu, HI, USA.
© 2020 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-6708-0/20/04 ...$15.00.
http://dx.doi.org/10.1145/10.1145/3313831.3376267
and are used by politicians and government bodies [36, 55],
travel companies [54], media outlets, and public figures (e.g.,
singer Katy Perry who has one of the largest Twitter follow-
ings [51]). Emoji have even been discussed within official
court transcripts [35], and resulted in convictions [23].
People interpret emoji differently, and emoji design variations
across different platforms (e.g., iOS vs Android) can exac-
erbate misunderstandings [45, 64]. Furthermore, emoji are
often used beyond their original intended meaning, which adds
another layer of complexity to disambiguating the intended
use of an emoji [64, 74]. Prior research on emoji has largely
focused on those with typical vision. However, it is estimated
that 36 million people worldwide are blind and 217 million
have a moderate to severe visual impairment [73]. Prior work
highlighted challenges visually impaired people face when
using technology [7] and social media [22, 49]. However, it is
not clear what accessibility challenges occur with emoji.
The popularity of emoji means that any inaccessibility could
lead to social exclusion, leading to a reduced quality of life.
We surveyed 58 visually impaired people to understand the
context around how they use and encounter emoji. Our survey
findings highlighted challenges in searching for emoji, emoji
design, misunderstanding and use in context, and the use of
technology. We identified that emoji introduce more severe
challenges for screen reader users. Therefore, we conducted
11 semi-structured interviews with visually impaired screen
reader users to understand more about their experiences with
emoji. Our thematic analysis demonstrated that technology
was both an enabler and a barrier, emoji descriptors can hinder
communication and use of emoji impacts on social interaction.
Using the findings from our two studies, we introduce best
practice for using emoji with the aim of reducing accessibility
challenges described by participants. We also propose recom-
mendations to platform developers, social media companies,
and the Unicode Consortium to address technical challenges
that emoji pose to users of assistive technology.
Paper Contributions
: We introduce three contributions: 1)
Findings from an online survey with 58 visually impaired
participants that report challenges experienced when using
and encountering emoji online, which validate and extend
anecdotal online discussion of emoji inaccessibility. 2) A the-
matic analysis of 11 interviews with screen reader users that
describes how emoji introduce new barriers when accessing
textual content. 3) We collate our findings into Best Prac-
tices and Technical Recommendations to improve the future
accessibility of emoji for visually impaired people.
BACKGROUND AND RELATED WORK
Computer-mediated communication (CMC), is communica-
tion between people that is facilitated by computers, such as
email or social media [32]. Unfortunately, CMC removes im-
portant non-verbal cues from in-person interactions [33, 37,
47]. To address this, people used emoticons [15], which are
ASCII characters depicting emotion (e.g., using a semi-colon
and right parenthesis
:)
to show a smiley face). Use of emoti-
cons can improve conversation [26] and message intent [40,
70], but have largely been replaced by emoji.
Emoji are a standardised set of unicode characters with visual
representations of emotion, expressions, and objects [28]. As
with any language, people have their own understanding and
approach to using emoji. Emoji are useful for clarifying or
enhancing message intent [14]. Emoji use also extends to
symbolising private jokes, pictorial stories, and maintaining or
showing interest in relationships [14, 56, 64, 74, 77]. However,
emoji can be misunderstood due to variations in how their de-
sign is interpreted [44, 64]. Further miscommunication occurs
because mobile platforms have their own designs (e.g., Apple
iOS vs Google Android) [64]. The unicode characters for each
emoji are constant across all platforms, yet the artwork varies.
For example, the ‘Face With Hand Over Mouth’ emoji has the
code point U+1F92D but can visually change (as shown in
Figure 1). Knowledge about this rendering difference is not
universal [46]. Research has shown there is a cultural gap with
emoji design and user understanding or perception [34, 41],
and personalities can also factor into how emoji are used [39].
Furthermore, searching for an emoji to use can also be dif-
ficult [53]. There has been some effort towards improving
emoji input methods. In particular, Pohl et al. [52] introduced
an emoji similarity model that can be used to improve emoji
keyboard layouts. Auto-selection of emoji based on input is
an alternative solution, with some work looking at improving
the accuracy of emoji prediction [76].
Visual Impairment and CMC Usage
There are many challenges that visually impaired people (e.g.,
blind, low vision, impaired colour vision) face daily. This
includes identifying product brands or names of objects, using
technology, accessing digital services such as websites, and
accessing equal social opportunities [7, 8, 13, 63, 72].
Social media remains one of the most popular ways in which
we stay connected and heavily feature emoji [18, 19, 21, 25].
An analysis of 50,000 visually impaired Facebook users indi-
cated that their engagement with Facebook was on par with
the general population [75], highlighting it is imperative that
accessibility is a key goal of social media. Morris et al. [49]
found that as content on Twitter has become more visual, blind
users were less likely to be able to participate. Twitter now
allows users to enable image descriptions, however only 0.1%
of images were found to include one [22]. Furthermore, Twit-
ter users who have enabled image descriptions only use it on
~50% of their images [22]. Alt text (alternative text) improves
the accessibility of online images for screen reader users, yet
has remained relatively unchanged since its inception [48].
Morris et al. [48] address this by proposing several ways in
which to improve the alt text experience when accessing visual
content. It cannot be expected that the assistive technology
used will resolve all potential problems, but that social media
platforms must also make accommodations to improve acces-
sibility [10]. Numerous accessibility challenges with social
media have been identified. However, there is insufficient
research on emoji communication challenges.
A. B. C.
Figure 1. A.) ‘Face With Hand Over Mouth’ emoji on iOS and macOS.
B.) ‘Face With Hand Over Mouth’ emoji on Android. C.) ‘Slightly Smil-
ing Face’ emoji on iOS and macOS.
Emoji Accessibility
The Unicode Consortium is a non-profit organization that so-
licits proposals for new emoji. They also author and maintain
a Common Locale Data Repository (CLDR) to store short
character names for each emoji [3]. These descriptors are then
used by screen readers to vocalise what the emoji visually
represents. If we consider again the ‘Face With Hand Over
Mouth’ emoji shown in Figure 1.A and 1.B, VoiceOver on
macOS will read aloud the descriptor face with hand over
mouth with rosy cheeks”. However, notice the descriptor is in-
consistent. The blushing is 3D shading of the iOS design (see
Figure 1.C). A screen reader user familiar with the descriptor
may not realise the emoji does not match the descriptor.
There have been some attempts to improve emoji accessibility.
For example, to address when emoji are sometimes not recog-
nised by screen readers, one suggestion is to force the emoji
to be recognised as an image and to give it an aria-label [71].
This technique can also be used to address the fact that emoji
can have several meanings by providing the user with alter-
native descriptions [57]. Another approach to improve emoji
accessibility for visually impaired people involved an investi-
gation into tactile emoji [11]. Although the study found that
visually impaired people could successfully identify tactile
emoji to the represented emotion, it is not clear what the par-
ticipant’s current behaviour for using emoji were (if any) and
what types of emoji accessibility challenges they encounter.
Individuals with low vision have expressed challenges of us-
ing emoji [69], such as distinguishing between them without
the use of assistive technology, and difficulty inserting emoji
into text without using dictation software. Solutions discussed
include changing the skin colour of the ‘thumbs up’ emoji
(using a skin tone modifier such as ‘Fitzpatrick Type-4’
) to improve the visibility against certain backgrounds or
requesting friends apply different colours to the ‘thumbs up’
emoji [69]. Work has explored designing new emoji specifi-
cally for visually impaired people [38], yet this is unlikely to
solve problems caused by emoji use in the general population.
Sufficient attention on emoji accessibility is required. There-
fore, we first conducted an online survey to understand how
visually impaired people use and encounter emoji online.
ONLINE QUESTIONNAIRE METHOD
We ran an online questionnaire with people who self-identified
as visually impaired (e.g., Blind, impaired colour vision) to
address three research questions: 1) How do visually impaired
people use emoji? 2) Do visually impaired people encounter
emoji? 3) What challenges do emoji and other forms of non-
textual information present to visually impaired people?
Materials
The online questionnaire had 29 questions (12 closed-ended
and 17 open-ended) across four sections. The first section con-
tained eight questions and gathered basic demographic infor-
mation: age, gender (we used an open text field in anticipation
of a small sample size typical of accessibility research [58]),
visual impairments, when visual impairments were acquired,
visual acuity, assistive technology used to browse the inter-
net and use messaging services, and message sending/posting
frequency. The second section focused on using emoji and
contained six questions. Participants were asked if they ever
used emoji and if the response was “Yes”, then the participants
were asked how frequently they use emoji, to provide some
examples of where and when emoji are used, to describe the
reasons for using emoji, what the likelihood of using an emoji
was within and outside of a work context. Participants who do
not use emoji were asked to describe the reasons.
The third section focused on participants’ experience of en-
countering emoji. Participants were asked if they had ever
encountered emoji in content that someone else had written
and if the response was “Yes”, then the participants were
asked how frequently they encounter emoji, to provide some
examples of where and when emoji are encountered, what
advantages there are to emoji being included within content,
what disadvantages there are to emoji being included within
content, and what the likelihood of encountering an emoji was
within and outside of a work context. Participants who did not
encounter emoji were asked to describe the reasons.
The final section explored the challenges emoji and other non-
textual information present when used in content. Participants
were asked how well they understood why other people use
emoji in content, what challenges they experience when using
emoji, what challenges they experience when encountering
emoji, if they were aware emoji look visually different across
platforms, and what other forms of non-textual information
they encounter in content. Finally, participants were asked
if they send other forms of non-textual information, and the
advantages and disadvantages of each compared to emoji.
Procedure
Ethical approval was obtained from our IRB. We distributed
the survey through social media (e.g., Facebook, Twitter), Red-
dit (r/samplesize, r/blind, r/colorblind, r/glaucoma), university
mailing lists, and by contacting charities and organisations
(e.g., RNIB - rnib.org.uk, NFB - nfb.org). Admin permission
was sought in all cases where we were outside of a group
space. The participants were given an opportunity to enter into
a prize draw for a $50 USD (or equivalent) Amazon voucher.
The survey was open from May 15 2019 for three weeks.
Participants
In total, 66 participants completed the questionnaire. Eight
participants were removed from our analysis (five were under
18 years old, two did not respond to any questions, and one did
not have a visual impairment). The remaining 58 participants
(Male = 43, Female = 12, Undisclosed = 3) were aged between
18-57 years old (M = 29.59, SD = 10.11).
We asked our participants about the number of visual impair-
ments they have due to diversity in disability e.g. somebody
could be colour blind from birth and later develop glaucoma.
Data revealed: Single impairment (32 participants), two im-
pairments (11), three impairments (5), four impairments (1),
five impairments (2), and not given (7). The type of visual
impairment varied greatly amongst the participants: Impaired
Colour Vision (30 participants), nerve damage (9), blind (8),
myopia (8), retina issues (5), eye development issues (4), nys-
tagmus (3), macula issues (3), albinism (2), aniridia (2), astig-
matism (2), low vision (2), photophobia (2). Finally, there was
one participant for each of the following Axenfeld-Rieger syn-
drome, congenital cataracts, corneal edema, Leber congenital
amaurosis, and punctate inner choroidopathy (PIC), and seven
participants did not report details of their visual impairment.
Our participants also reported on the development or occur-
rence of their visual impairments: Since birth (47 participants),
0-5 years (3), 5-10 years (6), 10-15 years (2), 16+ years (3).
There was one unclear response and two did not respond.
Participants were asked to rate their visual acuity using the tex-
tual descriptions proposed by the World Health Organisation
(WHO) [20]: “None or Mild (equal to or better than 20/70)”
(27 participants), “Moderate (worse than 20/70 and equal to
or better than 20/200)” (9), “Severe (worse than 20/200 and
equal to or better than 20/400)” (5), and “Blind (worse than
20/400)” (12). Four participants provided different responses:
colorblind, I don’t know sorry”, “normal with glasses”, “with
glasses, left acuity can be brought to None or Mild, besides
blind spot which is 0; right acuity is blind”, and “one eye is
20/40 with correction. One eye is worse than 20/400 with
correction”. One participant did not respond to the question.
Our participants used a variety of assistive technologies when
browsing the internet and using messaging services. We found
colour filters (both digital and physical, such as tinted glasses)
and identifiers were used by 18 participants. Overall, screen
reader software was used by 17 participants: JAWS (6 partici-
pants), VoiceOver (5 participants), NVDA (3), and TalkBack
(3). Other assistive technologies participants reported using
were: Magnification (10), glasses/contact lenses (5), braille or
braille displays (4), none (2), and not given (11).
To build an understanding of the participants behaviour we
asked how often they send messages (e.g., text messages, e-
mail) and post messages online (e.g., Facebook posts, tweets).
We found 49 participants send messages at least once a day and
nine participants at least once a week. Finally, 27 participants
reported they post messages online at least once a day, 19
participants at least once a week, four participants at least
once a month, and eight participants less than once a month.
Analysis
We analysed our open-ended responses using open coding [67]
based on an existing procedure [65]. We analysed each ques-
tion independently using a four-step process: 1) Initial coding:
The first author read all responses and generated initial codes
with a data-driven approach. Codes were collated and col-
lapsed into an initial codebook. 2) Evaluating codes: The
first two authors independently coded 1/3 of the responses
(randomly-selected) using the initial codebook, agreeing to
identify each ‘mention’ once. Codes and descriptions were re-
fined by discussing disagreements. 3) Coding the full data set:
The same authors separately re-coded all responses with the
updated codebook and rules. 4) Defining themes: The same
authors reviewed the final coding to identify similarities that
allowed thematic grouping. We collated codes into themes
and therefore did not calculate survey inter-rater reliability
because codes were not the final outcome of our analysis [43].
ONLINE QUESTIONNAIRE FINDINGS
We present our findings, using closed-response data and par-
ticipant quotes, under three thematic sections: 1) Using Emoji,
2) Encountering Emoji and 3) Challenges of Emoji.
Using Emoji
We asked our participants if they have ever used an emoji (i.e.
sent or posted online) and found that 56 participants had and
only two participants had not. The reasons given for not using
emoji, by the two participants, included: the size of emoji (1
participant), that they did not think about using emoji (1), and
that using text-to-speech made it difficult (1). Participants who
used emoji reported varying frequencies, as summarised in
Table 1. Our subjective scale ranged from “Less than once a
month” to “At least once per day” because further granularity
is difficult for participants to accurately recall [61]. Overall,
55% of participants used emoji at least once per day and 74%
of participants sent emoji at least once per week.
Frequency of Using Emoji No. of Participants
At Least Once a Day 32
At Least Once a Week 11
At Least Once a Month 5
Less than once a month 7
Not Given 3
Table 1. Frequency participants reported using emoji.
Overall, participants reported diverse examples of where and
when they used emoji: private (35 participants) and public
contexts (20), during conversations (9), for conveying emotion
and reactions (9), within a work or productivity context (7),
for fun or to add humour (7), for clarification in messages
(4), when at home (1) or travelling (1), and as a result of
systems automatically suggesting or inserting emoji (1). Two
participants mentioned not sending or using emoji and five
participants did not answer or provided a non-useful response.
Reasons for Using Emoji
Participants described many reasons for why they use emoji.
Participants mentioned the benefit of clarifying or enhancing
message content (46 participants), the fun aspect or adding
humour (26), the speed or ease of making a statement or re-
plying to a message (10), the ability for unique usage such as
assigning alternative meaning to emoji (3), using emoji with
friends and family (1), utilising the less formal nature of emoji
(1), using emoji for work or with colleagues (1), the social
pressure to fit in with others (1), and as a result of systems au-
tomatically suggesting or inserting emoji (1). Two participants
mentioned not sending or using emoji and three participants
did not answer the question or provided a non-useful response.
Many participants mentioned the use of emoji to enhance and
clarify content within communication, for example to avoid
“...words being taken the wrong way.(P27). Some participants
conveyed clear enjoyment of using emoji, noting that emoji
were “cute” (P66) and that their use “adds something fun to
messages” (P10). P51 specifically commented that they use
emoji to be “...perceived as someone who is able to embrace
sighted culture”. We also asked participants to report what
the likelihood was of them using emoji within and outside of a
work context (as summarised in Table 2). Overall, participants
are more likely to use emoji outside of their work environment.
Likelihood of Within Outside
Using Emoji Work Context Work Context
Always 0 5
Likely 6 28
Neutral 4 11
Unlikely 28 11
Never 17 1
Not Given 3 2
Table 2. Participants’ reported likelihood of using an emoji within and
outside a work context (Neutral is neither likely nor unlikely).
Encountering Emoji
We asked our participants if they have ever encountered (i.e.,
read) an emoji in text that someone else has written (e.g.,
in a message or tweet) and found 56 participants had, one
participant had not, and one participant did not answer the
question. Participants reported varying frequencies of encoun-
tering emoji as summarised in Table 3.
Frequency of Encountering Emoji No. of Participants
At Least Once a Day 45
At Least Once a Week 8
At Least Once a Month 1
Less than once a month 2
Not Given 2
Table 3. Frequency participants reported encountering emoji.
Overall, participants reported a variety of situations where
they have encountered emoji: in private (43 participants) and
public (32) contexts, with friends and family (14), at work or
with colleagues (7), when emotion and reactions needed to
be conveyed (7), at home (1), and during significant events
or anniversaries (1). Three participants did not answer the
question or provided a non-useful response.
To better understand the context around when participants
encountered emoji, we asked them to report what the likeli-
hood was of them encountering emoji within and outside of a
work context (as summarised in Table 4). Overall, participants
were more likely to encounter emoji outside of their work
environment, as was also the case with using emoji.
Likelihood of Within Outside
Encountering Emoji Work Context Work Context
Always
Likely
Neutral
1
7
5
10
34
7
Unlikely
Never
33
9
4
0
Not Given 3 2
Table 4. Participants’ reported likelihood of encountering emoji within
and outside a work context (Neutral is neither likely nor unlikely).
Advantages of Encountering Emoji
Participants described many advantages of emoji being in-
cluded within content, including: for clarification within
messages (37 participants), increased speed or ease of mak-
ing/replying to a statement (9), for fun (5), and because emoji
are useful as a universal language (1). Nine participants men-
tioned negative aspects of emoji or statements about emoji
offering no advantages with four participants not answering or
providing non useful responses. As with using emoji, the most
popular advantage was to clarify or enhance content within
a message, such as described by P46: “It can convey things
beyond words”. Speed and ease of use were also mentioned,
such as they “save time and convey [emotion] clearly” (P58),
and emoji are “easier to express a feeling” (P6). One partici-
pant also discussed that the “empathy aspect is also valuable,
particularly when people tell a story with emoji” (P55).
Disadvantages of Encountering Emoji
Participants outlined disadvantages to emoji being included
within content. The responses included: emoji are informal
or non-serious nature (15 participants), over reliance of emoji
over real words (12), the risk of confusion and misunderstand-
ing (12), technology challenges due to emoji (11), repeated
auditory feedback when multiple emoji are used (6), visual
clutter caused by multiple emoji within a message (5), find-
ing an emoji to use (2) and the difficulty of perceiving emoji
(1). Eight participants mentioned negative aspects of emoji
or statements about emoji offering no advantages and three
participants did not answer or provided non-useful responses.
Challenges of Emoji
Participants self-reported how well they understood why other
people use certain emoji in their writing. We categorised
participants’ comments on their understanding into Excellent
(13 participants), Good (22) and Poor (4). This was based on
a broad understanding of participant comments and not as a
judgement of their level of ‘correctness’. Some participants
gave specific examples of why they think others use emoji.
These included: the ability to more expressive (9 participants),
some emoji are used because they are easy to understand
based on the look or surrounding context (8), to try and avoid
a misunderstanding (7), for fun or to add a personal touch
(5), when there is good knowledge of the person and their
communication style (4), to keep messages concise (3), and
because emoji are useful as a universal language (1). Nine
participants did not answer the question or provided a non-
useful response and one participant’s response was unclear.
Challenges of Using Emoji
Participants reported the challenges they experienced when
using emoji and included: searching for an emoji to use (26
participants), challenges related to emoji visual design (13),
resulting in confusion and misunderstanding (8), the technol-
ogy challenge that emoji introduce (8), issues due to emoji
size (4), limited access to appropriate emoji for the context (4),
and the situation or message not agreeing with the emoji (3).
Overall, 10 participants reported no challenges and eight did
not answer the question or provided a non-useful response.
Searching for Emoji:
Overall, 45% of participants reported
challenges when searching for an emoji to use. In particular,
this was related to finding an emoji that fitted what they were
trying to convey in written text such as P4 who said that it was
“...hard to find the one [to] represent what I mean”.
Technology:
Participants mentioned that they sometimes ex-
perienced issues related to selecting emoji. For example:
P44: “Sometimes if you miss click an emoji it might get weird.
Like if you send a heart [ ] to someone you’d never send a
heart to and then have to explain it was an error, which also
might be weird.
Participants also reported that touch screen settings were chal-
lenging: “not being able to find them quickly or 3D touch
being a little too twitchy...so getting the wrong ones” (P50).
Visual Design:
Emoji design also caused challenges for 22%
of the participants. For participants who had some residual
vision, this was often related to the use of colour such as P6
who described that “the colors of the heart [emoji] can be too
similar.. For blind participants, differences between design
of the visual emoji and the description were challenging:
P28:“Some emoji [are] useless or just have a bad design (I
was told the ‘pray’ emoji [ ] is actually a ‘high five’).
Misunderstanding: This relates to the use of visual represen-
tations of things that blind users had not experienced. This
sometimes made it difficult to select an emoji.
P38: “...I entered the word ‘happy’, and it suggested many
faces, which were all described to me; however, as I have
never had vision, I was unable to know which face was the
most appropriate for my situation.
Emoji Size:
For participants with residual vision, they de-
scribed trying to identify specific emoji, but finding it difficult
when many are presented on small mobile device screens: “I
only use a select few as most expressions or objects are too
small to identify” (P59).
Limited Access:
Some participants noted that there are “lim-
ited images to choose from” (P19). This may be because the
emoji are displayed across multiple screens, which can make
it a challenge to find a specific one.
P03: “So many emoji without a search bar for example. It
leaves me going back to endless pages of obsolete ones.
Context:
For some participants, context was a challenge when
selecting and sending an emoji. Participants noted specific
Figure 2. Three recreations of real tweets illustrating problems reported by our survey and interview participants. A) A tweet where the author has
used emoji to draw a picture of three large flowers. B) A tweet sent by a conference attendee. Rather than using the word airport they use two plane
emoji and asks fellow attendees to reply with an emoji. C) A tweet sent by a rail company containing important information about rail delays.
emoji, e.g. the ‘peach’ emoji, which has an alternate meaning
within popular culture: “One example is the peach emoji [ ],
I have never used it myself, but only recently became aware
that it’s generally accepted to be a butt.(P9). See Figure 2.B
for another example of context issues.
Challenges of Encountering Emoji
We asked participants if they were aware that emoji looked
visually different across platforms. Forty-eight participants
were aware, eight participants did not know, and four partici-
pants provided no response. Participants noted “There should
be standardization.” (P19) and gave examples of experiences:
P43: “I had no clue about this... [my sister] told me that on
iOS, emojis look more high-quality and visually appealing”.
In addition, all participants were asked what challenges they
experienced when reading emoji. Challenges given were the
risk of confusion and misunderstanding (14 participants), the
technology challenge that come with emoji (13), issues with
emoji size (10), understanding why an emoji was used in a
particular context (9), challenges related to emoji design (7),
and challenges related to people misusing emoji (7). Fourteen
participants reported no challenges and five participants did
not answer the question or provided a non-useful response.
Misunderstanding:
This was the most common challenge
reported and was related to subjective understanding of intent:
P43: “Sometimes when [people] send an emoji to a blind
person, the emoji is meant for a different thing and more often
than not, does not sound the same as it might look.
Participant level of exposure was not a factor in the misun-
derstandings reported: 10/14 participants who reported the
biggest challenge to be risk of confusion and misunderstanding
also encountered emoji at least once per day.
Technology:
Participants commented that there was a lot of
variation in emoji between different devices and applications,
and that some emoji may not be supported on different plat-
forms or devices such as screen readers:
P19: “The facial expressions vary from app to app, which
means that a super excited face on one app might be an an
excited yet angry face on another.
Context:
Participants discussed the importance of context,
with some explaining how they were not always able to deter-
mine the intent of a message from the emoji descriptors:
P32: “The alt text may not match the context that the user
is trying to provide making the overall intent of the message
confusing to a screen reader user.
This was more problematic when emoji were used as decora-
tion (e.g., the pictorial drawing shown in Figure 2.A), and the
descriptors were not related to the surrounding semantics.
Visual Design:
For participants with residual vision, the small
size of the emoji contributed to challenges in receiving emoji
such as P7 described: “Sometimes the [emoji] face is too
small to understand.. The visual features of the emoji can
also contribute to this challenge. P10 described being “unable
to distinguish different colors of the emojis”. P20 reported
having “difficulty seeing the expression on the emoji..
Misuse:
Finally, participants noted situations where they con-
sidered emoji to be misused. For example, P53 reported that
“Too many [emoji] used in one message make it laborious to
read ...” and this is challenging to interpret when using a
screen reader or other assistive technology. An example of
overuse can be seen in Figure 2.A and 2.B.
Sending Other Non-Textual Information
We asked participants if they send other forms of non-textual
information, and to state any advantages and disadvantages
of each one compared to emoji. The non-textual information
noted by participants was emoticons (16 participants), GIFs
(14), images (10), audio (6), video (5), memes (2), ASCII art
(2), stickers (2), and 27 participants did not provide a response.
Many participants described emoticons as other methods of
conveying non-textual information, and did so positively:
P51: “Emoticons are easier to type than emoji for me, espe-
cially when typing in Braille and not at a computer.
Finally, participants described how other non-textual informa-
tion allowed them to convey more expression and detail:
P66: “Gifs, audio, and video allow for a more full context of
what someone is saying or feeling than an emoji or emoticon
can. Emojis and emoticons are easier to use though.
Summary of Questionnaire Findings
Our findings highlighted numerous challenges faced by visu-
ally impaired people when using and encountering emoji. This
included searching for emoji to use, emoji design, misunder-
standing and use in context, and the use of technology. The
negative impact emoji had on users of assistive technology was
considerable, especially for screen reader users. These users
are typically reliant on the emoji descriptor (e.g., ‘Face with
Tears of Joy’ emoji ), which can make emoji challenging
to understand, as the descriptor may not match the intended
use. To understand more about the challenges screen reader
users encounter when using and encountering emoji online,
we conducted one-to-one interviews with screen reader users.
SCREEN READER USER INTERVIEWS METHOD
We had two main research questions guiding our interviews: 1)
What challenges are experienced by screen reader users when
sending emoji? and 2) What challenges are experienced by
screen reader users when perceiving and understanding emoji?
Procedure
After obtaining ethical approval from our IRB, we conducted
semi-structured one-to-one interviews using online messaging
tools. We did this for two reasons: First, interviewing over
a messaging service would allow participants to share emoji
if they wanted to provide examples; Second, there would be
more convenience for participants (e.g., not having to go some-
where private to talk over the phone). We selected the tool
in collaboration with each participant to ensure accessibility
(since our participants were users of screen readers). The third
author conducted all interviews. A pilot interview was held
with the second author prior to beginning the study. Partic-
ipants were recruited using the same methods used for the
survey, and were reimbursed for their time with an Amazon
voucher equivalent to $20 USD. The mean interview time was
62 minutes (max 70 minutes, min 48 minutes). Participants
completed a pre-questionnaire to gather demographic data (see
supplementary material) that was anonymised for analysis.
Participants
We interviewed 11 participants (Male = 8; Female = 2; Agen-
der = 1), aged between 18-37 years old (M = 28, SD = 6.15).
Of these 11 participants, 10 believed that their visual impair-
ment impacted their use of emoji. Participants were asked
to rate their visual acuity using the textual descriptions pro-
posed by the World Health Organisation (WHO) [20]: “Severe
(worse than 20/200 and equal to or better than 20/400” (1
participant), and “Blind (worse than 20/400)” (9 participants).
One participant provided a different response: “Full blindness
with zero light perception but physical eyes still remain”.
We asked participants about the visual impairment(s) they
have: Single impairment (8 participants), and two impairments
(3). The type of visual impairment varied greatly amongst the
participants: Blind (8 participants), Low Vision (1), Retina
Issues (2), Nerve Damage (1), Axenfeld-Rieger syndrome (1).
Our participants also reported on the development or occur-
rence of their visual impairments: Since birth (4 participants),
0-5 years (4), 5-10 years (0), 10-15 years (1), 16+ years (2).
All participants were self-reported screen reader users and
used a variety of software: Voiceover (10 participants), NVDA
(5), JAWS (3), and Talkback (1). All participants reported
sending messages (e.g., Facebook messages, SMS) at least
once per day. Posting messages online (e.g. discussion forums,
Facebook posts) was more varied with participants reporting
this action at least once per day (7 participants), once per week
(1), once per month (1) and less than once per month (2).
We asked participants how often they sent emoji. This was an
open field so more details could be provided. All participants
had sent emoji; responses ranged from “every day” to “hardly
ever”. Two participants also mentioned using emoticons more
often than emoji. We also asked participants how often they
receive emoji. Responses were less varied, with participants
stating “almost every day” as a minimum frequency, with
most participants stating “daily” or “all of the time”.
Analysis
We analysed our interview transcripts using thematic analy-
sis [9]. The first and third author read all responses and took
note of initial codes. Codes were generated using a data-driven
approach then collated and collapsed. The same authors then
reviewed the final coding and identified similarities to allow
thematic grouping by creating an initial thematic map. Our
final thematic map is shown in Figure 3. We did not con-
duct inter-rater reliability because it is not part of Braun and
Clarke’s checklist for good thematic analysis [9], and there is
debate if it is suitable for this type of analysis [5, 42, 24].
SCREEN READER USER INTERVIEWS FINDINGS
Through our thematic analysis, we identified three themes: 1)
Technology is both an enabler and a barrier, 2) Emoji descrip-
tors can hinder communications, and 3) Use of emoji impacts
social interaction. We now explore, and scaffold the narrative
of each theme in detail using quotes from participants.
Technology is Both an Enabler and a Barrier
For visually impaired people, screen readers are crucial for
accessing visual content, however our interview participants
described that screen readers could be a barrier to emoji ac-
cessibility. In particular, participants highlighted a range of
challenges related to Searching and Selection, Output from
Technology, Up-to-Date and Knowledge About Technology.
Searching and Selection:
All of our participants found search-
ing and selection of emoji to be challenging. There are several
elements, such as the organisation of emoji lists, the available
mechanisms for finding emoji (e.g., keyword searching), and
knowing what is available. In particular, the increasing number
of emoji that exist is further highlighting this challenge.
P2:“...finding the right one to send. I either don’t know
whether it exists or what it is, or where to find it. Sighted
people just glance at a screen and can find them pretty quickly,
while we have to go through all of them.
In relation to the large numbers of emoji, P3 recommended
that some emoji are grouped together to simplify searching,
e.g. “It would help to have an option to change the skin tone
instead of having them all there”. Some participants outlined
alternative solutions for when they are unable to find a specific
emoji:
P9: “I have spent a lot of time looking for emoji that I know
exist but cannot find...I have sometimes found an old instance
of the emoji in a previous conversation and copied it.
Output from Technology:
Our participants noted that it could
be challenging to identify emoji within output, especially
where emoji are disabled or incompatible. Different screen
reading technologies may describe emoji in different ways.
P4: “JAWS describes [ ] as ‘face with look of triumph’
while Voiceover describes it as ‘huffing with anger face’ and
according to my sighted brother, Voiceover’s description is
more accurate.
USE OF EMOJI IMPACTS SOCIAL INTERACTION
Conversational
Flow
Cultural
Differences
TECHNOLOGY IS BOTH AN ENABLER AND A BARRIER
Output from
Technology
Knowledge about
Technology
Up-to-Date
Searching and
Selection
EMOJI DESCRIPTIONS CAN
HINDER COMMUNICATION
Figure 3. Final thematic map of three main themes and their sub-themes: 1) Technology is both an enabler and a barrier, 2) Emoji descriptions can
hinder communications, and 3) Use of emoji impacts social interaction.
Participants also noted that reading emoji can lengthen the
output from screen readers, which can be inconvenient.
P7: “Whenever someone sends a string of emojis as a joke, it
is annoying to have to hear them all strung together.
Up-to-Date:
Our participants highlighted that their assistive
technologies were not always up to date and this meant that
emoji were not always fully supported. For example, new
emoji were not added, or that there was a reliance on third party
applications such as screen reader libraries or soft keyboards.
P1: “The [NVDA] plugin is very out of date and supports
~100 mixed emoji and emoticons, [but] the built-in dictionary
supports ~3000 but that’s mixed emoji and symbols...
Knowledge About Technology:
Our participants highlighted
that different screen readers were available and they needed
to be technically aware in order to access different features
that could help them. The level of competence varied between
participants, e.g., one participant was aware of the punctuation
settings on NVDA, which could avoid repeated emoji being
read out in their entirety, but others were not. Indeed, some
participants recommended such a feature to us as a solution.
Emoji Descriptors Can Hinder Communications
On a screen reader, emoji descriptors are output as speech or
braille and describe the visual design of an emoji. However,
the descriptor does not always accurately describe the visual
design, which can lead to challenges when using emoji:
P6: “Emoji is something fun for sighted texters...but for me
it’s just an extra string of words. ...like the grinning face
emoji [ ]; it looks fun and cute when you look at it, but
Voiceover describes it as ‘grinning face with clenched teeth
emoji’ which sounds more like a grimace than a big smile).
For complex emoji, the descriptors can also be verbose, which
makes communication with a screen reader cumbersome. P7
added there was a “user education issue” and that “if sighted
users knew what the [descriptor] was, it may help”.
Use of Emoji Impacts Social Interaction
Our participants described how using emoji in conversations
could lead to communication breakdown and social exclusion.
Our participants highlighted challenges related to Poor Use in
Context, Conversational Flow and Cultural Differences.
Poor Use in Context:
Our participants highlighted that emoji
used in different contexts can lead to specific challenges. Dec-
orative emoji, e.g. emoji in usernames on social media, caused
challenges as many decorative emoji could be announced by a
screen reader. An example of this is shown in Figure 2.A.
P7: “Try listening to ‘cat with heart shaped eyes fireworks
sparkles watermelon kissing face flag of Andorra’ a few times
in a row and you get the frustration.
It is possible to avoid screen readers announcing emoji in cer-
tain contexts, e.g. in usernames, as reported by one participant,
but this was reliant on an unofficial plugin. Descriptors often
did not match the intended purpose, e.g. emoji were selected
based on their visual representation, leading to misunderstand-
ing. See Figure 2.C for an example.
P8: ”Email subjects have emoji now; Ebay put a [ ] to show
your order has been sent. For a long time, I [was] puzzled as
to why they’d write the word ‘truck’ there.
Conversation Flow:
Our participants discussed how emoji
enhance conversations, such as enabling quick replies. Mis-
understandings could change the conversation tone, and were
more likely when unfamiliar emoji were used. Some partici-
pants reported ignoring a conversation when emoji were used
extensively, so both context and content was missed. For exam-
ple, P2 did not know about an important test being rescheduled
because the information was lost within emoji:
P2: “in the middle of [multiple heart emoji], someone posted
something else, which was also important. I didn’t pay atten-
tion [to] the wave of hearts, so I didn’t know”.
Cultural Differences:
Our participants highlighted differ-
ences between sighted and non-sighted culture and a desire
for social inclusion. Participants described aiming to engage
with sighted popular culture using emoji, but often relied on
emoticons as they were platform independent and more easily
understood by their visually impaired peers. Participants who
had been blind since birth commented that the link between
visual representation and intent can be challenging.
P8: “Imagine you’re totally blind, you’d never seen the ges-
ture. So that has to be learned. You can write ‘no’ or ‘that’s
bad’ etc. Choosing to send a pictorial representation of a
negative feeling may pose more of a challenge.
In addition, P6 discussed feeling excluded from society:
P6: “It’s a bit frustrating and depressing. I don’t follow many
people and it’s sad to suddenly be shut out of content or a
conversation solely because of a text decoration trend.
Summary of Interview Findings
Our findings reveal emoji challenges experienced by screen
reader users. We found that screen reader technology is both
an enabler and a barrier. Emoji descriptors also introduce prob-
lems and they can result in misunderstandings, and therefore
the use of emoji has a significant impact on social interaction.
BEST PRACTICES & RECOMMENDATIONS
Emoji are part of most social platforms and modern commu-
nication. Our findings describe why people may choose not
to use emoji, but encountering emoji is something that users
have little influence over. Emoji accessibility is an area not
fully understood and is having a detrimental impact on the
social inclusion of visually impaired users. To address this,
we use our findings to introduce emoji use best practices and
recommendations for future emoji development.
Best Practice When Using Emoji
People who use emoji should consider the sender, the reader
or recipient, and the platform that is being used to both send
and receive the emoji. Understanding the needs of the end
user should be a primary consideration.
1) Number of Emoji:
Repeated emoji can cause considerable
annoyance and frustration. If you wish to say that something
is amusing and are using the ‘Face with Tears of Joy’ emoji,
consider that each time you add that emoji may increase the
number of times the descriptor is read out:
“So funny! vs. “So funny!
Therefore, consider how many emoji are necessary in content.
2) Placement of Emoji:
Consider that a sentence with emoji
will be read by a screen reader as if the emoji were text. This
is especially important because the descriptors may not match
with your expectations (e.g., is read as “sun”, not “sunny”).
“It is today.” vs. “It is sunny today .
Emoji in usernames (such as on Twitter) should be avoided.
At a minimum, consider placing emoji at the end of your
username, which means that screen reader users can skip past
the emoji once they understand who is posting. Therefore,
consider placing decorative emoji at the end of content, or at
a minimum at the end of each line of content.
3) Purpose of Emoji:
The descriptors for emoji are not always
a clear indication of the visual design or the emotive intent.
Such information should also be represented in the surround-
ing text. Therefore, consider that emoji (or the descriptor)
should not be used to convey critical information in content.
4) Consideration of Reader:
An understanding of how differ-
ent users can perceive emoji is important. When your emoji
has a greater reach (such as on a public platform), there is a
greater chance that the emoji will be encountered by visually
impaired people. Therefore, consider the wider context before
using emoji and ensure that the accessibility of your content
is evaluated before sending or sharing.
Recommendations for Future Emoji Releases
Our recommendations discuss technical issues that need to be
considered by vendors and organisations to further improve the
accessibility of emoji prior to future emoji standard releases.
1) Emoji Descriptors:
Discrepancies between the visual de-
sign of emoji and their descriptors can cause confusion and
misunderstanding. We recommend that descriptors and visual
designs are approved for consistency by the Unicode consor-
tium. Progressive detail [48] could provide additional context
for screen reader users and allow screen reader users to make
more informed use of emoji. Vendors and assistive technology
designers should consider implementing progressive detail.
The presentation of duplicate emoji also varies between screen
readers, with some allowing users to reduce repetition, e.g.,
“three ‘Face with Tears of Joy’ emoji”. As emoji become more
widely used, this setting should be implemented on all screen
readers and made more prominent to users.
Additions to the emoji standard could also increase issues
caused by descriptors. For instance, the Unicode Committee is
reviewing whether to add colour modifiers to Unicode Emoji
V13.0 [30]. This mechanism would use the emoji colour
characters (e.g., seven coloured square characters at U+1F7E6
– U+1F7EB) to allow additional emoji representations such as
a glass of ‘White Wine’ (‘Wine Glass’ + ‘White Square’
). However, these coloured emoji would be represented
by two emoji, so the underlying emoji descriptors would be
“Wine Glass, White Square” and not ‘White Wine’.
2) Unsupported Emoji:
Unsupported emoji occur when one
platform, e.g. Apple, releases new emoji faster than others, e.g.
Android. New emoji are then encountered on devices with an
outdated emoji standard. The new emoji are not rendered cor-
rectly, nor do they have an updated list of the emoji descriptors
as a fallback. We recommend that new emoji are embargoed
until a specific date after a standard has been approved.
Further challenges due to unsupported emoji would arise if
Unicode approve QID Emoji Tag Sequences or QID emoji in
Emoji Standard V13.0 [30]. QID emoji would allow for com-
munities and companies to use this mechanism to put together
their own sets of emoji. The character of each of these emoji is
established by reference to a Wikidata QID (unique identifier
used by Wikidata). For instance, the ‘Sauropod’ emoji +
Q14384 (Triceratops QID – wikidata.org/wiki/Q14384) could
result in a ‘Triceratops’ emoji , but only if a valid visual
design was available. However, if there was no visual design
available, the fallback for screen readers discussed by Uni-
code [30], would be to indicate that there was an emoji present
and not provide any indication to what that emoji represented.
We recommend that Unicode think carefully about the acces-
sibility implications of allowing external organisations and
companies to add additional non-standardised emoji.
3) Platform Visual Differences:
Platform differences cause
misunderstandings between users, and these are greater when
a user has a visual impairment. The Unicode design guide-
lines [29] state that platform differences are possible yet a
design that varies significantly from other vendors’ represen-
tations may cause interoperability problems. We recommend
that visual designs by each platform should be approved by
Unicode to ensure adherence to the emoji descriptor.
4) Facilitating Emoji Use:
Emoji selection currently requires
a visual search. Some work has been completed toward al-
ternative emoji selection [53, 17], but none are specifically
designed for visually impaired people. Vendors should de-
velop alternative emoji input methods. Employing co-design
methods with visually impaired people is imperative in the
design and evaluation of new solutions. For example, custom
gesture-based input could be one direction [4, 6, 50].
5) Diversity in Design Process:
Our participants did not dis-
cuss how emoji designs were/were not representative of people
with different abilities. However, it is unclear if users are repre-
sented within Unicode [68], or are involved in the development
of new standards. Inclusive design processes are established
practice for the development of accessible technology [31].
We recommend that an accessibility sub-committee is formed
to include people with a diverse range of abilities, which will
lead to more representative emoji in terms of both design and
access needs. There has also been a call to democratise emoji
design through more public engagement [62], which could be
an additional step towards improving emoji accessibility.
DISCUSSION
Summary of Contributions
1) Questionnaire data on emoji usage from 58 visually im-
paired participants: Our findings demonstrate that emoji in-
troduce a number of challenges for visually impaired people
when using, and encountering emoji online. This includes
searching for emoji to send within content, emoji design, mis-
understanding emoji intent and their use in context, and the
impact emoji has on assistive technology. Our findings extend
prior work in this area [14, 44, 45, 64, 74], and confirm anec-
dotal online discussion regarding how emoji can introduce
accessibility challenges for visually impaired people, which
are more severe for screen reader users.
2) Thematic analysis of 11 interviews with screen reader users:
Our interview findings with screen reader users demonstrate
that screen reading technologies are both an enabler and a
barrier, emoji descriptors can hinder communication, and the
use of emoji impacts social interaction. These findings show
that the use of emoji within textual content has reintroduced
accessibility barriers to these users, which had previously been
levelled with the use of screen reading technologies.
3) Best practices and technical recommendations: We used
our findings to inform best practices when using emoji. These
should be considered by all users who post content that could
be read by visually impaired people, and is important for pub-
lic bodies and companies who could be in breach of equality
laws [1, 2]. We introduce recommendations to improve the
accessibility of emoji for visually impaired people. These
should be considered by all organisations involved in emoji,
such as the Unicode Consortium, platform developers (e.g.,
Apple, Google), social media companies (e.g., Twitter) and
developers of screen readers and other assistive technology.
Limitations
Fifty-eight visually impaired people completed our survey, and
while accessibility research recruitment is a challenge [59], our
results may not generalise to the wider population. We focused
on visual impairment challenges, yet some challenges may be
from cultural differences [34, 41]. However, our survey was
in English and thus the majority of our participants may be
from English speaking countries, potentially limiting cultural
influences of misunderstanding. Further exploring cultural
differences and emoji accessibility would be interesting.
There is a limitation due to respondents’ ability to accurately
recall their experiences of using emoji. However, the majority
of our participants used (58%) and encountered (77%) emoji
daily, so the impact of this on our findings may be minimal.
Our survey participants reported a wide variety of visual im-
pairments. However, only ~34% of participants described their
visual acuity to be worse than "None or Mild" (i.e., worse than
20/70). This could be explained by the number of visual im-
pairments reported by participants, along with the number of
participants with colour vision deficiency within our sample.
We did not ask about participants’ social media usage since
emoji are found in many different platforms of communication.
Our findings support this reasoning since emoji issues raised
included but were not limited to social media.
We had 11 participants take part in our interviews. Due to this
sample size our results may not generalise to all screen reader
users. We were unable to obtain a gender balanced group, yet
it is well known that recruiting participants for accessibility re-
search is a challenge [59]. We also cannot determine who may
have took part in the prior survey since it was kept anonymous.
Participants took part in our interviews over messaging tools
using assistive technology. This could have limited partici-
pants’ expression versus an in-person interview, although this
is unlikely to significantly affect the quality of the data [16].
There may also be a self-selection bias for participants who
were more confident using assistive tech.
Generalisations & Future Work
We focused on understanding challenges faced by visually im-
paired people due to emoji. Some of these challenges may also
apply to users with typical vision as a result of situationally-
induced impairments and disabilities (SIIDs) [66]. For exam-
ple, SIIDs caused by screen glare or low screen brightness may
result in challenges with selecting emoji, due to their small
dimensions. In addition, this is likely to be an issue on smart-
watches and on augmented reality screens. The use of Voice
Assistants (VA), such as Amazon Alexa (alexa.amazon.com)
and car-based support such as CarPlay (apple.com/carplay) are
now commonplace. VAs announce content, and thus need to
read emoji descriptors in similar ways to screen readers. This
may introduce similar challenges, such as misunderstandings
of emotive intent, for users in different contexts.
CONCLUSION
Emoji are prevalent within communication, however the acces-
sibility challenges they introduce are not well understood. We
conducted an online questionnaire with 58 visually impaired
participants to explore their experiences. We found that emoji
introduce several specific challenges for screen reader users,
notably around social communication. We then conducted
11 interviews with screen-reader users to further understand
the challenges they face when encountering emoji. Partici-
pants raised issues that ultimately resulted in social exclusion.
Considering the challenges we have identified, the quality of
communication will continue to diminish as new and more
complex emoji are released. To address this, people should
consider our best practices when using emoji, and vendors
and organisations should consider our recommendations when
determining the future direction of emoji.
ACKNOWLEDGEMENTS
We thank our participants for taking part in this research, and
to Erin Brady for supporting early stages of this work. This
research was awarded funding via the Bournemouth University
ACORN (Acceleration Of Research & Networking) Fund.
REFERENCES
[1] 2000. Section 508. Government of the United States of
America. access-board.gov/guidelines-and-
standards/communications-and-it/about-the-section-
508-standards/section-508-standards Accessed:
2019-09-17.
[2] 2010. The Equality Act. London: The Stationary Office.
legislation.gov.uk/ukpga/2010/15/contents Accessed:
2019-09-17.
[3] 2019. Unicode® Emoji Chart Format.
https://unicode.org/emoji/format.html#col-name.
(2019). Accessed: 2019-03-22.
[4] Jessalyn Alvina, Carla F. Griggio, Xiaojun Bi, and
Wendy E. Mackay. 2017. CommandBoard: Creating a
General-Purpose Command Gesture Input Space for
Soft Keyboard. In Proceedings of the 30th Annual ACM
Symposium on User Interface Software and Technology
(UIST ’17). Association for Computing Machinery, New
York, NY, USA, 17–28. DOI:
http://dx.doi.org/10.1145/3126594.3126639
[5] David Armstrong, Ann Gosling, John Weinman, and
Theresa Marteau. 1997. The place of inter-rater
reliability in qualitative research: an empirical study.
Sociology 31, 3 (1997), 597–606. DOI:
http://dx.doi.org/10.1177/0038038597031003015
[6] Shiri Azenkot, Cynthia L. Bennett, and Richard E.
Ladner. 2013. DigiTaps: Eyes-free Number Entry on
Touchscreens with Minimal Audio Feedback. In
Proceedings of the 26th Annual ACM Symposium on
User Interface Software and Technology (UIST ’13).
ACM, New York, NY, USA, 85–90. DOI:
http://dx.doi.org/10.1145/2501988.2502056
[7]
Syed Masum Billah, Vikas Ashok, Donald E. Porter, and
I.V. Ramakrishnan. 2017. Ubiquitous Accessibility for
People with Visual Impairments: Are We There Yet?. In
Proceedings of the 2017 CHI Conference on Human
Factors in Computing Systems (CHI ’17). ACM, New
York, NY, USA, 5862–5868. DOI:
http://dx.doi.org/10.1145/3025453.3025731
[8]
Erin Brady, Meredith Ringel Morris, Yu Zhong, Samuel
White, and Jeffrey P. Bigham. 2013. Visual Challenges
in the Everyday Lives of Blind People. In Proceedings
of the SIGCHI Conference on Human Factors in
Computing Systems (CHI ’13). ACM, New York, NY,
USA, 2117–2126. DOI:
http://dx.doi.org/10.1145/2470654.2481291
[9] Virginia Braun and Victoria Clarke. 2006. Using
thematic analysis in psychology. Qualitative research in
psychology 3, 2 (2006), 77–101. DOI:
http://dx.doi.org/10.1191/1478088706qp063oa
[10] Maria Claudia Buzzi, Marina Buzzi, and Barbara
Leporini. 2011. Web 2.0: Twitter and the Blind. In
Proceedings of the 9th ACM SIGCHI Italian Chapter
International Conference on Computer-Human
Interaction: Facing Complexity (CHItaly). ACM, New
York, NY, USA, 151–156. DOI:
http://dx.doi.org/10.1145/2037296.2037333
[11] Yuri Choi, Kyung Hoon Hyun, and Ji-Hyun Lee. 2017.
Image-Based Tactile Emojis: Improved Interpretation of
Message Intention and Subtle Nuance for Visually
Impaired Individuals. Human–Computer Interaction 0, 0
(2017), 1–30. DOI:
http://dx.doi.org/10.1080/07370024.2017.1324305
[12] Moodle Education Claudia Dent. 2017. Enhance your
online communications through more support for emoji
characters in Moodle 3.3.
https://moodle.com/news/enhance-online-
communications-support-emoji-characters-moodle-3-3.
(2017). Accessed: 2019-09-04.
[13] Barry L Cole. 2004. The handicap of abnormal colour
vision. Clinical and Experimental Optometry 87, 4-5
(2004), 258–275.
[14] Henriette Cramer, Paloma de Juan, and Joel Tetreault.
2016. Sender-intended Functions of Emojis in US
Messaging. In Proceedings of the 18th International
Conference on Human-Computer Interaction with
Mobile Devices and Services (MobileHCI ’16). ACM,
New York, NY, USA, 504–509. DOI:
http://dx.doi.org/10.1145/2935334.2935370
[15] Daantje Derks, Arjan ER Bos, and Jasper
Von Grumbkow. 2007. Emoticons and social interaction
on the Internet: the importance of social context.
Comput hum behav 23, 1 (2007), 842–849.
[16] Jill P. Dimond, Casey Fiesler, Betsy DiSalvo, Jon Pelc,
and Amy S. Bruckman. 2012. Qualitative Data
Collection Technologies: A Comparison of Instant
Messaging, Email, and Phone. In Proceedings of the
17th ACM International Conference on Supporting
Group Work (GROUP ’12). ACM, New York, NY, USA,
277–280. DOI:
http://dx.doi.org/10.1145/2389176.2389218
[17] Abdallah El Ali, Torben Wallbaum, Merlin Wasmann,
Wilko Heuten, and Susanne CJ Boll. 2017. Face2Emoji:
Using Facial Emotional Expressions to Filter Emojis. In
Proceedings of the 2017 CHI Conference Extended
Abstracts on Human Factors in Computing Systems
(CHI EA ’17). ACM, New York, NY, USA, 1577–1584.
DOI:http://dx.doi.org/10.1145/3027063.3053086
[18] emojipedia. 2017. 5 Billion Emojis Sent Daily on
Messenger. https://blog.emojipedia.org/5-billion-
emojis-sent-daily-on-messenger/. (2017). Accessed:
2019-08-28.
[19] Instagram Engineering. 2015. Emojineering Part 1:
Machine Learning for Emoji Trends.
https://instagram-engineering.com/emojineering-part-1-
machine-learning-for-emoji-trendsmachine-learning-
for-emoji-trends-7f5f9cb979ad. (2015). Accessed:
2019-08-28.
[20] World Health Organisation (WHO) Programme for the
Prevention of Blindness and Deafness. 2003.
Consultation on development of standards for
characterization of vision loss and visual functioning.
https://apps.who.int/iris/handle/10665/68601. (2003).
Accessed: 2019-09-03.
[21] Justin Garrity. 2016. How Custom Emojis Crowned
Twitter the Hashtag Registrar.
https://medium.com/@justinogarrity/how-custom-
emojis-crowned-twitter-the-hashtag-registrar-
43e5fa60c557. (2016). Accessed:
2019-09-04.
[22] Cole Gleason, Patrick Carrington, Cameron Cassidy,
Meredith Ringel Morris, Kris M. Kitani, and Jeffrey P.
Bigham. 2019. “It’s Almost Like They’re Trying to Hide
It”: How User-Provided Image Descriptions Have Failed
to Make Twitter Accessible. In The World Wide Web
Conference (WWW ’19). ACM, New York, NY, USA,
549–559. DOI:
http://dx.doi.org/10.1145/3308558.3313605
[23] Eric Goldman. 2018. Emojis and the Law. Wash. L. Rev.
93 (2018), 1227.
[24] Benjamin M. Gorman and David R. Flatla. 2017. A
Framework for Speechreading Acquisition Tools. In
Proceedings of the 2017 CHI Conference on Human
Factors in Computing Systems (CHI ’17). ACM, New
York, NY, USA, 519–530. DOI:
http://dx.doi.org/10.1145/3025453.3025560
[25]
Tim Highfield and Tama Leaver. 2016. Instagrammatics
and digital methods: Studying visual social media, from
selfies and GIFs to memes and emoji. Communication
Research and Practice 2, 1 (2016), 47–62.
[26]
Albert H Huang, David C Yen, and Xiaoni Zhang. 2008.
Exploring the potential effects of emoticons. Inform
Manage 45, 7 (2008), 466–473.
[27] Unicode Inc. 2019a. Emoji Data for UTS.
https://www.unicode.org/Public/emoji/12.0/emoji-
data.txt. (2019). Accessed:
2019-08-28.
[28] Unicode Inc. 2019b. Unicode® Full Emoji List, v12.0. .
(2019). Accessed: 2019-09-18.
[29]
Unicode Inc. 2019c. Unicode
®
Technical Standard #51 -
Design Guidelines.
unicode.org/reports/tr51/#Design_Guidelines. (2019).
Accessed: 2019-09-18.
[30]
(UTC) Unicode Inc. 2019d. Working Draft for Proposed
Update Unicode® Technical Standard #51.
unicode.org/L2/L2019/19203r2-wd-uts51-17-
draft.pdf#page=31. (2019). Accessed:
2019-09-18.
[31] Simon Keates John Clarkson, Roger Coleman and
Cherie Lebbon. 2003. Inclusive Design: Designing for
the Whole Population. Springer-Verlag London. DOI:
http://dx.doi.org/10.1007/978-1- 4471-0001- 0
[32] Sigrid Kelsey and Kirk St. Amant. 2008. Handbook of
Research on Computer Mediated Communication. IGI
Global, Hershey, PA.
[33] Adam Kendon. 1967. Some functions of gaze-direction
in social interaction. Acta Psychologica 26 (1967), 22 –
63. DOI:http://dx.doi.org/https:
//doi.org/10.1016/0001-6918(67)90005- 4
[34] Philippe Kimura-Thollander and Neha Kumar. 2019.
Examining the “Global” Language of Emojis:
Designing for Cultural Representation. In Proceedings
of the 2019 CHI Conference on Human Factors in
Computing Systems (CHI ’19). ACM, New York, NY,
USA, Article 495, 14 pages. DOI:
http://dx.doi.org/10.1145/3290605.3300725
[35] Elizabeth Kirley and Marilyn McMahon. 2017. The
emoji factor: Humanizing the emerging law of digital
speech. Tenn. L. Rev. 85 (2017), 517.
[36] Matthew Knott. 2015. Emoji diplomacy? DFAT asked
meaning behind Julie Bishop’s use of a red-faced emoji
to describe Vladimir Putin. Sydney Morning Herald,
https://www.smh.com.au/politics/federal/emoji-
diplomacy-dfat-asked-meaning-behind-julie-bishops-
use-of-a-redfaced-emoji-to-describe-vladimir-putin-
20151022-gkfgoc.html. (2015). Accessed:
2019-09-04.
[37] Justin Kruger, Nicholas Epley, Jason Parker, and
Zhi-Wen Ng. 2005. Egocentrism over e-mail: Can we
communicate as well as we think? J. Pers. Soc. Psychol.
89, 6 (2005), 925–936. DOI:
http://dx.doi.org/10.1037/0022-3514.89.6.925
[38]
Howard Lake. 2019. RNIB Aims to Redesign Emojis for
Visually Impaired Users.
fundraising.co.uk/2019/07/17/rnib-aims-to-redesign-
emojis-for-visually-impaired-users. (2019). Accessed:
2019-09-04.
[39] Weijian Li, Yuxiao Chen, Tianran Hu, and Jiebo Luo.
2018. Mining the Relationship between Emoji Usage
Patterns and Personality. In Twelfth International AAAI
Conference on Web and Social Media.
[40] Shao-Kang Lo. 2008. The nonverbal communication
functions of emoticons in computer-mediated
communication. CyberPsychology & Behavior 11, 5
(2008), 595–597.
[41] Xuan Lu, Wei Ai, Xuanzhe Liu, Qian Li, Ning Wang,
Gang Huang, and Qiaozhu Mei. 2016. Learning from
the Ubiquitous Language: An Empirical Analysis of
Emoji Usage of Smartphone Users. In Proceedings of
the 2016 ACM International Joint Conference on
Pervasive and Ubiquitous Computing (UbiComp ’16).
ACM, New York, NY, USA, 770–780. DOI:
http://dx.doi.org/10.1145/2971648.2971724
[42] David F Marks and Lucy Yardley. 2004. Research
methods for clinical and health psychology. Sage.
[43]
Nora McDonald, Sarita Schoenebeck, and Andrea Forte.
2019. Reliability and Inter-Rater Reliability in
Qualitative Research: Norms and Guidelines for CSCW
and HCI Practice. Proc. ACM Hum.-Comput. Interact. 3,
CSCW, Article Article 72 (Nov. 2019), 23 pages. DOI:
http://dx.doi.org/10.1145/3359174
[44] Hannah Miller, Daniel Kluver, Jacob Thebault-Spieker,
Loren Terveen, and Brent Hecht. 2017. Understanding
emoji ambiguity in context: The role of text in
emoji-related miscommunication. In Eleventh
International AAAI Conference on Web and Social
Media.
[45] Hannah Jean Miller, Jacob Thebault-Spieker, Shuo
Chang, Isaac Johnson, Loren Terveen, and Brent Hecht.
2016. “Blissfully Happy” or “Ready to Fight”: Varying
Interpretations of Emoji. In Tenth International AAAI
Conference on Web and Social Media.
[46] Hannah Miller Hillberg, Zachary Levonian, Daniel
Kluver, Loren Terveen, and Brent Hecht. 2018. What I
See is What You Don’T Get: The Effects of (Not)
Seeing Emoji Rendering Differences Across Platforms.
Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article
124 (Nov. 2018), 24 pages. DOI:
http://dx.doi.org/10.1145/3274393
[47] Sherwyn P Morreale, Brian H Spitzberg, and J Kevin
Barge. 2006. Human communication: Motivation,
knowledge, and skills. Cengage Learning.
[48] Meredith Ringel Morris, Jazette Johnson, Cynthia L.
Bennett, and Edward Cutrell. 2018. Rich
Representations of Visual Content for Screen Reader
Users. In Proceedings of the 2018 CHI Conference on
Human Factors in Computing Systems (CHI ’18). ACM,
New York, NY, USA, Article 59, 11 pages. DOI:
http://dx.doi.org/10.1145/3173574.3173633
[49] Meredith Ringel Morris, Annuska Zolyomi, Catherine
Yao, Sina Bahram, Jeffrey P. Bigham, and Shaun K.
Kane. 2016. "With Most of It Being Pictures Now, I
Rarely Use It": Understanding Twitter’s Evolving
Accessibility to Blind Users. In Proceedings of the 2016
CHI Conference on Human Factors in Computing
Systems (CHI ’16). ACM, New York, NY, USA,
5506–5516. DOI:
http://dx.doi.org/10.1145/2858036.2858116
[50] Leah Perlmutter, Bindita Chaudhuri, Justin Petelka,
Philip Garrison, James Fogarty, Jacob O. Wobbrock, and
Richard E. Ladner. 2019. Demonstration of GestureCalc:
An Eyes-Free Calculator for Touch Screens. In The 21st
International ACM SIGACCESS Conference on
Computers and Accessibility (ASSETS ’19). ACM, New
York, NY, USA, 667–669. DOI:
http://dx.doi.org/10.1145/3308561.3354595
[51] Katy Perry. n.d. https://twitter.com/katyperry. (n.d.).
Accessed: 2019-09-04.
[52] Henning Pohl, Christian Domin, and Michael Rohs.
2017. Beyond Just Text: Semantic Emoji Similarity
Modeling to Support Expressive Communication. ACM
Trans. Comput.-Hum. Interact. 24, 1, Article 6 (March
2017), 42 pages. DOI:
http://dx.doi.org/10.1145/3039685
[53] Henning Pohl, Dennis Stanke, and Michael Rohs. 2016.
EmojiZoom: Emoji Entry via Large Overview Maps. In
Proceedings of the 18th International Conference on
Human-Computer Interaction with Mobile Devices and
Services (MobileHCI ’16). ACM, New York, NY, USA,
510–517. DOI:
http://dx.doi.org/10.1145/2935334.2935382
[54] National Rail. n.d. https://twitter.com/nationalrailenq.
(n.d.). Accessed: 2019-09-16.
[55] Maya Rhodan. 2014. President Obama Is Reaching Out
to Millennials About the Economy Using...Emoji.
https://time.com/3484876/obama-emoji/. (2014).
Accessed: 2019-09-04.
[56] David Rodrigues, Diniz Lopes, Marília Prada, Dominic
Thompson, and Margarida V. Garrido. 2017. A frown
emoji can be worth a thousand words: Perceptions of
emoji use in text messages exchanged between romantic
partners. Telematics and Informatics 34, 8 (2017), 1532 –
1543. DOI:http://dx.doi.org/https:
//doi.org/10.1016/j.tele.2017.07.001
[57] Adrian Roselli. 2016. Accessible Emoji, Tweaked.
http://adrianroselli.com/2016/12/accessible-emoji-
tweaked.html. (2016). Accessed:
2019-03-18.
[58] Morgan Klaus Scheuerman. 2019. HCI Guidelines for
Gender Equality and Inclusivity; updated 5/13/2019.
https://www.morgan-klaus.com/sigchi-gender-
guidelines. (2019). Accessed:
2019-20-09.
[59]
Andrew Sears and Vicki L. Hanson. 2012. Representing
Users in Accessibility Research. ACM Trans. Access.
Comput. 4, 2, Article Article 7 (March 2012), 6 pages.
DOI:http://dx.doi.org/10.1145/2141943.2141945
[60] Katy Steinmetz. 2015. Oxford’s 2015 Word of the Year
Is This Emoji. TIME, https://time.com/4114886/oxford-
word-of-the-year-2015-emoji/. (2015). Accessed:
2019-09-04.
[61]
Gerhard Strube. 1987. Answering survey questions: The
role of memory. In Social information processing and
survey methodology. Springer, 86–101.
[62] Selina Sutton and Shaun Lawson. 2017. A Provocation
for Rethinking and Democratising Emoji Design. In
Proceedings of the 2017 ACM Conference Companion
Publication on Designing Interactive Systems (DIS ’17
Companion). ACM, New York, NY, USA, 7–12. DOI:
http://dx.doi.org/10.1145/3064857.3079109
[63] Sarit Felicia Anais Szpiro, Shafeka Hashash, Yuhang
Zhao, and Shiri Azenkot. 2016. How People with Low
Vision Access Computing Devices: Understanding
Challenges and Opportunities. In Proceedings of the
18th International ACM SIGACCESS Conference on
Computers and Accessibility (ASSETS ’16). ACM, New
York, NY, USA, 171–180. DOI:
http://dx.doi.org/10.1145/2982142.2982168
[64]
Garreth W. Tigwell and David R. Flatla. 2016. Oh That’s
What You Meant!: Reducing Emoji Misunderstanding.
In Proceedings of the 18th International Conference on
Human-Computer Interaction with Mobile Devices and
Services Adjunct (MobileHCI ’16). ACM, New York,
NY, USA, 859–866. DOI:
http://dx.doi.org/10.1145/2957265.2961844
[65] Garreth W. Tigwell, David R. Flatla, and Rachel
Menzies. 2018. It’s Not Just the Light: Understanding
the Factors Causing Situational Visual Impairments
During Mobile Interaction. In Proceedings of the 10th
Nordic Conference on Human-Computer Interaction
(NordiCHI ’18). ACM, New York, NY, USA, 338–351.
DOI:http://dx.doi.org/10.1145/3240167.3240207
[66]
Garreth W. Tigwell, Zhanna Sarsenbayeva, Benjamin M.
Gorman, David R. Flatla, Jorge Goncalves, Yeliz
Yesilada, and Jacob O. Wobbrock. 2019. Addressing the
Challenges of Situationally-Induced Impairments and
Disabilities in Mobile Interaction. In Extended Abstracts
of the 2019 CHI Conference on Human Factors in
Computing Systems (CHI EA ’19). ACM, New York, NY,
USA, Article W30, 8 pages. DOI:
http://dx.doi.org/10.1145/3290607.3299029
[67] Sarah J. Tracy. 2013. Qualitative Research Methods:
Collecting Evidence, Crafting Analysis, Communicating
Impact. Wiley-Blackwell.
[68] Unicode. 2019. Members - Unicode.
https://home.unicode.org/membership/members/.
(2019). Accessed: 2019-09-18.
[69] Veroniiiica. 2018. How Do People with Vision
Impairments...Use Emoji?
http://www.perkinselearning.org/technology/blog/how-
do-people-vision-impairments-use-emoji. (2018).
Accessed: 2019-03-18.
[70] Joseph B Walther and Kyle P D’Addario. 2001. The
impacts of emoticons on message interpretation in
computer-mediated communication. Social science
computer review 19, 3 (2001), 324–347.
[71] Léonie Watson. 2016. Accessible emoji.
https://tink.uk/accessible-emoji/. (2016). Accessed:
2019-03-18.
[72] Brian Wentz, Harry Hochheiser, and Jonathan Lazar.
2013. A survey of blind users on the usability of email
applications. Universal Access in the Information
Society 12, 3 (01 Aug 2013), 327–336. DOI:
http://dx.doi.org/10.1007/s10209-012- 0285-9
[73]
World Health Organisation (WHO). 2018. Blindness and
vision impairment. https://www.who.int/news-room/fact-
sheets/detail/blindness-and-visual-impairment. (2018).
Accessed: 2019-09-04.
[74] Sarah Wiseman and Sandy J. J. Gould. 2018.
Repurposing Emoji for Personalised Communication:
Why Pizza Slice Emoji; Means “I Love You”. In
Proceedings of the 2018 CHI Conference on Human
Factors in Computing Systems (CHI ’18). ACM, New
York, NY, USA, Article 152, 10 pages. DOI:
http://dx.doi.org/10.1145/3173574.3173726
[75] Shaomei Wu and Lada A. Adamic. 2014. Visually
Impaired Users on an Online Social Network. In
Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (CHI ’14). ACM, New
York, NY, USA, 3133–3142. DOI:
http://dx.doi.org/10.1145/2556288.2557415
[76] Peijun Zhao, Jia Jia, Yongsheng An, Jie Liang, Lexing
Xie, and Jiebo Luo. 2018. Analyzing and Predicting
Emoji Usages in Social Media. In Companion
Proceedings of the The Web Conference 2018 (WWW
’18). International World Wide Web Conferences
Steering Committee, Republic and Canton of Geneva,
Switzerland, 327–334. DOI:
http://dx.doi.org/10.1145/3184558.3186344
[77] Rui Zhou, Jasmine Hentschel, and Neha Kumar. 2017.
Goodbye Text, Hello Emoji: Mobile Communication on
WeChat in China. In Proceedings of the 2017 CHI
Conference on Human Factors in Computing Systems
(CHI ’17). ACM, New York, NY, USA, 748–759. DOI:
http://dx.doi.org/10.1145/3025453.3025800
... Furthermore, if we focus on emoji, which are an extremely popular mode of non-textual communication and the successor to emoticons, people often report that they try to avoid using emoji in work environments and with people who do not share close bonds. As people have varied interpretations of emoji and they consider it a non-serious and unprofessional way of communication [5,41,63,78,79]. ...
... These drawbacks afect virtual team communication by creating confusion, which leads to more conficts, and adversely impacts shared interpersonal relationships in a virtual workspace [26]. Messages can be supplemented with non-textual responses such as emoji, GIF, stickers, and memes to overcome misinterpretations, communicate efectively, and increase social connections in virtual settings [59,63,75,79], but the non-textual responses are typically viewed as less appropriate to use in formal settings [20,78,79]. We suspect that new collaborators-students or professionals who join a new team virtually, with no familiarity to the team-likely face challenges and tensions with using non-textual communication to connect with colleagues, and there may be design opportunities on various platforms to support an efcient way for new collaborators to use non-textual responses in virtual workspaces. ...
... These drawbacks afect virtual team communication by creating confusion, which leads to more conficts, and adversely impacts shared interpersonal relationships in a virtual workspace [26]. Messages can be supplemented with non-textual responses such as emoji, GIF, stickers, and memes to overcome misinterpretations, communicate efectively, and increase social connections in virtual settings [59,63,75,79], but the non-textual responses are typically viewed as less appropriate to use in formal settings [20,78,79]. We suspect that new collaborators-students or professionals who join a new team virtually, with no familiarity to the team-likely face challenges and tensions with using non-textual communication to connect with colleagues, and there may be design opportunities on various platforms to support an efcient way for new collaborators to use non-textual responses in virtual workspaces. ...
Conference Paper
Full-text available
Virtual workspaces rapidly increased during the COVID-19 pandemic, and for many new collaborators, working remotely was their first introduction to their colleagues. Building rapport is essential for a healthy work environment, and while this can be achieved through non-textual responses within chat-based systems (e.g., emoji, GIF, stickers, memes), those non-textual responses are typically associated with personal relationships and informal settings. We studied the experiences of new collaborators (questionnaire N=49; interview N=14) in using non-textual responses to communicate with unacquainted teams and the effect of non-textual responses on new collaborators’ interpersonal bonds. We found new collaborators selectively and progressively use non-textual responses to establish interpersonal bonds. Moreover, the use of non-textual responses has exposed several limitations when used on various platforms. We conclude with design recommendations such as expanding the scope of interpretable non-textual responses and reducing selection time.
... Tigwell et al. [4] conducted a survey to find out the problems faced by people with visual impairments in relation to emojis. Their findings highlight the issue that assistive technologies, instead of facilitating the inclusion of emojis in CMC, may complicate it, thus being an obstacle to social inclusion. ...
... A good example of such ambiguity is given by the emoji whose Unicode name is "Face with Look of Triumph", which was originally meant to convey positive emotions associated with pride and personal satisfaction [4]. In spite of its Unicode definition and original meaning, the emoji is currently used on many platforms to suggest negative feelings, such as frustration, anger or contempt. ...
Chapter
We present an emoji picker designed to enrich emojis selection on mobile devices using audio cues. The aim is to make emojis selection more intuitive by better identify their meanings. Unlike the typical emoji input components currently in use (known as “pickers”), in our component each emotion-related item is represented by both an emoji and a non-verbal vocal cue, and it is displayed according to a two-dimensional model suggesting the pleasantness and intensity of the emotion itself. The component was embedded in an Android app in order to exploit touchscreen interaction together with audio cues to ease the selection process by using more than one channel (visual and auditory). Since the component adds non-visual information that drives the emoji selection, it may be particularly useful for users with visual impairments. In order to investigate the feasibility of the approach and the acceptability/usability of the emoji picker component, a preliminary remote evaluation test involving both sighted and visually impaired users was performed. Analysis of the data collected through the evaluation test shows that all the participants, whether sighted or visually impaired, rated the usability of our picker as good, and also evaluated positively the model adopted to add semantic value to emojis.
... Similarly, the airbags of In-Flat can also be used as Braille outputs, as well as to input Braille characters. In-Flat can also be an alternative channel to support emoji accessibility for visually-impaired users, as they often struggle to perceive emojis [221]. ...
Thesis
As one of the most important non-verbal communication channels, touch is widely used for different purposes. It is a powerful force in human physical and psychological development, shaping social structures as well as communicating emotions. However, even though current information and communication technology (ICT) systems enable the use of various non-verbal languages, the support of communicating through the sense of touch is still insufficient. Inspired by the cross-modal interaction of human perception, the approach I present in this dissertation is to use multimodality to improve mediated touch interaction. Following this approach, I present three devices that provide empirical contributions to multimodal touch interaction: VisualTouch, SansTouch, and In-Flat. To understand if multimodal stimuli can improve the emotional perception of touch, I present the VisualTouch device, and quantitatively evaluate the cross-modal interaction between the visual and tactile modality. To investigate the use of different modalities in real touch communication, I present the SansTouch device, which provides empirical insights on multimodal interaction and skin-like touch generation in the context of face-to-face communication. Going one step forward in the use of multimodal stimuli in touch interaction, I present the In-Flat device, an input/output touch overlay for smartphones. In-Flat not only provides further insights on the skin-like touch generation, but also a better understanding of the role that mediated touch plays in more general contexts. In summary, this dissertation strives to bridge the gap between touch communication and HCI, by contributing to the design and understanding of multimodal stimuli in mediated touch interaction.
... Voicemoji [56] explored voice-based emoji entry for visually impaired users. A few other studies also tackled challenges related to the accessibility and inclusiveness of emoticons [31,48]. Increased attention has been paid to user customization of emoticons [19], for example, in generating new emojis based on users' sketch and text input [35], or allowing two intimate users to co-customize their emoticon shortcuts (DearBoard) [20]. ...
Preprint
Full-text available
Emoticons are indispensable in online communications. With users' growing needs for more customized and expressive emoticons, recent messaging applications begin to support (limited) multi-modal emoticons: e.g., enhancing emoticons with animations or vibrotactile feedback. However, little empirical knowledge has been accumulated concerning how people create, share and experience multi-modal emoticons in everyday communication, and how to better support them through design. To tackle this, we developed VibEmoji, a user-authoring multi-modal emoticon interface for mobile messaging. Extending existing designs, VibEmoji grants users greater flexibility to combine various emoticons, vibrations, and animations on-the-fly, and offers non-aggressive recommendations based on these components' emotional relevance. Using VibEmoji as a probe, we conducted a four-week field study with 20 participants, to gain new understandings from in-the-wild usage and experience, and extract implications for design. We thereby contribute both a novel system and various insights for supporting users' creation and communication of multi-modal emoticons (to be published at ACM CHI '22).
... Closed-ended questions are reported by frequency of responses. Open-ended questions were analysed independently using open coding [63], based on existing procedure [61,62]. We used the following four-step process: ...
Preprint
Full-text available
Subtitles can help improve the understanding of media content. People enable subtitles based on individual characteristics (e.g., language or hearing ability), viewing environment, or media context (e.g., drama, quiz show). However, some people find that subtitles can be distracting and that they negatively impact their viewing experience. We explore the challenges and opportunities surrounding interaction with real-time personalisation of subtitled content. To understand how people currently interact with subtitles, we first conducted an online questionnaire with 102 participants. We used our findings to elicit requirements for a new approach called Adaptive Subtitles that allows the viewer to alter which speakers have subtitles displayed in real-time. We evaluated our approach with 19 participants to understand the interaction trade-offs and challenges within real-time adaptations of subtitled media. Our evaluation findings suggest that granular controls and structured onboarding allow viewers to make informed trade-offs when adapting media content, leading to improved viewing experiences.
Article
Visualizations are now widely used across disciplines to understand and communicate data. The benefit of visualizations lies in leveraging our natural visual perception. However, the sole dependency on vision can produce unintended discrimination against people with visual impairments. While the visualization field has seen enormous growth in recent years, supporting people with disabilities is much less explored. In this work, we examine approaches to support this marginalized user group, focusing on visual disabilities. We collected and analyzed papers published for the last 20 years on visualization accessibility. We mapped a design space for accessible visualization that includes seven dimensions: user group, literacy task, chart type, interaction, information granularity, sensory modality, assistive technology. We described the current knowledge gap in light of the latest advances in visualization and presented a preliminary accessibility model by synthesizing findings from existing research. Finally, we reflected on the dimensions and discussed opportunities and challenges for future research.
Conference Paper
Full-text available
Situationally-induced impairments and disabilities (SIIDs) make it difficult for users of interactive computing systems to perform tasks due to context (e.g., listening to a phone call when in a noisy crowd) rather than a result of a congenital or acquired impairment (e.g., hearing damage). SIIDs are a great concern when considering the ubiquitousness of technology in a wide range of contexts. Considering our daily reliance on technology, and mobile technology in particular, it is increasingly important that we fully understand and model how SIIDs occur. Similarly, we must identify appropriate methods for sensing and adapting technology to reduce the effects of SIIDs. In this workshop, we will bring together researchers working on understanding, sensing, modelling, and adapting technologies to ameliorate the effects of SIIDs. This workshop will provide a venue to identify existing research gaps, new directions for future research, and opportunities for future collaboration.
Conference Paper
Full-text available
Mobile technologies are used in increasingly diverse and challenging environments. With the predominantly visual nature of mobile devices, Situational Visual Impairments (SVIs) are a growing concern. However, fundamental knowledge is lacking about the causes of SVIs, how people deal with SVIs, and whether their solutions are effective. To address this, we first conducted a convenience-sampled online questionnaire with 174 participants, and identified many causes and (ineffective) solutions. To firmly ground our initial results, we then conducted a two-week ecological momentary assessment with 24 participants, balanced by age and gender across Australia and Scotland. We confirmed that SVIs are experienced often and during typical mobile tasks, and can be very frustrating. We identify a range of factors causing SVIs, discuss mobile design implications, and introduce an SVI Context Model rooted in empirical evidence. The contributions in this paper will support the development of new effective SVI solutions.
Article
What does reliability mean for building a grounded theory? What about when writing an auto-ethnography? When is it appropriate to use measures like inter-rater reliability (IRR)? Reliability is a familiar concept in traditional scientific practice, but how, and even whether to establish reliability in qualitative research is an oft-debated question. For researchers in highly interdisciplinary fields like computer-supported cooperative work (CSCW) and human-computer interaction (HCI), the question is particularly complex as collaborators bring diverse epistemologies and training to their research. In this article, we use two approaches to understand reliability in qualitative research. We first investigate and describe local norms in the CSCW and HCI literature, then we combine examples from these findings with guidelines from methods literature to help researchers answer questions like: "should I calculate IRR?" Drawing on a meta-analysis of a representative sample of CSCW and HCI papers from 2016-2018, we find that authors use a variety of approaches to communicate reliability; notably, IRR is rare, occurring in around 1/9 of qualitative papers. We reflect on current practices and propose guidelines for reporting on reliability in qualitative research using IRR as a central example of a form of agreement. The guidelines are designed to generate discussion and orient new CSCW and HCI scholars and reviewers to reliability in qualitative research.
Conference Paper
Keypad-based character input in existing digital calculator applications on touch screen devices requires precise, targeted key presses that are time-consuming and error-prone for many screen reader users. We demonstrate GestureCalc, a digital calculator that uses target-free gestures for arithmetic tasks. It allows eyes-free target-less input of digits and operations through taps and directional swipes with one to three fingers, guided by minimal audio feedback. A study of the effectiveness of GestureCalc for screen reader users appears in a full paper by the authors at this conference.
Conference Paper
To make images on Twitter and other social media platforms accessible to screen reader users, image descriptions (alternative text) need to be added that describe the information contained within the image. The lack of alternative text has been an enduring accessibility problem since the “alt” attribute was added in HTML 2.0 over 20 years ago, and the rise of user-generated content has only increased the number of images shared. As of 2016, Twitter provides users the ability to turn on a feature that allows descriptions to be added to images in their tweets, presumably in an effort to combat this accessibility problem. What has remained unknown is whether simply enabling users to provide alternative text has an impact on experienced accessibility. In this paper, we present a study of 1.09 million tweets with images, finding that only 0.1% of those tweets included descriptions. In a separate analysis of the timelines of 94 blind Twitter users, we found that these image tweets included descriptions more often. Even users with the feature turned on only write descriptions for about half of the images they tweet. To better understand why users provide alternative text descriptions (or not), we interviewed 20 Twitter users who have written image descriptions. Users did not remember to add alternative text, did not have time to add it, or did not know what to include when writing the descriptions. Our findings indicate that simply making it possible to provide image descriptions is not enough, and reveal future directions for automated tools that may support users in writing high-quality descriptions.
Conference Paper
Emojis are becoming an increasingly popular mode of communication between individuals worldwide, with researchers claiming them to be a type of "ubiquitous language'' that can span different languages due to its pictorial nature. Our study uses a combination of methods to examine how emojis are adopted and perceived by individuals from diverse cultural backgrounds and 45 countries. Our survey and interview findings point to the existence of a cultural gap between user perceptions and the current emoji standard. Using participatory design, we sought to address this gap by designing 40 emojis and conducted another survey to evaluate their acceptability compared to existing Japanese emojis. We also draw on participant observation from a Unicode Consortium meeting on emoji addition. Our analysis leads us to discuss how emojis might be made more inclusive, diverse, and representative of the populations that use them.
Article
Emojis are an increasingly important way we express ourselves. Though emojis may be cute and fun, their usage can lead to misunderstandings with significant legal stakes- such as whether someone should be obligated by contract, liable for sexual harassment, or sent to jail. Our legal system has substantial experience interpreting new forms of content, so it should be equipped to handle emojis. Nevertheless, some special attributes of emojis create extra interpretative challenges. This Article identifies those attributes and proposes how courts should handle them. One particularly troublesome interpretative challenge arises from the different ways platforms depict emojis that are nominally standardized through the Unicode Consortium. These differences can unexpectedly create misunderstandings. The diversity of emoji depictions is not technologically required, nor does it necessarily benefit users. Instead, it likely reflects platforms’ concerns about intellectual property protection for emojis, which forces them to introduce unnecessary variations that create avoidable confusion. Thus, intellectual property may be hindering our ability to communicate with each other. This Article will discuss how to limit this unwanted consequence. © 2018, University of Washington School of Law. All rights reserved.
Article
Emoji are popular in digital communication, but they are rendered differently on different viewing platforms (e.g., iOS, Android). It is unknown how many people are aware that emoji have multiple renderings, or whether they would change their emoji-bearing messages if they could see how these messages render on recipients' devices. We developed software to expose the multi-rendering nature of emoji and explored whether this increased visibility would affect how people communicate with emoji. Through a survey of 710 Twitter users who recently posted an emoji-bearing tweet, we found that at least 25% of respondents were unaware that the emoji they posted could appear differently to their followers. Additionally, after being shown how one of their tweets rendered across platforms, 20% of respondents reported that they would have edited or not sent the tweet. These statistics reflect millions of potentially regretful tweets shared per day because people cannot see emoji rendering differences across platforms. Our results motivate the development of tools that increase the visibility of emoji rendering differences across platforms, and we contribute our cross-platform emoji rendering software to facilitate this effort.
Conference Paper
Recent studies have found that people interpret emoji characters inconsistently, creating significant potential for miscommunication. However, this research examined emoji in isolation, without consideration of any surrounding text. Prior work has hypothesized that examining emoji in their natural textual contexts would substantially reduce potential for miscommunication. To investigate this hypothesis, we carried out a controlled study with 2,482 participants who interpreted emoji both in isolation and in multiple textual contexts. After comparing the variability of emoji interpretation in each condition, we found that our results do not support the hypothesis in prior work: when emoji are interpreted in textual contexts, the potential for miscommunication appears to be roughly the same. We also identify directions for future research to better understand the interplay between emoji and textual context.
Conference Paper
Alt text (short for "alternative text") is descriptive text associated with an image in HTML and other document formats. Screen reader technologies speak the alt text aloud to people who are visually impaired. Introduced with HTML 2.0 in 1995, the alt attribute has not evolved despite significant changes in technology over the past two decades. In light of the expanding volume, purpose, and importance of digital imagery, we reflect on how alt text could be supplemented to offer a richer experience of visual content to screen reader users. Our contributions include articulating the design space of representations of visual content for screen reader users, prototypes illustrating several points within this design space, and evaluations of several of these new image representations with people who are blind. We close by discussing the implications of our taxonomy, prototypes, and user study findings.