Conference PaperPDF Available

Towards a System for Converting Text to Sign Language in Macedonian

Authors:

Abstract

The paper presents initial results in the design and development of a system for automatic conversion of text to sign language in Macedonian. The system will be an essential part of a larger system for the automatic generation of Macedonian sign language based on text. This system will facilitate the digital inclusion and will ease communication with the Macedonian deaf and hearing impaired community. The system is implemented as a web application which allows input text to be encoded in the equivalent sequence of sign language signs. The initial results show an average sign error rate of 4.49%. Online testing was also organized that confirmed these promising results.
Towards a System for Converting Text to Sign
Language in Macedonian
Stefan Spasovski1, Branislav Gerazov1, Risto Chavdarov1, Viktorija Smilevska2, Aneta Crvenkovska,
Tomislav Kartalov1, Zoran Ivanovski1, and Toni Bachvarovski3
1Faculty of Electrical Engineering and Information Technologies
Ss Cyril and Methodius University in Skopje, Macedonia
2Elementary school ”Kuzman Josifovski - Pitu”, Skopje, Macedonia
3Association for Assistive Technologies “Open the Windows”, Skopje, Macedonia
stefanspasovski11@gmail.com, gerazov@feit.ukim.edu.mk
Abstract—The paper presents initial results in the design and
development of a system for automatic conversion of text to sign
language in Macedonian. The system will be an essential part
of a larger system for the automatic generation of Macedonian
sign language based on text. This system will facilitate the digital
inclusion and will ease communication with the Macedonian deaf
and hearing impaired community. The system is implemented as
a web application which allows input text to be encoded in the
equivalent sequence of sign language signs. The initial results
show an average sign error rate of 4.49%. Online testing was
also organized that confirmed these promising results.
Index Terms—natural language processing, assistive technol-
ogy, text-to-sign language, sign language, deafness
I. INTRODUCTION
Deafness is defined as a condition of extreme hearing loss,
i.e. having very little or no hearing at all. The American
Speech-Language-Hearing Association (ASHA) defines pro-
found hearing loss as only being able to hear sounds above
90 dB, with severe hearing loss ranging between 71 – 90 dB
[1]. Macedonian Law, places the threshold at 80 dB. The
estimated number of deaf and hearing impaired people in
Macedonia is around 6000 according to information from 2006
[2], which is 0.3% of Macedonia’s population [3]. This is
comparable to the percentage of deaf people in places like
the United States (0.38%) [4] and in Germany (0.28%) [5].
In today’s high-tech information-rich world, the digital
inclusion of people with disabilities is becoming increasingly
important. The deaf and hearing impaired are generally able
to directly use and interact with computers and smart devices,
and can follow traditional visual media such as TV and
newspapers. However, they find it hard to read text at speed.
This is especially true for those born deaf or hearing impaired,
as they have never heard the sounds of phonemes that phonetic
orthography transcribes with graphemes in written text. As
a consequence, for them phonetic transcription is not much
different from logograms, such as the Chinese characters used
to write Mandarin and other Asian languages. This problem
can be mitigated through offering live sign language (SL)
translation, but most TV broadcasters do not offer this service.
The problem is even more pronounced in Macedonia, where
there are only around 30 certified SL translators.1
One way to ease the digital inclusion of the deaf and hearing
impaired is through assistive systems able to automatically
convert text-to-sign language. One example of such a system
is the HandTalk App in which a virtual avatar named Hugo
converts text to sign language on the users smart device.2
These systems are made up of two essential parts: i) a text-
to-SL converter that transforms textual input to a sequence of
signs or gestures, and ii) a SL generator that uses the sequence
of signs to generate sign language, usually via 3D rendering of
an animated character, i.e. avatar. Sign language differs from
spoken language, in that it does not support inflection. For
example, to create the future tense in Macedonian we add the
future particle before the verb. Tense is not formed in that way
in sign language. Instead the speaker uses the infinitive form
of the verb together with signs like “later” to signify that they
are speaking about a future event. The same holds true for
verb conjugation, singular and plural, case and articles.
Although sign languages across the world do share signs,
there are still different standardized sign languages, such
as: American Sign Language (ASL), Italian Sign Language
(LIS), Indian Sign Language (ISL), Vietnamese Sign Lan-
guage (VSL), and Macedonian Sign Language (MSL). The
text-to-SL converter can encode the signs in various formats.
One famous format that dates back to 1984 is the Hamburg
Notation System for Sign Languages (HamNoSys), which
encodes signs through a set of pictograms or symbols [6].
As an extension to HamNoSys, the Signing Gesture Mark-up
Language (SiGML) describes the symbols using XML tags [7].
This extension allows the storage and use of the transcriptions
in computer based systems, such as 3D rendering software,
that can be used to generate sign language via an animated
avatar.
Text-to-SL systems have been developed for many of the
world languages, such as English [8], German [9], Vietnamese
1http://www.deafmkd.org.mk/lista-na-tolkuvaci/
2https://handtalk.me/en/app/
[10], Kurdish [11], Arabic [12], Brazilian Portuguese [13],
Punjabi [14], Korean [15] etc. Most systems rely on a simple
word-to-sign mapping, i.e. each word token from the input text
is looked up in a lexicon of signs, and if no match is found it is
spelled out using a sequence of alphabet letter signs [11]. More
advanced rule-based-systems map the input text grammar to
natural sign language grammar [15]. Recently, the application
of machine learning has allowed improved performance in
these systems, directly applying methods used in the area of
Machine Translation [10].
In Macedonia, work has mostly been done on the generation
of Macedonian Sign Language via virtual avatars. Koceski
and Koceska [16] developed and evaluated a 3D virtual tutor
for MSL. Joksimoski et al. [17] presented a 3D visualization
system that extensively uses animation and game concepts for
accurately generating sign languages using 3D avatars.
Here, we present a text-to-SL system that translates an input
Macedonian text into an output sequence of Macedonian Sign
Language signs. The system is built on a rule based algorithm,
which analyses input text, comparing the input word tokens to
a lexicon of some 200 signs. The performance of the system
is evaluated with a set of test sentences and the results show
a Sign Error Rate (SER) of 4.49%. We augment this analysis
with online testing of the system, which confirms the validity
of the initial results. The system can be used as the basis
for building a complete system for text based sign language
generation in Macedonian.
II. ALGORITHM
A. Sign mappings organization
We organize the data by placing the words and signs in five
different files:
list of signs,
list of names,
skip list,
dictionary of word to sign mappings, and
dictionary of phrases that map directly to sequences of
signs.
In the presented system we have 221 signs.
B. Text preprocessing
Text input is first normalised by converting all upper case
characters to lower case and removing all punctuation. We
then divide the text string into into a list of word tokens. We
initialise two empty lists for storing the sign sequence output
and unrecognized word tokens, and define a flag to be used
when a phrase has been recognized. With that we are ready
to begin the translation process.
C. Main loop
We go through the word tokens in the input list one by one
and run them through multiple checks. Firstly, we check if the
word token is part of a phrase by concatenating the succeeding
token from the list. If it is, then we append the sequence of
signs corresponding to that phrase, skip the second word from
the next iteration and move on from there. Next, if the token
Input text
Text normalization
Is it part of
a phrase
Is it in
skip list
Is it a name
Is it a sign
Is it in
dictionary
Similarity
check
Is it similar
to a sign
Is the root
similar to a
sign
Append sign
to output
Append input
to unknown list
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Main loop
Fig. 1. Block diagram of the algorithm for converting text to sign language
signs.
is in the skip list it is ignored and the next token is processed.
If not in the skip list, we check if the input word token is
part of the names or signs lists. If found the word token is
appended as is to the output sign sequence. If not, the token
is looked up in the dictionary of word to sign mappings, and
if found it’s sign mapping is appended to the output. If the
word still has not been found in any of the checks then we
continue with similarity checks.
D. Similarity checks
The final part of the main loop consists of similarity checks.
The system comprises two different similarity checks. They
are based on the “gestalt pattern matching” algorithm sug-
gested by Ratcliff and Obershelp in the 1980s [18]. The idea
being to find the longest contiguous matching subsequence
that contains no “junk” elements. This is applied recursively
to the pieces of the sequences to the left and to the right of
the matching subsequence. We use the implementation of the
algorithm in the difflib3Python library.
The similarity check outputs a list of 3 strings sorted
from the most likely to the list likely match. Based on out
experiments we get the best results when the cutoff is equal
to 0.7. One other problem that we encounter is the fact that
sometimes the most likely match (the first element of the
string) is not at all the most likely, and that the second or
third string in the list is the correct answer. We augment the
similarity search algorithm with a Character Error Rate (CER)
to select one of the three offered outputs:
CER =
S+D+I
N=
S+D+I
S+D+C(1)
where Sis the number of character substitutions, Dis the
number of character deletions, Iis the number of character
insertions, Cis the number of correct characters, and Nis the
total number of characters in the reference, i.e. N=S+D+C.
III. EXP ER IM EN TS
We used two experiments to evaluate the performance of
the proposed system. The first was based on an internal test
set, and the second was based on online testing.
A. Test set evaluation
We developed an internal test set comprising 124 sentences
for which we provided reference translations to sign language
sequences of signs. We took special care to have a varied test
set both in terms of sentence length as well as ample coverage
of the set of signs known to the system.
B. Online evaluation
To provide a platform for testing the proposed system, we
developed a web application that provides a user interface for
online testing. The web app was developed based on Flask4.
HTML was used to build the site layout, while Flask was used
to render the website, receive user input and return the results
of the translation process.
The website lets the user input words for translation. After
submitting the input the translation process starts and the
output from the proposed system including the output sign
sequence and the list of unrecognized word tokens. There is
also a text form that the user can use to give feedback. For
our online tests this was the correct sign sequence in case the
system returned an erroneous one. To improve coverage of the
known signs in the online tests, the signs known by the system
are listed at the end of the web page.
3https://docs.python.org/3/library/difflib.html
4https://flask.palletsprojects.com/en/2.0.x/
Fig. 2. The web app developed for the online testing.
C. Metrics
To assess the level of performance of the proposed system
we use the Word Error Rate (WER) to compare the output sign
sequence with the reference sign sequence translation, either
defined in the test set, or input by the online participants. The
WER is similar to the CER used in the similarity checks and
is defined as:
WER =
S+D+I
N=
S+D+I
S+D+C(2)
where Sis the number of word substitutions, Dis the
number of word deletions, Iis the number of word insertions,
Cis the number of correct words, and Nis the number of
words in the reference, i.e. N=S+D+C.
IV. RES ULT S
The evaluation of the system’s performance using the test
set was an average of 4.49% WER, i.e. about 1 erroneous sign
in 20 signs output. A small part of these errors are due to the
difference between the output signs, as they are given in the
system’s dictionary and as they are provided in the reference
translations. These are minor differences comprising one or
two characters added in some of the signs.
Errors occurred most notably when input words were ex-
panded compared to the way they are found in the system’s
database. For example, when a word is written in the diminu-
tive plural form. In those cases either the algorithm finds a
similar word which is not the correct sign corresponding to
the word token, or appends the token to the unrecognized word
token list. Such errors do not occur when the input word has
a few changes, or the word is long enough that the relative
number of changes is minimal.
The online testing results were also promising, with minimal
errors in the output sequences for the input provided by the
user. Most errors could be attributed to: i) the word translations
not being part of the signs listed in the web app or ii) being
synonyms to words that are provided in the system’s dictionary
and which have not been added to the system. In some of these
cases the words are appended to the unrecognized word token
list, but a frequently enough a similar, but erroneous, sign is
found.
V. CONCLUSION
The proposed system is able to translate an input text
sequence in to an output sign language sign sequence. The
system is based on a rule-based algorithm that converts the
input word tokens sequentially into their sign language equiv-
alents. Even though supporting a limited sign vocabulary, the
results from the internal and online testing are promising. With
future improvements the system can be used in combination
with a sign language generator to create a complete text-to-
sign language solution. This would be of great help for the
digital inclusion and communication with the deaf and hearing
impaired community.
REFERENCES
[1] A. S.-L.-H. Association. (2021) Hearing loss (ages 5+). [Online]. Avail-
able: https://www.asha.org/practice-portal/clinical-topics/hearing-loss/
[2] Dnevnik. (2006) Around 6,000 deaf people are ask-
ing for sign language news on mtv. [Online]. Avail-
able: https://web.archive.org/web/20110727190206/http://star.dnevnik.
com.mk/default.aspx?pbroj=1782&stID=10852
[3] Republic of North Macedonia State Statistical Office. (2002) Census
of Population, Households and Dwellings in the Republic of North
Macedonia. [Online]. Available: https://www.stat.gov.mk/Publikacii/
knigaXIII.pdf
[4] G. R. I. Charles Reilly. (2011) Snapshot of deaf
and hard of hearing people, postsecondary attendance and
unemployment. [Online]. Available: https://www.gallaudet.edu/
office-of-international-affairs/demographics/deaf-employment- reports/
[13] D. Stiehl, L. Addams, L. S. Oliveira, C. Guimar˜
aes, and A. Britto,
“Towards a signwriting recognition system,” in 2015 13th International
Conference on Document Analysis and Recognition (ICDAR). IEEE,
2015, pp. 26–30.
[5] T. G. N. T. Board. (2021) Information for deaf visitors to germany.
[Online]. Available: https://www.germany.travel/en/accessible-germany/
disability-friendly- travel-for/deafness.html
[6] T. Hanke, “HamNoSys-representing sign language data in language
resources and language processing contexts,” in LREC, vol. 4, 2004,
pp. 1–6.
[7] K. Kaur and P. Kumar, “Hamnosys to sigml conversion system for sign
language automation,” Procedia Computer Science, vol. 89, pp. 794–
803, 2016.
[8] M. Varghese and S. K. Nambiar, “English to sigml conversion for sign
language generation,” in 2018 International Conference on Circuits and
Systems in Digital Enterprise Technology (ICCSDET). IEEE, 2018, pp.
1–6.
[9] S. Ebling and J. Glauert, “Building a swiss german sign language avatar
with jasigning and evaluating it among the deaf community,” Universal
Access in the Information Society, vol. 15, no. 4, pp. 577–587, 2016.
[10] N. C. Ngon and Q. L. Da, “Application of hamnosys and avatar 3d
jasigning to construction of vietnamese sign language animations,” The
University of Danang-Journal of Science and Technology, pp. 61–65,
2017.
[11] Z. Kamal and H. Hassani, “Towards kurdish text to sign translation,” in
Proceedings of the LREC2020 9th Workshop on the Representation and
Processing of Sign Languages: Sign Language Resources in the Service
of the Language Community, Technological Challenges and Application
Perspectives, 2020, pp. 117–122.
[12] N. Aouiti, “Towards an automatic translation from arabic text to sign
language,” in Fourth International Conference on Information and
Communication Technology and Accessibility (ICTA). IEEE, 2013, pp.
1–4.
[14] A. S. Dhanjal and W. Singh, “An automatic conversion of punjabi
text to indian sign language,” EAI Endorsed Transactions on Scalable
Information Systems, vol. 7, no. 28, p. e9, 2020.
[15] S. H. Kang and S. H. Park, “Toward korean text-to-sign language
translation system (test).”
[16] S. Koceski and N. Koceska, “Development and evaluation of a 3d virtual
tutor for macedonian sign language,” in International Conference on
Information Technology and Development of Education – ITRO 2015,
Zrenjanin, Serbia, 2015.
[17] B. Joksimoski, I. Chorbev, K. Zdravkova, and D. Mihajlov, “Toward
3d avatar visualization of macedonian sign language,” in International
Conference on ICT Innovations. Springer, 2015, pp. 195–203.
[18] J. W. Ratcliff and D. E. Metzener, “Pattern matching: the Gestalt
approach,” Dr Dobbs Journal, vol. 13, no. 7, p. 46, 1988.
ResearchGate has not been able to resolve any citations for this publication.
Article
Full-text available
An escalating focus worldwide on the development of persons with disabilities results in the disability-inclusive 2030 UNagenda for sustainable development. This paper discusses the development of Punjabi text to Indian Sign Language (ISL)conversion system aiming for the better communication and education of hearing-impaired people. The developed systemaccepts Punjabi text as input and deliver output as 3D animation. The currently created corpus involves more than 100commonly used Punjabi words under various categories. This system has been tested on all the prepared words. Results ofthe developed system are very encouraging and the work is still going on with full enthusiasm. Since there is no universalor written form of sign language available, so generating synthetic sign animation is the best solution. In near future it canbe implemented as a tutoring system to promote sign language education among common people of Punjab.
Article
Full-text available
Sign language is a visual gesture language. Deaf people use sign language as their main source for communication. They use their hands, face and arms for conveying thoughts and meanings. Sign language is not having any written form and generating signs using video approach is very expensive, so, sign animation is the best solution. Research on Indian Sign Language (ISL) Automation is limited because of lack of Indian Sign Language knowledge and the unavailability of such tools which provide any education for Indian Sign Language. An automation system is designed in which HamNoSys is generated corresponding to the ISL words. HamNoSys is further converted into XML form known as SiGML. A virtual human in tool JA SiGML URL APP takes input as SiGML and generates signs corresponding to the given words. The existing tool available for generation of SiGML is eSIGNEditor which contains a database of words for American and British Sign Language while sign language in India is different from that in America and Britain. So a system is designed to convert the HamNoSys to SiGML, so as to process it further to the SiGML player to animate the signs for ISL words.
Conference Paper
Full-text available
Communication represents a very important need especially for deaf persons who use sign languages. For this reason, facilitating this communication through Information and Communication Technologies attracted many research. In this context, the Research Lab. LaTICE of the University of Tunis has been devoted during many years to develop the project Websign [1] aiming at using avatars to translate automatically a written text to sign language. The architecture of Websign is based on many modules i.e. linguistic treatment, creation sign dictionaries, animating the avatar, etc.. However, until now, Websign has not yet developed all steps of treatment related to Arabic language and in particular its linguistic step. The aim of this work is to provide an automatic translation system from an Arabic text into an animated sign language with a special focus on syntactic and semantic analysis of the input text in order to extract the maximum of inherent information that can be useful to the translation step.
Conference Paper
Full-text available
This paper gives a short overview of the Hamburg Notation System for Sign Languages (HamNoSys) and describes its application areas in language resources for sign languages and in sign language processing.
Conference Paper
The resources and technologies for sign language processing of resourceful languages are emerging, while the low-resource languages are falling behind. Kurdish is a multi-dialect language, and it is considered a low-resource language. It is spoken by approximately 30 million people in several countries, which denotes that it has a large community with hearing-impairments as well. This paper reports on a project which aims to develop the necessary data and tools to process the sign language for Sorani as one of the spoken Kurdish dialects. We present the results of developing a dataset in HamNoSys and its corresponding SiGML form for the Kurdish Sign lexicon. We use this dataset to implement a sign-supported Kurdish tool to check the accuracy of the sign lexicon. We tested the tool by presenting it to hearing-impaired individuals. The experiment showed that 100% of the translated letters were understandable by a hearing-impaired person. The percentages were 65% for isolated words, and approximately 30% for the words in sentences. The data is publicly available at https://github.com/KurdishBLARK/KurdishSignLanguage for non-commercial use under the CC BY-NC-SA 4.0 licence.
Article
There are two kinds of sign languages used in Korea. One is Korean Sign Language, which has the identical rules with that of Korean grammar, the other is Korean Natural Sign Language(KnSL), which is actually used by deaf person. KnSL differs from Korean Language in morpheme characteristics. Therefore, morpheme translation rules must be defined first for translation between these two. So, in this paper, we define and implement morpheme translation rules for our TeST system (Korean Text - to - Sign Language Translation system). We successfully translated about 98KnSL with the rules proposed in this paper, and we also found the fact that special translation rules are necessary in many postpositions and endings. Therefore, we propose some translation rules which are adopted to Korean conjugation and we also propose KnSL morpheme creation network by using morpheme translation rules. 1 Introduction In these days, there are high level of interests in improvements of social welfare in the wor...
Around 6,000 deaf people are asking for sign language news on mtv
  • Dnevnik
Dnevnik. (2006) Around 6,000 deaf people are asking for sign language news on mtv. [Online]. Available: https://web.archive.org/web/20110727190206/http://star.dnevnik. com.mk/default.aspx?pbroj=1782&stID=10852
Snapshot of deaf and hard of hearing people, postsecondary attendance and unemployment
  • G R I Charles Reilly
G. R. I. Charles Reilly. (2011) Snapshot of deaf and hard of hearing people, postsecondary attendance and unemployment. [Online].