Towards a System for Converting Text to Sign
Language in Macedonian
Stefan Spasovski1, Branislav Gerazov1, Risto Chavdarov1, Viktorija Smilevska2, Aneta Crvenkovska,
Tomislav Kartalov1, Zoran Ivanovski1, and Toni Bachvarovski3
1Faculty of Electrical Engineering and Information Technologies
Ss Cyril and Methodius University in Skopje, Macedonia
2Elementary school ”Kuzman Josifovski - Pitu”, Skopje, Macedonia
3Association for Assistive Technologies “Open the Windows”, Skopje, Macedonia
Abstract—The paper presents initial results in the design and
development of a system for automatic conversion of text to sign
language in Macedonian. The system will be an essential part
of a larger system for the automatic generation of Macedonian
sign language based on text. This system will facilitate the digital
inclusion and will ease communication with the Macedonian deaf
and hearing impaired community. The system is implemented as
a web application which allows input text to be encoded in the
equivalent sequence of sign language signs. The initial results
show an average sign error rate of 4.49%. Online testing was
also organized that conﬁrmed these promising results.
Index Terms—natural language processing, assistive technol-
ogy, text-to-sign language, sign language, deafness
Deafness is deﬁned as a condition of extreme hearing loss,
i.e. having very little or no hearing at all. The American
Speech-Language-Hearing Association (ASHA) deﬁnes pro-
found hearing loss as only being able to hear sounds above
90 dB, with severe hearing loss ranging between 71 – 90 dB
. Macedonian Law, places the threshold at 80 dB. The
estimated number of deaf and hearing impaired people in
Macedonia is around 6000 according to information from 2006
, which is 0.3% of Macedonia’s population . This is
comparable to the percentage of deaf people in places like
the United States (0.38%)  and in Germany (0.28%) .
In today’s high-tech information-rich world, the digital
inclusion of people with disabilities is becoming increasingly
important. The deaf and hearing impaired are generally able
to directly use and interact with computers and smart devices,
and can follow traditional visual media such as TV and
newspapers. However, they ﬁnd it hard to read text at speed.
This is especially true for those born deaf or hearing impaired,
as they have never heard the sounds of phonemes that phonetic
orthography transcribes with graphemes in written text. As
a consequence, for them phonetic transcription is not much
different from logograms, such as the Chinese characters used
to write Mandarin and other Asian languages. This problem
can be mitigated through offering live sign language (SL)
translation, but most TV broadcasters do not offer this service.
The problem is even more pronounced in Macedonia, where
there are only around 30 certiﬁed SL translators.1
One way to ease the digital inclusion of the deaf and hearing
impaired is through assistive systems able to automatically
convert text-to-sign language. One example of such a system
is the HandTalk App in which a virtual avatar named Hugo
converts text to sign language on the users smart device.2
These systems are made up of two essential parts: i) a text-
to-SL converter that transforms textual input to a sequence of
signs or gestures, and ii) a SL generator that uses the sequence
of signs to generate sign language, usually via 3D rendering of
an animated character, i.e. avatar. Sign language differs from
spoken language, in that it does not support inﬂection. For
example, to create the future tense in Macedonian we add the
future particle before the verb. Tense is not formed in that way
in sign language. Instead the speaker uses the inﬁnitive form
of the verb together with signs like “later” to signify that they
are speaking about a future event. The same holds true for
verb conjugation, singular and plural, case and articles.
Although sign languages across the world do share signs,
there are still different standardized sign languages, such
as: American Sign Language (ASL), Italian Sign Language
(LIS), Indian Sign Language (ISL), Vietnamese Sign Lan-
guage (VSL), and Macedonian Sign Language (MSL). The
text-to-SL converter can encode the signs in various formats.
One famous format that dates back to 1984 is the Hamburg
Notation System for Sign Languages (HamNoSys), which
encodes signs through a set of pictograms or symbols .
As an extension to HamNoSys, the Signing Gesture Mark-up
Language (SiGML) describes the symbols using XML tags .
This extension allows the storage and use of the transcriptions
in computer based systems, such as 3D rendering software,
that can be used to generate sign language via an animated
Text-to-SL systems have been developed for many of the
world languages, such as English , German , Vietnamese
, Kurdish , Arabic , Brazilian Portuguese ,
Punjabi , Korean  etc. Most systems rely on a simple
word-to-sign mapping, i.e. each word token from the input text
is looked up in a lexicon of signs, and if no match is found it is
spelled out using a sequence of alphabet letter signs . More
advanced rule-based-systems map the input text grammar to
natural sign language grammar . Recently, the application
of machine learning has allowed improved performance in
these systems, directly applying methods used in the area of
Machine Translation .
In Macedonia, work has mostly been done on the generation
of Macedonian Sign Language via virtual avatars. Koceski
and Koceska  developed and evaluated a 3D virtual tutor
for MSL. Joksimoski et al.  presented a 3D visualization
system that extensively uses animation and game concepts for
accurately generating sign languages using 3D avatars.
Here, we present a text-to-SL system that translates an input
Macedonian text into an output sequence of Macedonian Sign
Language signs. The system is built on a rule based algorithm,
which analyses input text, comparing the input word tokens to
a lexicon of some 200 signs. The performance of the system
is evaluated with a set of test sentences and the results show
a Sign Error Rate (SER) of 4.49%. We augment this analysis
with online testing of the system, which conﬁrms the validity
of the initial results. The system can be used as the basis
for building a complete system for text based sign language
generation in Macedonian.
A. Sign mappings organization
We organize the data by placing the words and signs in ﬁve
•list of signs,
•list of names,
•dictionary of word to sign mappings, and
•dictionary of phrases that map directly to sequences of
In the presented system we have 221 signs.
B. Text preprocessing
Text input is ﬁrst normalised by converting all upper case
characters to lower case and removing all punctuation. We
then divide the text string into into a list of word tokens. We
initialise two empty lists for storing the sign sequence output
and unrecognized word tokens, and deﬁne a ﬂag to be used
when a phrase has been recognized. With that we are ready
to begin the translation process.
C. Main loop
We go through the word tokens in the input list one by one
and run them through multiple checks. Firstly, we check if the
word token is part of a phrase by concatenating the succeeding
token from the list. If it is, then we append the sequence of
signs corresponding to that phrase, skip the second word from
the next iteration and move on from there. Next, if the token
Is it part of
Is it in
Is it a name
Is it a sign
Is it in
Is it similar
to a sign
Is the root
similar to a
to unknown list
Fig. 1. Block diagram of the algorithm for converting text to sign language
is in the skip list it is ignored and the next token is processed.
If not in the skip list, we check if the input word token is
part of the names or signs lists. If found the word token is
appended as is to the output sign sequence. If not, the token
is looked up in the dictionary of word to sign mappings, and
if found it’s sign mapping is appended to the output. If the
word still has not been found in any of the checks then we
continue with similarity checks.
D. Similarity checks
The ﬁnal part of the main loop consists of similarity checks.
The system comprises two different similarity checks. They
are based on the “gestalt pattern matching” algorithm sug-
gested by Ratcliff and Obershelp in the 1980s . The idea
being to ﬁnd the longest contiguous matching subsequence
that contains no “junk” elements. This is applied recursively
to the pieces of the sequences to the left and to the right of
the matching subsequence. We use the implementation of the
algorithm in the difﬂib3Python library.
The similarity check outputs a list of 3 strings sorted
from the most likely to the list likely match. Based on out
experiments we get the best results when the cutoff is equal
to 0.7. One other problem that we encounter is the fact that
sometimes the most likely match (the ﬁrst element of the
string) is not at all the most likely, and that the second or
third string in the list is the correct answer. We augment the
similarity search algorithm with a Character Error Rate (CER)
to select one of the three offered outputs:
where Sis the number of character substitutions, Dis the
number of character deletions, Iis the number of character
insertions, Cis the number of correct characters, and Nis the
total number of characters in the reference, i.e. N=S+D+C.
III. EXP ER IM EN TS
We used two experiments to evaluate the performance of
the proposed system. The ﬁrst was based on an internal test
set, and the second was based on online testing.
A. Test set evaluation
We developed an internal test set comprising 124 sentences
for which we provided reference translations to sign language
sequences of signs. We took special care to have a varied test
set both in terms of sentence length as well as ample coverage
of the set of signs known to the system.
B. Online evaluation
To provide a platform for testing the proposed system, we
developed a web application that provides a user interface for
online testing. The web app was developed based on Flask4.
HTML was used to build the site layout, while Flask was used
to render the website, receive user input and return the results
of the translation process.
The website lets the user input words for translation. After
submitting the input the translation process starts and the
output from the proposed system including the output sign
sequence and the list of unrecognized word tokens. There is
also a text form that the user can use to give feedback. For
our online tests this was the correct sign sequence in case the
system returned an erroneous one. To improve coverage of the
known signs in the online tests, the signs known by the system
are listed at the end of the web page.
Fig. 2. The web app developed for the online testing.
To assess the level of performance of the proposed system
we use the Word Error Rate (WER) to compare the output sign
sequence with the reference sign sequence translation, either
deﬁned in the test set, or input by the online participants. The
WER is similar to the CER used in the similarity checks and
is deﬁned as:
where Sis the number of word substitutions, Dis the
number of word deletions, Iis the number of word insertions,
Cis the number of correct words, and Nis the number of
words in the reference, i.e. N=S+D+C.
IV. RES ULT S
The evaluation of the system’s performance using the test
set was an average of 4.49% WER, i.e. about 1 erroneous sign
in 20 signs output. A small part of these errors are due to the
difference between the output signs, as they are given in the
system’s dictionary and as they are provided in the reference
translations. These are minor differences comprising one or
two characters added in some of the signs.
Errors occurred most notably when input words were ex-
panded compared to the way they are found in the system’s
database. For example, when a word is written in the diminu-
tive plural form. In those cases either the algorithm ﬁnds a
similar word which is not the correct sign corresponding to
the word token, or appends the token to the unrecognized word
token list. Such errors do not occur when the input word has
a few changes, or the word is long enough that the relative
number of changes is minimal.
The online testing results were also promising, with minimal
errors in the output sequences for the input provided by the
user. Most errors could be attributed to: i) the word translations
not being part of the signs listed in the web app or ii) being
synonyms to words that are provided in the system’s dictionary
and which have not been added to the system. In some of these
cases the words are appended to the unrecognized word token
list, but a frequently enough a similar, but erroneous, sign is
The proposed system is able to translate an input text
sequence in to an output sign language sign sequence. The
system is based on a rule-based algorithm that converts the
input word tokens sequentially into their sign language equiv-
alents. Even though supporting a limited sign vocabulary, the
results from the internal and online testing are promising. With
future improvements the system can be used in combination
with a sign language generator to create a complete text-to-
sign language solution. This would be of great help for the
digital inclusion and communication with the deaf and hearing
 A. S.-L.-H. Association. (2021) Hearing loss (ages 5+). [Online]. Avail-
 Dnevnik. (2006) Around 6,000 deaf people are ask-
ing for sign language news on mtv. [Online]. Avail-
 Republic of North Macedonia State Statistical Ofﬁce. (2002) Census
of Population, Households and Dwellings in the Republic of North
Macedonia. [Online]. Available: https://www.stat.gov.mk/Publikacii/
 G. R. I. Charles Reilly. (2011) Snapshot of deaf
and hard of hearing people, postsecondary attendance and
unemployment. [Online]. Available: https://www.gallaudet.edu/
 D. Stiehl, L. Addams, L. S. Oliveira, C. Guimar˜
aes, and A. Britto,
“Towards a signwriting recognition system,” in 2015 13th International
Conference on Document Analysis and Recognition (ICDAR). IEEE,
2015, pp. 26–30.
 T. G. N. T. Board. (2021) Information for deaf visitors to germany.
[Online]. Available: https://www.germany.travel/en/accessible-germany/
 T. Hanke, “HamNoSys-representing sign language data in language
resources and language processing contexts,” in LREC, vol. 4, 2004,
 K. Kaur and P. Kumar, “Hamnosys to sigml conversion system for sign
language automation,” Procedia Computer Science, vol. 89, pp. 794–
 M. Varghese and S. K. Nambiar, “English to sigml conversion for sign
language generation,” in 2018 International Conference on Circuits and
Systems in Digital Enterprise Technology (ICCSDET). IEEE, 2018, pp.
 S. Ebling and J. Glauert, “Building a swiss german sign language avatar
with jasigning and evaluating it among the deaf community,” Universal
Access in the Information Society, vol. 15, no. 4, pp. 577–587, 2016.
 N. C. Ngon and Q. L. Da, “Application of hamnosys and avatar 3d
jasigning to construction of vietnamese sign language animations,” The
University of Danang-Journal of Science and Technology, pp. 61–65,
 Z. Kamal and H. Hassani, “Towards kurdish text to sign translation,” in
Proceedings of the LREC2020 9th Workshop on the Representation and
Processing of Sign Languages: Sign Language Resources in the Service
of the Language Community, Technological Challenges and Application
Perspectives, 2020, pp. 117–122.
 N. Aouiti, “Towards an automatic translation from arabic text to sign
language,” in Fourth International Conference on Information and
Communication Technology and Accessibility (ICTA). IEEE, 2013, pp.
 A. S. Dhanjal and W. Singh, “An automatic conversion of punjabi
text to indian sign language,” EAI Endorsed Transactions on Scalable
Information Systems, vol. 7, no. 28, p. e9, 2020.
 S. H. Kang and S. H. Park, “Toward korean text-to-sign language
translation system (test).”
 S. Koceski and N. Koceska, “Development and evaluation of a 3d virtual
tutor for macedonian sign language,” in International Conference on
Information Technology and Development of Education – ITRO 2015,
Zrenjanin, Serbia, 2015.
 B. Joksimoski, I. Chorbev, K. Zdravkova, and D. Mihajlov, “Toward
3d avatar visualization of macedonian sign language,” in International
Conference on ICT Innovations. Springer, 2015, pp. 195–203.
 J. W. Ratcliff and D. E. Metzener, “Pattern matching: the Gestalt
approach,” Dr Dobbs Journal, vol. 13, no. 7, p. 46, 1988.