Conference PaperPDF Available

An Approach for Hand Sign Language Using Static Signs (Comparative study)

Authors:
  • Director, IIIT Kottayam, Kerala, India Institute of National Importance

Abstract

The sign language recognition and translation can be achieved which involves static and dynamic gesture recognition in which static gestures can easily be interpreted. But, for Dynamic gestures the sign language can be recognized with motion of the hands or movement of it for which Image based i.e. vision based mechanism can be applied. The Vision based technique involves either capturing the image of action or position or even directly visually recognizing the action. This technique which is very easy to understand as vision based can include both images as well as sequences of actions. Then, the feature i.e. signs action extraction and processing can be done according to type of sign recognized during visual gestures. Then compare with existing gestures and display the message and can be translated into text and speech as well. Also different image preprocessing algorithms can be developed.
An Approach for Hand Sign Language Using
Static Signs (Comparative study)
Rajshekhar B. Waghe Prof.Pravin Futane Dr. Rajiv V Dharaskar
Student, Department of Computer Engg, Department of Computer Engg, Professor & HOD,
Sinhgad College of Engineering, Sinhgad College of Engineering, Dept. Of Computer Engg.
University of Pune (India) University of Pune (India) G.H.Raisoni COE, Nagpur
r.waghe@rediff.com pravinfutane9@gmail.com rajiv.dharaskar@gmail.com
Abstract- The sign language recognition and translation
can be achieved which involves static and dynamic
gesture recognition in which static gestures can easily be
interpreted. But, for Dynamic gestures the sign
language can be recognized with motion of the hands or
movement of it for which Image based i.e. vision based
mechanism can be applied. The Vision based technique
involves either capturing the image of action or position
or even directly visually recognizing the action. This
technique which is very easy to understand as vision
based can include both images as well as sequences of
actions. Then, the feature i.e. signs action extraction and
processing can be done according to type of sign
recognized during visual gestures. Then compare with
existing gestures and display the message and can be
translated into text and speech as well. Also different
image preprocessing algorithms can be developed.
Keywords- Recognition, Static Signs Analysis, Sign
language recognition, gestures, image processing
I. INTRODUCTION
For many deaf people, sign language is the principle
means of communication. A sign language is a
language which, conveys the meaning in visual
gesture patterns, uses visually transmitted sign
patterns and simultaneously combining hand shapes,
orientation and movement of the hands, arms or
body, and facial expressions to express the thoughts
via different methods like visual based or by dynamic
actions.
One problem is that very few people who are not
themselves deaf ever learn to sign. This therefore
increases the isolation of deaf people; they may be
confined in many of their interactions to
communicating only with other deaf people. It seems
that technology might have a role to play here, if
computers could be programmed to recognize sign
language and to translate it into another form, such as
speech or written text.
II. LITERATURE REVIEW
There are a number of sign languages that emerged
from French Sign Language (LSF), or were the result
of language contact between local community sign
languages and LSF. These include: French Sign
Language, Quebec Sign Language, American Sign
Language, Brazilian Sign Language (LIBRAS) and
others. A subset of this group includes languages that
have been heavily influenced by American Sign
Language (ASL), or are regional varieties of ASL.
Bolivian Sign Language is sometimes considered a
dialect of ASL. Sign language differs from oral
language in its relation to writing. The phonemic
systems of oral languages are primarily sequential,
that is, the majority of phonemes are produced in a
sequence one after another, although many languages
also have non-sequential aspects such as tone. Most
deaf signers read and write the oral language of their
country. However, there have been several attempts
at developing scripts for sign language.
A review of the abstraction approach can be found in
[1]. We must understand that HMM topology was
used and is applied for Sign language as stated in [2].
The research on hand gesture and sign recognition
has two main dimensions: isolated and continuous
recognition. Isolated recognition focuses on a single
hand gesture that is performed by the user and
attempts to recognize it. In continuous recognition,
user is expected to perform gestures one after the
other and the aim is to recognize every gesture that
the user performs. The continuous recognition
problem is slightly different for hand gesture
recognition and sign language recognition systems.
For instance the method discussed in [3] is based on
basic approach for sign language and its feature
detection and extraction and pattern matching. On the
other hand researchers like [4] concentrate on the
accuracy of stored standard sign with the runtime
sign and their results.
In hand gesture controlled environments, the problem
can be considered as a gesture spotting problem,
where the task is to differentiate the meaningful
gestures of the user from the unrelated ones. In sign
language recognition, the continuous recognition
problem includes the co-articulation problem. The
preceding sign acts the succeeding one, which
complicates the recognition task as the transitions
between the signs should be explicitly modeled and
incorporated to the recognition system. Moreover,
language models are used to be able to perform on
large-vocabulary databases.
The proposing methodology is based on Sign
Language Recognition based on Hand Gestures [1]
and positions using one of the HMM methodologies.
The methodology of Sign language Recognition is
reviewed here.
III. VISION BASED APPROACH PROCESS
The basic belief of the algorithm is that you don’t
need to understand the meaning of a whole sign
completely so as to effectively generate its match.
The principle of working is based on run time static
sign orientation and position. This proposed method
is aimed to develop an automatic Indian Sign
Language Recognition platform for hearing impaired
community of India. The system can recognize
different hand gestures of Indian Sign Language.
Humans naturally use gestures to communicate.
Gestures are natural means for conveying
information and used by humans for diverse purposes
ranging from pointing at a person to get his/her
attention to conveying information in day-to-day life.
Gesture literally means “an expressive movement of
a part of the body, the hand or head, in order to bring
forward intentions and attitude”. Gesture recognition
is the interpretation of a given gesture into text and
speech form. This importance has motivated to use
gestures for communicating with computers. The
focus of the proposed work is to develop a Human
Computer Interaction (HCI) platform in context to
Indian Sign language (INSL). The development of a
system for translating Indian sign language into
spoken language would be great help for deaf as well
as hearing people of the country.
The process discussed here is very similar. It relies
on the use of one of the Image processing algorithms,
which are effective and in correspondence with
HMM for Sign language interpretation.
A. Method
The process involves two layer classifications. At
first, coarse classification is done according to
detection of hand motion and tracking the hand
location and second classification is based on key
frame selection and hand shape recognition of key
frames. Motion history image and Fourier descriptor
are used for motion direction recognition and key
frame selection respectively. Generic cosine
descriptor (GCD) has been proposed for feature
extraction of hand postures. GCD is invariant (not
changeable) to scale, translation and rotation of hand
shapes. The system can test different hand gestures of
persons.
B. Technique
Gesture (or sign language) has been widely used in
the deaf community. In the foreseeable future,
gesture inputs can be ideally applied for human-
computer interface. Review of the most recent works
related to hand gesture interface techniques: glove
based technique, vision-based technique, and analysis
of drawing gesture. Vision-based technique is the
most natural way of constructing a human-computer
interface which has many applications. However, it
has difficulties in (1) segmentation of moving hands;
(2) tracking and analysis of hand motion; and (3)
recognition. Sign language consists of static hand
gesture and dynamic hand gesture. Hidden Markov
Model (HMM) [4] is implemented for visual
recognition of complex, structured hand gestures
such as ASL. They used moments and normalization
to separate the rough posture estimate from spatial
specific (translation, rotation, and scaling).
C. Flow-analysis
Figure 3.1 shows the flow diagram of the sign
language recognition system. The use of two levels
classifier is done. Firstly, the recognition is only,
based on global analysis of motion. After first
classification, shape information is used for key
frames selection. Key frames are used for Feature
extraction and hand shape analysis. Second classifier
is used for hand shape matching and gesture
recognition.
Image
First classifier based on orientation &
Position of hand
Key frame
Feature extraction
Second classifier based
on hand shapes in key
Sign detecting
Pattern matching
Recognizing Sign
Figure 1 Flow diagram of proposed sign language
recognition system.
Image is captured as Input is taken in either RGB or
gray scale for in 1st stage of flow. Then based on
image classification is done either on first and second
classifier which involves position and orientation and
hand shape in static sign of image. Then key frame
analysis is done.
The hand gesture image sequence is analyzed for key
frame selection. Since the hand shapes between two
consecutive view models are very similar to each
other, we only need to select some key frames for the
stored model generation and the input model
generation. The closed boundary of segmented hand
shape can be described by a Fourier descriptor (FD)
vector or any transform methodology on images.
The phase of feature extraction involves in sign
language recognition, it is desirable to use a shape
representation technique that will sufficiently
describe the shape of the hand while also being
capable of fast computations, enabling the
recognition to be done in real time. It is also desirable
for the technique to be invariant to translation,
rotation, and scaling. In addition, a method that will
allow for easy matching would be beneficial
compared the shape descriptors in terms of good
retrieval accuracy, compact features, general
application, low computational complexity, robust
retrieval performance, and hierarchical coarse-to-fine
representation. The generic Fourier descriptor is
obtained by extracting various scaled Fourier
coefficients from the 2-D Fourier transform of the
polar-raster sampled image.
Then, sign detection is done of static sign. After,
detecting the stored standard model of static sign and
runtime model is matched known to be as pattern
matching. Hence, a static sign is recognized with
some amount of common/different accuracy.
Implementing the proposed work may involve
trajectory motion tracking, key frame selection,
computation of Fourier descriptor, feature extraction
and analysis, pattern matching and finally based on
hand shapes recognition can be done.
D. Measures
In sign language recognition both hand movements
and hand shape variation are meaningful. The
contributions of this work can be summarized as
follows: First: A general method for sign language
recognition system which consists of two layer
classifier. One layer for classifying signs according to
their hand movements and another one for classifying
each group of movements, based on their hand
shapes. Second, we use a GCD features (any image
transform) to recognize hand shapes. It is invariant to
scale, translation and rotation.
IV. STANDARD TEMPLATE OF STATIC SIGNS
The above signs are stored to be as static
signs in database as stored model and can be
performed even after the summary of result
is generated.
The run time static sign is matched (pattern)
(input model) with the stored model
(standard model).
The accuracy of sign is measured with the
help of matching values (of generated
transform) which may not be totally precise
or exact but approaches a relative value.
Figure 2 Standard hand Signs
V. APPLICATIONS
The concept of Sign language recognition can be
applied to almost any area but basic purpose is for
mostly non-verbal communication for people who are
deaf and impaired.
Some of applications are as follows:
SLR’s basic application is for Interactive
Communication
Can be used for E-learning in an Online
System
SLR Assisted Sign Language Education
Sign Tutor: An Interactive System for Sign
Language Tutoring
Can be applied in field of deaf and dumb
reading and understanding
VI. ADVANTAGES AND DISADVANTAGES
A. ADVANTAGES
The main advantage is that the entire process of Sign
language recognition is bridging of the
communication gap. It helps in also converting visual
gestures translated into the speech form.
Several Static signs can be recognized with more
accuracy based on different features of hand signs.
The results obtained can be improved and refined
into a later stage based on the preciseness of sign.
B. DISADVANTAGES
The image pre-processing is required so as to
normalize the image to be recognizable from.
The constraint of using colored gloves is a drawback
to be removed.
Different features are required to be acquired for
recognizable to different hands.
Lot of speed up computations and image
transformations are done which leads more
computational time.
As such many of drawbacks are removable but totally
cannot be eliminated to get more closer and accurate
results.
As the Sign language recognition is not yet
Standardized for Indian Sign Language, there is a lot
of wide scope of development and future work to be
done.
VII. CONCLUSIONS
To conclude we state that the comparative study
simultaneously uses both the static signs and
combination of those parameters.
As stated above, automatic sign language recognition
offers enhancement of communication capabilities
for the speech and hearing impaired, promising
improved social opportunities and integration. The
objective of the proposed research/project work is to
build a system that uses natural gestures as a
modality for recognition in the vision based setup.
The focus of the proposed project is to develop a
platform in context to Indian Sign language. In a
country like India there is a need of automatic sign
language recognition system, which can cater the
need of hearing impaired people. The ultimate gain of
the proposed system is enormous.
VIII. REFERENCES
[1] Paulraj M P, Sazali Yaacob, Hazry Desa, Hema C.R, .et al.,
Extraction of Head and Hand Gesture Features for
Recognition of Sign Language” International Conference on
Electronic Design 2008, ICED 2008, pp.1-6, 2008.
[2] Tadashi Matsuo, Yoshiaki Shirai, Nobutaka Shimada,
Automatic Generation of HMM Topology for Sign
Language Recognition”, 19th International Conference on
Pattern Recognition, pp.1-4, ICPR 2008.
[3] Mairyam Pahlevanzadeh, Mansour Vafadoost, Majid
Shahnazi “Sign Language Recognition”, International
Telecommunication Research Center, IEEE-2007.
[4] Masaru Maebatake, Iori Suzuki, Sign Language Recognition
Based on Position and Movement Using Multi-Stream HMM
”, Proceedings of the 2008 Second International Symposium
on Universal Communication, pp. 478-481, 2008 .
[5] Stavros Theodorakis, Athanassios Katsamanis, “Product-
HMMS For Automatic Sign Language Recognition”,
Proceedings of the 2009 IEEE International Conference on
Acoustics, Speech and Signal Processing, pp. 1601-1604,
2009.
[6] Van R. Culver, “A Hybrid Sign Language Recognition
System”, ISWC, Proceedings of the Eighth International
Symposium on Wearable Computers, pp.30 - 33, 2004.
[7] Jung-Ning Huang, Pi-Fuei Hsieh and Chung-Hsien Wu,
Vision-Based Recognition of Hand Shapes in Taiwanese
Sign Language”, Springer Link, Volume 3784/2005, pp. 224-
231, 2005.
[8] Qi Wang, Xilin Chen (2006), “Sign Language using
Homography”, IEEE International Conference on
Multimedia and Expo, pp.429-432, 2006.
[9] Andreas Domingo, Rini Akmeliawati, et al., “Pattern
Matching for Automatic Sign Language Translation System
using LabVIEW ”, International Conference on Intelligent
and Advanced Systems, 2007, ICIAS 2007, pp. 660-665,
2007.
[10] Jeroen F. Lichtenauer, Emile A. Hendriks, and Marcel J.T.
Reinders , “Sign Language Recognition by Combining
Statistical DTW and Independent Classification”, IEEE
Transactions on Pattern Analysis and Machine Intelligence,
Volume: 30, Issue: 11, pp. 2040-2046, 2008.
Conference Paper
Full-text available
We address multistream sign language recognition and focus on efficient multistream integration schemes. Alternative approaches are investigated and the application of Product-HMMs (PHMM) is proposed. The PHMM is a variant of the general multistream HMM that also allows for partial asynchrony between the streams. Experiments in classification and isolated sign recognition for the Greek sign language using different fusion methods, show that the PHMMs perform the best. Fusing movement and shape information with the PHMMs has increased sign classification performance by 1,2% in comparison to the Parallel HMM fusion model. Isolated sign recognition rate increased by 8,3% over movement only models and by 1,5% over movement-shape models using multistream HMMs.
Conference Paper
Full-text available
Sign language is the primary communication method that impaired hearing people used in their daily life. Sign language recognition has gained a lot of attention recently by researchers in computer vision. Sign language recognition systems in general require the knowledge of the hand's position, shape, motion, orientation and facial expression. In this paper we present a simple method for converting sign language into voice signals using features obtained from head and hand gestures which can be used by hearing impaired person to communicate with an ordinary person. A simple feature extraction method based on the area of the objects in a binary image and Discrete Cosine Transform (DCT) is proposed for extracting the features from the video sign language. A simple neural network models is developed for the recognition of gestures using the features computed from the video stream. An audio system is installed to play the particular word corresponding to the gestures. Experimental results demonstrate that the recognition rate of the proposed neural network models is about 91%.
Conference Paper
Full-text available
The pixel-based shape representation has been sensitive to rotation. In this paper, we propose a pixel-based descriptor that is invariant with rotation and scale for the hand shape recognition in Taiwanese Sign Language (TSL). Based on the property that a hand shape is characteristic of a unique pointing direction, angle normalization is used to meet the rotation-invariant requirement. With angle normalization, the traces of class covariance matrices have been reduced almost all over the classes of hand shapes, implying a less overlap between classes. It is confirmed by the experiments that show an increase in recognition accuracy.
Conference Paper
Full-text available
It is difficult to recognize sign language in different viewpoint. The HMM method is hindered by the difficulty of extracting view invariant features. The general template matching methods have a strong constraint such as accurate alignment between the template sign and the test sign. In the paper, we introduce a novel approach for viewpoint invariant sign language recognition. The proposed approach requires no view invariant features, low training and no alignment. Its basic idea is to consider a sign as a series of tiny hand motions and utilize the HOMOGRAPHY of tiny hand motions. Using the word of "homography", we mean that there are the same tiny hand motions as well as their appearance order in different performances of the same sign. The experimental results demonstrate the efficiency of the proposed method.
Conference Paper
Full-text available
Sign language is used for communicating to people with hearing difficulties. Recognition of a sign language image sequence is challenging because of the variety of hand shapes and hand motions. We propose a method to automatically construct a transitional structure(topology) of a Hidden Markov Model(HMM) for recognizing sign language words. Unlike conventional HMM, the constructed topology has branches and junctions in order to represent a flexible structure. The proposed method consists of segmentation of a motion, and construction of the topology from segments. The topology is constructed from an initial topology by modifying it. With experiments, we show the effectiveness of the proposed method.
Article
Full-text available
To recognize speech, handwriting or sign language, many hybrid approaches have been proposed that combine Dynamic Time Warping (DTW) or Hidden Markov Models (HMM) with discriminative classifiers. However, all methods rely directly on the likelihood models of DTW/HMM. We hypothesize that time warping and classification should be separated because of conflicting likelihood modelling demands. To overcome these restrictions, we propose to use Statistical DTW (SDTW) only for time warping, while classifying the warped features with a different method. Two novel statistical classifiers are proposed (CDFD and Q-DFFM), both using a selection of discriminative features (DF), and are shown to outperform HMM and SDTW. However, we have found that combining likelihoods of multiple models in a second classification stage degrades performance of the proposed classifiers, while improving performance with HMM and SDTW. A proof-of-concept experiment, combining DFFM mappings of multiple SDTW models with SDTW likelihoods, shows that also for model-combining, hybrid classification can provide significant improvement over SDTW. Although recognition is mainly based on 3D hand motion features, these results can be expected to generalize to recognition with more detailed measurements such as hand/body pose and facial expression.
Conference Paper
In this paper, an image processing algorithm is presented for the interpretation of the Taiwanese sign language, which is one of the sign languages used by the majority of the deaf community. The process involves two layer classifications. At first, coarse classification is done according to detection of hand motion and tracking the hand location and second classification is based on key frame selection and hand shape recognition of key frames. Motion history image and Fourier descriptor are used for motion direction recognition and key frame selection respectively. Generic cosine descriptor (GCD) has been proposed for feature extraction of hand postures. GCD is invariant to scale, translation and rotation of hand shapes. Our system tests 15 different hand gestures of 10 persons. The experimental results show that our system can achieve 100% recognition rate for test persons.
Conference Paper
This work describes an isolated sign language recognition (SLR) system that combines features from a video camera and an instrumented glove. Various combinations of features were tested on American Sign Language (ASL) vocabularies ranging from 10 to 200 words. The most accurate feature vector set included all available camera and glove features.
Sign Language Recognition Based on Position and Movement Using Multi-Stream HMM
  • Masaru Maebatake
  • Iori Suzuki
Masaru Maebatake, Iori Suzuki, "Sign Language Recognition Based on Position and Movement Using Multi-Stream HMM ", Proceedings of the 2008 Second International Symposium on Universal Communication, pp. 478-481, 2008.
Pattern Matching for Automatic Sign Language Translation System using LabVIEW
  • Andreas Domingo
  • Rini Akmeliawati
Andreas Domingo, Rini Akmeliawati, et al., "Pattern Matching for Automatic Sign Language Translation System using LabVIEW ", International Conference on Intelligent and Advanced Systems, 2007, ICIAS 2007, pp. 660-665, 2007.