Conference PaperPDF Available

Proposed framework of gesture sign data acquisition and recognition in context with Indian sign language

Authors:
  • Director, IIIT Kottayam, Kerala, India Institute of National Importance

Abstract and Figures

The major challenge that faces Indian Sign Language recognition now is developing methods/processes that will scale well in a better shape with increasing research findings by various researchers and supports by various institutions & government organization working for the standardization of sign language. Sign language recognition is best expresses through the gestures. Gesture is the most important to exchange ideas, messages, thoughts etc among deaf and dumb people. Gestures are a basic concept of communication and were used by humans even before speech developed; they have the potential to be a huge fortification to an intuitive human-computer communication. In this work, we would like to present a framework to support a simple step by step approach to research, analysis and designing sign gesture-based data acquisition and recognition. We would like to propose sign gesture-based data acquisition framework for collecting input from multiple input sources. This framework further distils the multiple input sources into a generic engine from which the input is read and processed by a gesture data acquisition processor, an abstraction of data processing functionality such as parsing and sign gesture analysis will be done. Further it will be outputted into various possible options. This framework will supports and keep track of multiple sources input gestures including physical gestures, methods/approaches for processing mechanism, output technologies and user goals-as the basis from which the framework may extends as per the application requirements. Each section will be well defined and tested using pragmatic experiments supported with qualitative approaches and methods. The framework is intended as a helping aid to guide research, analysis and design, and presents a generic skeleton for providing a theoretical understanding of gesture sign data acquisition, processing and recognition. Further a simple method of codeword assignment with features like angle, orientations are identified and a theoretical finding about recognition is discussed.
Content may be subject to copyright.
1 NCILC 2013
Proposed framework of gesture sign data acquisition
and recognition in context with Indian sign language
Pravin R Futane
Research Scholar, dept. of Computer Science
Research Center, Amravati University
Amravati, India
pravinfutane9@gmail.com
Dr. R.V. Dharaskar
Dept of Computer Engineering
Ramanand Teerth Marathwada University
Nanded, India
rvdharaskar@rediffmail.com
AbstractThe major challenge that faces Indian Sign Language
recognition now is developing methods/processes that will scale well
in a better shape with increasing research findings by various
researchers and supports by various institutions & government
organization working for the standardization of sign language. Sign
language recognition is best expresses through the gestures. Gesture
is the most important to exchange ideas, messages, thoughts etc
among deaf and dumb people. Gestures are a basic concept of
communication and were used by humans even before speech
developed; they have the potential to be a huge fortification to an
intuitive human-computer communication.
In this work, we would like to present a framework to support a
simple step by step approach to research, analysis and designing sign
gesture-based data acquisition and recognition. We would like to
propose sign gesture-based data acquisition framework for collecting
input from multiple input sources. This framework further distils the
multiple input sources into a generic engine from which the input is
read and processed by a gesture data acquisition processor, an
abstraction of data processing functionality such as parsing and sign
gesture analysis will be done. Further it will be outputted into various
possible options. This framework will supports and keep track of
multiple sources input gestures including physical gestures,
methods/approaches for processing mechanism, output technologies
and user goals—as the basis from which the framework may extends
as per the application requirements. Each section will be well defined
and tested using pragmatic experiments supported with qualitative
approaches and methods. The framework is intended as a helping aid
to guide research, analysis and design, and presents a generic
skeleton for providing a theoretical understanding of gesture sign data
acquisition, processing and recognition. Further a simple method of
codeword assignment with features like angle, orientations are
identified and a theoretical finding about recognition is discussed.
Keywords—Indian sign language, framework, data
acquisition
INTRODUCTION
The Sign language is a general term that refers to any
gestural / visual language that makes use of specific shapes and
movements of the fingers, hands and arms, as well as
movements of the eyes, face, head and body. The sign
languages are complete natural languages, with their own
syntax and grammar. There is no international recognized and
standardized sign language for all deaf people especially Indian
sub continents whereas internationally many of sign language
recognition are in place such as American Sign Language,
British Sign Language, Isarel sign language, French Sign
language, even Chinese and Pakistan sign language..etc. As per
India is concerned the work is progressing as many government
organization, NGO’s and IIT Guwahati are working on similar
project. There are many varieties of sign language in the
region, including many pockets of home sign and informal sign
languages. There is no consensus regarding which of these
varieties constitute dialects of a language or separate
languages, but several researchers have identified relatedness
between the sign languages used in urban regions of India,
Pakistan and Nepal. It is unknown whether this group is related
to other languages of the subcontinent such as sign languages
in Bangladesh or Sri Lanka. Several researchers have explored
their possibilities and have achieved results to certain extent,
but progress in the recognition of sign language, as a whole
has been limited which gives us an edge to go ahead in this
domain. Although the Deaf problem is very severe in India but
not to that extend the efforts & work had been carried out
which leaves lots of scope for research in this area & has a
social relevance and open up opportunity for us to do for the
nation. The study reveals that it is quite feasible and will act as
a guiding tool to all researchers in the sign gesture language
acquisition and recognition domain.
RELATED STUDY OF INDIAN SIGN LANGUAGE
Indo-Pakistani Sign Language (IPSL) is the predominant
sign language variety in South Asia, used by at least several
hundred thousand deaf signers (2003).[1]
Reference [3] has discussed about the Indian sign language
(ISL), its history and the progress made in the development of
ISL. The details of the deaf population in India over the last
five decades are given in the Table. I showing the number of
deaf people in India, neither the 1991 nor the 2001 census
included disabilities; hence they were obtained by extrapolating
the 1981 data.[2]
1 NCILC 2013
TABLE ISHOWING NUMBER OF DEAF PEOPLE IN INDIA [2,3]
Year Source Data (total deaf
population in India)
1970 Taylor and taylor (1970) 2 million
1981 GOI, Ministry of Social welfare 1981 6,315,761
1991 Gopinath (1998) 7,770,753
2001 Vasishta (2001) 14 million
2011 UN report 18 million
As per the SIL Electronic Survey Reports 2008-006, the
authors have done the study on Assessment of regional
language varieties in Indian Sign Language. They have
examined in five cities to understand and prioritizing Indian
sign language literature development with reference to ISL
diversity, vitality and identity. The related finding is
summarized in table II. They have carried this survey with
lexical similarity analysis, dialect intelligibility testing with
recorded text tests (RTTs) and language attitude assessment
and resulted in one language with many dialects.[2]
TABLE II SHOWING FINDINGS OF VARIETIES ON INDIAN CONTINENTS[2]
Varieties Findings
Mumbai (Bombay)
most appropriate for initial literature
development,
having the highest prestige and closest
lexical similarity,
moderately high dialect intelligibility
Hyderabad
Hyderabad and Chennai were most
closely related with each other
followed by Hyderabad and Mumbai
Chennai (Madras)
should have literature developed in
them
Kolkata’s (Calcutta)
the least similar dialect followed by
Chennai
should have literature developed in
them
Through our study, it is clear that the Standard sign
language for the deaf in India will soon be done and many
organizations including Ramakrishna mission project is
addressing to it. The article from Hindustan Times, India dated
September 16, 2004 is discussed herewith to have more
understanding of this deaf problem [4].
In the absence of a standard sign language for the deaf in
India, one of the oldest associations of the deaf in the country
has taken it upon itself to formulate a comprehensive set of
symbols that could be adopted as the standard sign language of
the country.
"Signs differ from place to place and even from person to
person in the absence of standard signs," says DS Chauhan,
Honorary Secretary of the Delhi Association of the Deaf. As a
result two deaf persons from different parts of the country find
it difficult to communicate with each other. The Association,
in collaboration with the Jawaharlal Nehru University (JNU) is
in the process of setting up a cell to create a common sign
language and having petitioned the National Human Rights
Commission (NHRC) on the issue, is also preparing a report
on the issue to be submitted to President APJ Abdul Kalam.
[4]
PROPOSED FRAMEWORK
Following figure 1. depicts the framework proposed for
acquisition and recognition of sign gestures especially catering
to the needs of Indian community. The first module focused on
sign gesture data acquisition. Here the gesture data is acquire
from different input sources in the form of flat files having
videos and images or from devices such as cameras, digital
pen..etc. The different types through which the data is acquired
can be of following types such as vision based, device based or
the hybrid approach. In vision based the bare skin hand is used
and with image processing techniques the data will be
preprocessed before providing it to the gesture controller. In
device based it can be acquired through the handmade colored
gloves with finger tips colored and then some image processing
techniques and algorithms will be used and then made to pass
through the controller. The other option is to make use of
sensored equipped gloves with sensors at every finger. The data
acquired through this mean will be accurate but the sensor
gloves are too costly. The last type will be a hybrid approach
which can have a mix of device and vision approach used. So
the sign gesture data acquisition will have following gesture
descriptor such as type, mode and source. The type can be
classified as vision based, device based or the hybrid approach.
The mode can be stored or in real time and the source may be
the flat files in the form of video or images or might be
captured from the cameras.
Figure 1 Proposed framework for Sign gesture acquisition and recognition
Once the data is acquired, it needs to be preprocessed to
bring it in the unique form of fixed size irrespective of input
sources size, resolution and making it invariant to scale,
rotation and translation. So the common data structure and
algorithms such as skin hand localization and segmentation and
filtering techniques will be applied and specialized algorithms
1 NCILC 2013
will be used which will be specific to the application areas
where the problem domain will be addressing.
The controller will be the heart of this architecture through
which the system control the interfacing from the input,
processing algorithms, data structures used and outputting onto
the text or to speech. This module will take the preprocessed
input from any input sources and then hand feature is extracted,
normalizing it and then training the system with the ISL dataset
will be done. This may be stored in the database (if required).
The system will be tested with the testing ISL dataset and
system will output the recognizer module result in the form of
either text or in audio form.
PROPOSED SIMPLE CODEWORD ASSIGNMENT METHOD
In this approach a simple method based on shape features
and code word assignment is proposed. Sign gesture dataset
was taken of Indian sign language but focused on static
alphabet gestures. These alphabet gestures are made up of
single hand (08 gestures) and double hand (18 gestures) as
shown below in figure 2:
Figure 2. Categorization of ISL alphabet static gestures used in our proposed
approach.
The categorization is done as follows:
a) Single/Double hand categorization – by area
calculation
b) Front/Back hand categorization by skin area pixel
intensity
c) Curveness / Straightness categorization – by slope
d) Further classification based on shape whether
polygon, ellipse...etc, orientation such as vertical,
horizontal and with the finger open/close.
The code word is assigned by the simple logic pattern
arranged left to right in a fashion as shown below
Code word = Category a | Category b | Category c |
Category d
Code is assigned 0 if on left side and 1 on right side
hierarchy level wise and category wise 0 and 1 for single and
double for category a and so on for other categories as well.
RESULTS AND DISCUSSION
The dataset used was of Indian sign language which we
have created for single hand alphabets and the theoretical
findings with reference to the approach discussed in the paper
have been resulted as follows:
TABLE III THEORETICAL RESULTS FINDING FOR SINGLE HANDED ISL
GESTURES.
Sr.
No Gesture
sign Code word Remarks
1 C 0101 + other parameters Single handed
ISL alphabet
gestures
2 I 0110 + other parameters
3 J 0101 + other parameters
4 L 0111
5 O 0100
6 U 0110 + other parameters
7 V 0110 + other parameters
8 W 0110 + other parameters
The code word assignment is done as per the pattern
discussed earlier and we got 100 % accuracy for the { L } and
{ O } gesture sign of single handed alphabets(total 08 sign)
and for rest of gestures such as {C,J} having common code
assign and {I,U,V,W} having common code word need further
processing. So, the other simple features such as area, angle,
orientations, curveness and straightness are identified and
system will be trained accordingly. So the codeword resulted
in the following cluster set as { L, O, {C,J}, {I,U,V,W}}.
Similarly, the codeword assignment for double handed
gestures (total 18 sign) resulted in the cluster set as {
{A,B,D,K,P,Q}, Z, {M,N,R}, {E,F,I,S},{G,H,X,Y}} which
needs further feature extraction as code words can’t alone
enough to recognize. The same may be extended to the
number gesture dataset of ISL. Although at this moment we
don’t have achieved the desire results but in near future we
will definitely succeed.
CONCLUSION
Our research began with a review and analysis of the sign
gesture literature including Indian Sign Language whose
standardization is in process and had variants of region
language constructs, dialects…etc. Further, study of relevant
methods, literature; theoretically experimenting it with
thorough understanding of all the parameters, constraints; in-
1 NCILC 2013
depth analysis of method/approaches applies to sign
recognition; designing of sign gesture recognition systems and
its research insight findings, will definitely lead to the
development of the generic framework for sign language
recognition.
ACKNOWLEDGMENT
We would like to thank the BCUD, University of Pune for
supporting our work.
REFERENCES
[1] Vasishta, M., J. C. Woodward, and K. L. Wilson (1978). "Sign
language in India: regional variation within the deaf population".
Indian Journal of Applied Linguistics 4 (2): 66–74.
[2] Johnson, Jane E., Johnson, Russell J., “Assessment of regional
language varieties in Indian sign language”, SIL Electronic
Survey Reports 2008-006, pp 1-121.
[3] Pravin R.Futane, Rajiv V. Dharaskar, “Video gestures
identification and recognition using fourier descriptor and
general fuzzy minmax neural network for subset of Indian sign
language”, in proceedings of IEEE 12th International conference
on Hybrid Intelligent system, Pune India Dec’2012. in press.
[4] Article on “Standard sign language for the deaf in India soon”,
Hindustan Times, Press Trust of India, New Delhi, India - Sep
16, 2004
ResearchGate has not been able to resolve any citations for this publication.
Conference Paper
Sign languages are natural languages that use to communicate with deaf and mute people. There exist different sign languages in the world. But we focused on Indian Sign Language which is on the way of standardization & very less work has been done on it so far. We have focused on Indian sign language history and progress in this domain and work carried out by various researchers in Indian Sign language recognition. Also we have proposed an approach that will convert the video of full sentence gesture of Indian sign language to text. It will initially identify individual words from the video & convert them on to text. Finally, the system will process those words to form a meaningful sentence in compliance with the simple grammar rules.
Sign language in India: regional variation within the deaf population
  • M Vasishta
  • J C Woodward
  • K L Wilson
Vasishta, M., J. C. Woodward, and K. L. Wilson (1978). "Sign language in India: regional variation within the deaf population". Indian Journal of Applied Linguistics 4 (2): 66-74.