A SRN/HMM system for signer-independent continuous sign language recognition

Abstract

Sign language recognition is to provide an efficient and accurate mechanism to transcribe sign language into text or speech. State-of-the-art sign language recognition should be able to solve the signer-independent continuous problem for practical applications. A divide-and-conquer approach, which takes the problem of continuous Chinese Sign Language (CSL) recognition as subproblems of isolated CSL recognition, is presented for signer-independent continuous CSL recognition. In the proposed approach, the improved simple recurrent network (SRN) is used to segment the continuous CSL. The outputs of SRN are regarded as the states of hidden Markov models (HMM) in which the Lattice Viterbi algorithm is employed for searching for the best word sequence. Experimental results show that the SRN/HMM approach has a better performance than the standard HMM.

Full text

Page 1

A SRN/HMM System for Signer-independent Continuous
Sign Language Recognition
Gaolin Fang1, Wen Gao1,2
1Department of Computer Science and Engineering,
Harbin Institute of Technology, Harbin, 150001 China
2Institute of Computing Technology,
Chinese Academy of Sciences, Beijing, 100080, China
{fgl, wgao}@ict.ac.cn
Abstract
Sign language recognition is to provide an efficient
and accurate mechanism to transcribe sign language into
text or speech. State-of-the-art sign language recognition
should be able to solve the signer-independent continuous
problem for practical applications. A divide-and-conquer
approach, which takes the problem of continuous Chinese
Sign Language (CSL) recognition as subproblems of
isolated CSL recognition, is presented for signer-
independent continuous CSL recognition in this paper. In
the proposed approach, the improved simple recurrent
network (SRN) is used to segment the continuous CSL.
The outputs of SRN are regarded as the states of hidden
Markov models (HMM) in which the Lattice Viterbi
algorithm is employed for searching the best word
sequence. Experimental results show that SRN/HMM
approach has better performance than the standard HMM.
1. Introduction
Sign language as a kind of gestures is one of the most
natural means of exchanging information for most deaf
people. The aim of sign language recognition is to provide
an efficient and accurate mechanism to transcribe sign
language into text or speech so that communication
between deaf and hearing society can be more convenient.
Sign language recognition has emerged as one of the most
important research areas in the field of human-computer
interaction. In addition, it has many other applications,
such as controlling the motion of a human avatar in a
virtual environment (VE) via hand gesture recognition,
multimodal user interface in virtual reality (VR) system.
Attempts to automatically recognize sign language
began to appear in the literature in the 90’s. Previous work
on sign language recognition focuses primarily on finger
spelling recognition and isolated sign recognition. There
has been very little work on continuous sign language
recognition. Starner [1] used a view-based approach with
a single camera to extract two-dimensional features as
input to HMMs. The correct rate was 91% in recognizing
the sentences comprised 40 signs. By imposing a strict
grammar on this system, an accuracy of 97% was possible
with real-time performance. Liang and Ouhyoung [2] used
HMMs for continuous recognition of Taiwan Sign
language with a vocabulary between 71 and 250 signs
with Dataglove as input devices. However, their system
required that gestures performed by the signer be slow to
detect the word boundary. Vogler and Metaxas [3] used
computer vision methods to extract the three-dimensional
parameters of a signer’s arm motions, coupled the
computer vision methods and HMMs to recognize
continuous American sign language sentences with a
vocabulary of 53 signs. They modeled context-dependent
HMMs to alleviate the effects of movement epenthesis. An
accuracy of 89.9% was observed. In addition, they used
phonemes instead of whole signs as the basic units [4],
experimented with a 22 sign and achieved recognition
rates similar to sign-based approaches. A system has been
described in our previous work [5], which used Dataglove
as input device and HMMs as recognition method. It can
recognize 5177 isolated signs with 94.8% accuracy in real
time and recognize 200 sentences with 91.4% word
accuracy.
As the previous work showed, most researches on
continuous sign language recognition were done within
the signer-dependent domain. No previous work on
signer-independent continuous sign language recognition
was reported in the literature. For continuous sign
language recognition, a key problem is the effect of
movementepenthesis,that
between signs, which vary with the context of signs. For
signer-independent recognition, different signers vary
their hand shape size, body size, operation habit, rhythm
and so on, which bring about more difficulties in
recognition.Thereforerecognition
independent domain is more challenging than in the
is,transientmovements
inthesigner-
Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition (FGR’02)
0-7695-1602-5/02 $17.00 © 2002 IEEE

Page 2

signer-dependent one. Signer-independent continuous sign
language recognition is first investigated in this paper.
Our experiments show that continuous sign language has
the property of segments, so a divide-and-conquer
approach is proposed for continuous CSL recognition. It
divides the problem of continuous CSL recognition into
the subproblems of isolated CSL recognition. The SRN
modified to assimilate context information is regarded as
the segment detector of continuous CSL. The Lattice
Viterbi algorithm is employed to search the best word
sequence path in the output segments of SRN.
The organization of this paper is as follows. In Section
2 we propose the improved SRN and SRN-based
segmentation for continuous sign language. In Section 3
we introduce the Lattice Viterbi algorithm in the HMM
framework and discuss the computation of segment
emission probability. In Section 4 we show experimental
results and comparisons. The conclusion is given in the
last section.
2. SRN-based segmentation
2.1.
improvement
Simple recurrentnetworkandits
Elman proposed a simple recurrent network[6] in 1990
on the basis of the recurrent network described by Jordan.
Networks have the dynamic memory performance because
of the introduction of recurrent links. They have been
successfully applied to speech recognition [7], [8],
handwriting recognition [9], isolated sign language
recognition [10]. The SRN in Figure 1 has four units. The
input units receive the input vector
tI at time t , and the
H , feedback
WW
,
corresponding outputs are hidden units
t
units
the weight matrices of feedback units to hidden units,
input units to hidden units and hidden units to output units,
respectively.
In the network the feedback units is connected one-to-
one corresponding hidden units through a unit time-delay.
≥

t
The input units and the feedback units make up of the
combined input vector. Φ Ψ
,
hidden units, output units.
(
⋅+⋅=
tCtt
WIWCfH
(
−⋅=
Htt
WHfO
The activation function
f ⋅ is the standard sigmoid
( )
(
f xe
=+
1
. The errors between the
t C , output units
t
O . Defining
O
H
H
I
H
C
W
,
as
1
2
5 . 0
1
=


=
−
tH
C
t
t
(1)
is respectively the bias of
)
Φ−
I
)
Ψ
( )
HH
(2)
O
(3)
one,
)
x
−
−
1
network’s outputs and the targets are propagated back
using the generalized delta rule [11].
Figure 1. Simple recurrent network

Because of the introduction of feedback units, the
outputs of network depends not only on the external input
but also on the previous internal state which relies on the
result of the preceding all external inputs. So the SRN can
memorize and utilize a relatively larger preceding context
[6].
The SRN can memorize the preceding context, but it
cannot utilize following context information. Thus the
SRN is modified to efficiently utilize the following
context information in two ways. One is that the following
context vector is regarded as one of the input. Thus the
input vector is redefined as
calculations are the same as the standard SRN. The other
is that training samples are respectively in turn and in
reverse turn fed into the SRN with the same architecture,
and then one forward SRN and one backward SRN are
trained. So the context information can be assimilated
through two SRNs. The experiment of segmentation of
continuous sign language is performed in two improved
ways. But in the latter the outputs of the forward SRN
conflict with those of the backward SRN so that
experimental results are inferior to the former. Thus, we
employ the former method as the improved SRN.
[]
1
+
=
ttt
III
, the rest of
2.2. Segmentation of continuous sign language
Input preprocess: Two 18-sensor Datagloves and
three position trackers are used as input devices. Two
trackers are positioned on the wrist of hand and another is
fixed at back (used as the reference tracker). The
Datagloves collect the variation information of hand
shapes with the 18-dimensional data at each hand, and the
position trackers collect the variation information of
orientation, position, movement trajectory. The data from
position trackers can be converted as follows. The
reference Cartesian coordinate system of the trackers at
. . .
. . .
. . .. . .
Feedback
Output
Hidden
Input
Copy
O
H
W
H
C
W
H
I
W
O
C
I
Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition (FGR’02)
0-7695-1602-5/02 $17.00 © 2002 IEEE

Page 3

back is chosen, and then the position and orientation at
each hand with respect to the reference Cartesian
coordinate system are calculated as invariant features.
Through this transformation, the data are composed of
three-dimensional position vector and three-dimensional
orientation vector for each hand. Furthermore, we
calibrate the data of different signers by some fixed
postures because everyone varies his hand shape size,
body size, and operation habit. The data form a 48-
dimensional vector in total for two hands. However, the
dynamic range of each component is different. Each
component value is normalized to ensure its dynamic
range is 0-1.
Original experiment regards the 48-dimensional data as
the input of SRN, chooses 30 hidden units and 3 output
units. Training this network costs 28 hours in the
PIII450(192M Memory) PC. But the result is not satisfied
and the segment recall is only 87%. This approach is
unsuccessful in the scalability and accuracy. Here, we
only considers the case of detecting all actual segments
and doesn’t care the case of detecting several segments for
an actual segment. The latter will be solved by the Lattice
Viterbi algorithm searching in the HMM framework.
%100
segmentsactual allof Number
segmentscorrectofNumber
recallSegment
×=
(4)
Thus the above approach is modified. We use self-
organizing feature maps(SOFM) as the feature extraction
network. The outputs of SOFM have the strong segment
properties (Figure 2 shows an example). The output of
SOFM is regarded as the input of SRN by the encoding.
We select 48 input units and 256 output units for the
SOFM, and 16 input units and 15 hidden units and 3
output units for the SRN through trial and error. Training
the SRN costs 45 minutes in the PIII450(192M Memory)
PC. The segment recall is 98.8%. The input, output,
training and recognition of SRN will be discussed in the
following section.
?
??
???
???
???
???
???
???????? ???? ?? ??
Figure 2. The segments property of sign
language “???????”(when will we leave)
Input: The encoding of 256 output units of SOFM
requires 8 units, and the introduction of the following
context also demands 8 units. In total there are 16 input
units. The value of the input
Output: We define 3 output units: the left boundary of
segments 1, the right boundary of segments 2, the interior
of segments 3, and the corresponding units
{ } 1 , 0
∈
i
tI
,
16, , 2 , 1
=
L
i
.
1
to ,
2
to ,
3
to .
[
[
[
]
]
]



=
100
010
001
t O
(5)
Training: We cannot straightforward find the target
segments because sign language is continuous. Thus
automatic segmentation is employed to find the target
segments. Let the sample sentence in the training
wwwW
L
21
=
, the corresponding frame sequence
lt t tT
L
21
=
, we decide frame
or
1
+
m
w
, and if
mi
wt ∈
,
i
t
the right boundary of segments and frame
k
it belongs to word
+∈
m
w
, then frame
m
w
11
+
it is
1
+ it
is the left
boundary of segments. Each state probability of frame
w (
m
approach of isolated sign language recognition with the
isolated sign language model parameters (see Section 4).
Then the constrained Viterbi algorithm is used to search
the best segment sequence. The constrained refers to the
search path followed the sequence
regard the segment sequence result as the target output.
Back-propagation through time is introduced as the
learning algorithm of SRN. The samples in the training set
are transformed by the SOFM, and the SOFM outputs by
the encoding together with the following ones are fed into
the SRN, then the errors between the SRN outputs and the
targets are propagated back using back-propagation and
changes to the network weights are calculated. At the
beginning of learning, the weight matrices, the bias of
hidden units and output units are initialized the random
value (
, 1+−
, the feedback units are initialized to
activations of 0.5.
Recognition: Continuous sign language in the test set
is firstly fed into the SOFM. The quantized outputs are
formed by the feature extraction of SOFM. The SOFM
outputs by the encoding together with the following ones
are fed into the SRN. The segmentation result of SRN is
)( maxarg
t
i
the left and the right boundary of segments is used as
constraint in the segmentation.
it
belonging to word
m
k
L
1
=
) is calculated by the
k
www
L
21
. We
) 1
*
i
oi =
at time t . The adjacency property of
Output is 1
Output is 2
Output is 3
Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition (FGR’02)
0-7695-1602-5/02 $17.00 © 2002 IEEE

Page 4

3. HMM Framework
The segmentation result of SRN is fed into the HMM
framework. The state of HMM for one word may cover
one or several segments (2?4). Because different signs
vary their lengths and may be composed of several
segments, we should search the best path in those
segments through the Lattice Viterbi algorithm. The best
refers to two respects: the recombined segments sequence
is the best, and the word sequence selected from the best
segment sequence is the best. The standard Viterbi
algorithm is modified in order to adapt itself to the search
on lattices, and the modified algorithm is referred to as the
Lattice Viterbi algorithm.
To illustrate more conveniently, we define two Viterbi
algorithms: one is the isolated sign language Viterbi
algorithm. For the data frame sequence of the input which
is known as only one word component in advance, the
algorithm will search all states of the word for each frame
in recognition, and get the best state sequence.
The other is the continuous sign language Viterbi
algorithm. For the data frame sequence of the input whose
component cannot be known beforehand, the algorithm
will search not only all states of this word but also the
states of the rest words for each frame in recognition, and
get the best state sequence. It requires more time and has
less accuracy than the isolated sign language Viterbi
algorithm in recognizing the same data frame sequence.
Both the isolated sign language Viterbi algorithm and
the continuous sign language Viterbi algorithm in essence
belong to the standard one, because they search the frame
one by one. But the Lattice Viterbi algorithm that is
different from the standard one can span one or more
segments to search. It will be discussed in detail in the
following section.
3.1. Lattice Viterbi algorithm
We define an edge as a triple
segment t , ending at segment t′ and representing word
q ? where
Tt <≤
0
,
>
′
<
qtt
,,
form the set L . We introduce accumulator
),,(
qtt ′
δ
that collects the maximum probability of
covering the edge
>
′
<
qtt
,,
path, we define the auxiliary argument
previous triple argument pointer of the local maximum
),,(
qtt ′
δ
. We denote
b
probability of word q covering segments from position t
to t′.
)|(
qqP
′ is defined as the transition probability
from word q′ to q , which is estimated in some 3000
millionChinesewordscorpus
>
′
<
qtt
,,
, starting at
Ttt
≤
′
<
.Alltriple
. To keep track of the best
(
t ′
ψ
),,
qt
as the
),,(
qtt
′
as the emission
fromtheChinese
newspapers in the 1994-1995 year. The Lattice Viterbi
algorithm is as follows.
1) Intialization:
),, 0 (),, 0 (
qtbqt
=
δ
NULLqt
=
), , 0 (
ψ
2)Recursion:
max),,(
,,
Lqtt
>∈
′′ ′
<
(6)
(7)
),, (
tb
)|(),,(
qtqqPqttqtt
′′′ ′ ′
=
′
δδ
(8)
(9)
)|(),,(max
, ,
qt
>∈
′ ′ ′
arg
t
<
),,(
qqPqttqtt
L
′′ ′ ′
=
′
δψ
3) Termination:
*
P
=
),,(max
,,
qTt
,
1
q
qTt
L
δ
>∈
>=
<
(10)
),,(max
,
q
>∈
arg
t
<
,
,
**
1
qTtTt
LT
δ
<
(11)
4) Path backtracking:
Let
T =
,,
1
iii
qtt
+
qtt
kk
+
1
,,
*
1
qqkL
is the best path.
*
0t
>=
>=
1
, we
,
t
iterate
)
i
the function
,(
**
i
1
*
i
*
1
**
qt
−+
*
<
<
ψ
until
NULL
k
+
**
, and the word sequence
*
3.2.
probability
Computation ofsegment emission
The computation of segment emission probability is
similar to the one of isolated sign language recognition.
But it stores the probabilities of all possible candidate
words that are looked on as the emission probability
),, (
qttb
′
in the HMM framework. We get the best path
through the Lattice Viterbi algorithm search at last.
Figure 3. The architecture of SOFM/HMM
The SOFM/HMM method is employed for isolated
sign language recognition [12]. This method uses an
alternative probability density function (pdf) scheme that
each SOFM eigenveter centroid is regarded as one of the
components in the state of HMMs. And this component
forms the state pdf in term of the weighted sum. Then the
1
?
Ot
?
wjmi
cjm
m
2
?
Ot
?
3
?
Ot
?
Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition (FGR’02)
0-7695-1602-5/02 $17.00 © 2002 IEEE

Page 5

state pdf can be calculated by the Forward-Backward
Procedure(or by the Viterbi algorithm). SOFM weights
are iteratively updated in the supervision of computed
state pdfs. The parameters of SOFM and HMM are re-
estimatedthroughthe
algorithm. In this way we combine the powerful self-
organizing capability of SOFM with excellent temporal
processing properties of HMM so that we improve the
performance of HMM-based sign language recognition
systems. The architecture of SOFM/HMM sees Figure 3.
Expectation-Maximization
4. Experiments and Comparisons
The data are collected from 7 signers with each
performing 208 isolated signs 3 times. The vocabulary is
words from elementary textbooks of 1-2 grades for
Chinese deaf pupil. We at random select 5 from 7 signers,
and then select 2 group data from each signer. In total 10
group data are regarded as the isolated sign language
training set. The isolated sign language model parameters
are trained through the SOFM/HMM method with the
isolated sign language training set. The rest one group
data in 5 signers are referred to as the registered test set
(Reg.). The data from 2 signers are referred to as the
unregistered test set (Unreg.). The results of isolated sign
language recognition see Table 1. We select 2 from 5
signers in the isolated sign language training set,
respectively represented with A, B, and select 1 from the
rest 2 signers represented with C. There are 3 signers in
total. The 100 continuous sentences composed of a
vocabulary of 208 signs are respectively performed twice
by 3 signers with the natural speed. There are 6 group data
marked with A1, A2, B1, B2, C1, C2.We choose A1, B1as
the training set which is used in the SOFM, SRN and
embedded training. One of A2, B2is referred to as the
registered test set (Reg.), and one of C1, C2is referred to
as the unregistered test set (Unreg.). We compare the
performances of SRN/HMM with those of HMM in
signer-independent continuous sign language recognition
and the results see Table 2 and Table 3.
Table 1. The isolated sign language
recognition results
Signer
S1
S2
Reg.S3
S4
S5
Mean
S6
S7
Unreg.
Mean
SOFM/HMM
98.6%
95.2%
96.7%
91.3%
94.7%
95.3%
88.5%
88.0%
88.2%
Table 2. The continuous sign language
recognition results in Unreg.
Method
HMM
SRN/HMM
Recognition accuracy (%)
81.2 (S=35, I=25, D=9)
85.0 (S=33, I=6, D=16)
Table 3. The continuous sign language
recognition results in Reg.
Method
HMM
SRN/HMM
Recognition accuracy (%)
90.7 (S=13, I=18, D=3)
92.1 (S=12, I=5, D=12)
All experiments are performed with the Bigram
language model in the PIII450(192M Memory) PC. S, I
and D denote respectively the error number of substitution,
insertion and deletion. The total number of signs in the
test set is 367. Compared with the standard HMM, the
SRN/HMM has higher recognition accuracy. The possible
reasons for the results are as follows. Firstly, the HMM
uses the continuous sign language Viterbi algorithm which
is liable to be influenced by the movement epenthesis, but
the SRN/HMM alleviates the effects of movement
epenthesis by discarding the transient frames near the
sentences segmentation judged according to the output of
SOFM. Secondly, unlike the HMM which searches the
best state sequence, the SRN/HMM gets the best word
sequence. Thirdly, the SRN/HMM employs the isolated
Viterbi algorithm that can get the higher accuracy than the
continuous Viterbi algorithm. However, the divide-and-
conquer approach in the SRN/HMM may lead to the
accumulation of errors. Thus we introduce the soft-
segmentation instead of the fixed-segmentation in the
SRN segmentation and decide the boundary of words in
the Lattice Viterbi algorithm so as to improve the
performance of SRN/HMM.
5. Conclusion
While theHMM employstheimplicitword
segmentation procedure, we present a novel divide-and-
conquer approach which uses the explicit segmentation
procedureforsigner-independent
recognition, and which, unlike the HMM, isn’t liable to
the effects of movement epenthesis. The divide-and-
conquer approach is as follows. Firstly, the improved SRN
is introduced to segment the continuous sentences, and the
results of segmentation are regarded as the state inputs of
HMM. Secondly, the best word sequence is gotten through
the search of Lattice Viterbi algorithm in the sentence
segments.This approach
movement epenthesis, and experimental results show that
it increases the recognition accuracy. In addition, the
continuousCSL
alleviatesthe effectsof
Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition (FGR’02)
0-7695-1602-5/02 $17.00 © 2002 IEEE