A Novel Approach to Separate Handwritten Connected Digits

Abstract

This paper presents a novel approach to separate connected digits in handwritten numerals by employing two agents in the process. The first agent decides on candidate cut-point as the closest feature-point to the center of the deepest top-valley, if any. The second agent argues candidate cut-point as the closest feature-point to the center of the highest bottom-hill, if any. Then the actual cut-point is decided by negotiation, which is influenced by a degree of confidence in each candidate cut-point. Experiments conducted so far are promising and successful as well as justified employing multiple agents. The obtained results are very encouraging with a success factor of 97.8%.

Full text

Page 1

A Novel Approach to Separate Handwritten Connected Digits
Reda AlhajjAshraf Elnagar
Dept of Computer Science
University of Sharjah
Sharjah, U.A.E.
ashraf@sharjah.ac.ae
ADSA Lab & Dept of Computer Science
University of Calgary
Calgary, Alberta T2N 1N4, Canada
alhajj@cpsc.ucalgary.ca
Abstract
This paper presents a novel approach to separate con-
nected digits in handwritten numerals by employing two
agents in the process. The first agent decides on candi-
date cut-point as the closest feature-point to the center of
the deepest top-valley, if any. The second agent argues can-
didate cut-point as the closest feature-point to the center of
the highest bottom-hill, if any. Then the actual cut-point is
decided by negotiation, which is influenced by a degree of
confidence in each candidate cut-point. Experiments con-
ducted so far are promising and successful as well as jus-
tified employing multiple agents. The obtained results are
very encouraging with a success factor of 97.8%.
1. Introduction
This paper addresses an important and vital problem,
namely the separation of connected digits in handwritten
numerals. We mainly concentrate on separating pairs of
connected adjacent digits. The main idea is that no lim-
itations or boundaries are imposed on the handwriting of
people. While writing down numerals, it is possible to
have adjacent digits connected in several ways, including
single-touching, multiple-touching, long continuous touch-
ing, overlap, and may have some extra connecting stroke(s)
as noise.
Separating connected digits is being recognized as a seri-
ous and important task in the field of document image anal-
ysis and recognition, as evidenced by its pivotal role in nu-
merous applications. For instance, financial establishments,
such as banks and credit card firms, are looking for auto-
mated systems capable of reading and recognizing hand-
written checks and/or receipt amounts. Another application
could be seen in postal service departments to sort the in-
coming mail based on recognized handwritten postal zip
codes. In each of these examples, handwriting is uncon-
strained, leading to frequent undesired connection between
adjacent digits. To handle this, we utilize two agents. We
have been motivated by having different approaches lead-
ingto differentdecisionsaboutcandidatecut-point,depend-
ing on the perspective within which the problem is treated.
Also, agents have already proven to be very successful and
useful for many applications, e.g. [1].
Our approach can be summarized as follows. A hand-
written numeral is first scanned, normalized and thinned
by employing a parallel thinning algorithm; dark pixels
are marked as “1”. Feature points are extracted from the
thinned image. We differentiate between three types of fea-
ture points, namely end, branch, and cross points, which are
marked by replacing “1” in the thinned image by “3” for
the former and by“2” for the other two types. Then, two
agents get involved in the process. The first agent concen-
trates on the top part of the thinned image. It locates, as
candidate cut-point, the closest feature-point to the center
of the deepest valley. The other agent works from the bot-
tom side of the image. It locates, as candidate cut-point, the
closest feature-point to the center of the highest-hill. Af-
ter each of the two agents applies its own rules and derives
a candidate cut-point, the two agents negotiate and agree
on an actual cut-point. In most cases, both agents arrive at
the negotiation with candidate cut-points located along the
same vertical axis; one of them is peacefully chosen as the
actual cut-point. However, in case of two candidate cut-
points, each along a different vertical axis, both agents ne-
gotiate a compromise, which might not be any of the two
candidate cut-points. The decision is influenced by a degree
of confidence in each candidate cut-point. Experimental re-
sults showed that the negotiation process leads to a higher
success rate.
The rest of the paper is organized as follows. The two
agents are introduced in Section 2. Negotiation and conflict
resolution strategy is discussed in Section 3. Restoration
process is presented in Section 4. Experimental results are
reported in Section 5. Section 6 includes the conclusions
and future research directions.
2. The Employed Agents
Two agents have been employed in the process of sep-
arating connected numerals. The first agent is presented in
section 2.1. It concentrates on the top part of the numeral
and locates the deepest valley. The second agent is intro-
duced in Section 2.2, and concentrateson the bottom part of
the numeral to locate the highest hill.
2.1. Top-Valley Based Agent
In this section, we present the agent that locates a can-
didate cut-point by considering characteristics of the upper-
side of the thinned image. The basic idea is that segmen-
tation path lies somewhere close to the deepest top-valley
between two connected digits, if any. In other words, the
closest feature point to the center of the deepest valley lies
Proceedings of the Seventh International Conference on Document Analysis and Recognition (ICDAR 2003)
0-7695-1960-1/03 $17.00 © 2003 IEEE

Page 2

somewhere along the segmentation path. The success of
this agent depends on the availability of at least one valley
from the upper-side of the image. In case no such valley ex-
ists, this agent predicts the segmentation path to include the
closest feature point to the center of the image. It is iden-
tified as candidate cut-point with low degree of confidence.
This will be most probably resolved during the negotiation
with the other agent, which may report to the negotiation
process a candidate cut-point with a higher degree of confi-
dence. We elaborate more on the degree of confidence later
in this section.
The top-valley based agent works as follows. It traces
the contour of the thinned image of a handwritten numeral
string starting at the upper leftmost end point to locate all
existing valleys, if any. While doing this, it always keeps
track of the deepest encountered valley. The depth of a val-
ley is calculated as the difference between the position of
two pixels; the top-most pixel at which the contour changes
directiondownwards and the bottom-mostpixel after which
the contour changes direction upwards. Of course, the two
highest points on the two sides of a valley are considered in
calculating its depth, as formalized below.
There are three important pixels related to each valley;
the toppixels onboth sides and the deeppixelat the bottom,
denoted as
coordinates
image as the origin, the horizontal axis is the x-axis, and the
veritcal axis is the y-axis. Then, the depth
calculatedas follows:
is a function that returns the minimum of its arguments.
It is importantto have
turns the absolute value and
with
This requirement has been imposed in order to eliminate
considering valleys that exist due to some noisy strokes.
Also, this requirement eliminates valleys that may be de-
tected within a single digit. For instance, the handwriting
of any of the two digits 5 and 2 may lead to a valley from
the right-hand side and left-hand side, respectively. Acti-
vating the above rule successfully eliminated such valleys
from consideration and hence improved the whole separa-
tion process.
Thecenterofavalleyis calculatedas theaveragedistance
between each correspondingopposite points
the two sides of a valley, from point
?, ? and
?, respectively. Each point has two
? and
?, with the bottom left endpoint of the
? of a valley is
?
?
?????
?
??
?
?
?
?
?, where
???
?????
?
?
?
?
?
?
?, where
??? re-
? is a threshold chosen as
??
?,
? denoting the average height of the thinned image.
? and
? along
? up to point
?
?
???????
?. Formally,
?
??????
?
???? ?????
?
?
??
?
?
?
?
??, and
The closest feature point
valley is the point with
if more than one feature point qualify according to this rule,
thentheonewith
didate cut-point to be brought up in the negotiation process.
The selected candidate cut-point is marked by replacing its
?
??????
?
?
?
?
?
?
? to the center of the deepest
?????????
?
?
?
??????
??. However,
?????????
?
?
?
??????
?? isselectedascan-
feature point marker (“2” or “3”) by “y”, and its degree of
condifence is calculated as follows:
? ??? ??????????
?
?????????
?
??
??????
??,
where
depending on the involved agent.
? is either the depthof a valley orthe height ofa hill,
Fig. 1.An example image for which the top-valley agent alone failed
(a) original scanned image (b) after thinning (c) with feature-points
(d) after initial segmentation (e) final parts after restoration
The success of the top-valley based agent depends on the
degree of confidence in the reported candidate cut-point. A
low degree of confidence does not lead to successful seg-
mentation of connected digits. As the degree of confidence
increases, the segmentation of connected digits becomes
more successful. Experimentalresults showed that segmen-
tation based on low degree of confidence leads to distorted
two parts, whichcannotbe restoredintorecognizabledigits.
One such example is shown in Figure 1. This example il-
lustrates the importance of the negotiation process between
multiple agents in resolving the conflict.
2.2. Bottom-Hill Based Agent
In this section, we present the bottom-hill based agent.
The idea is simple. The original image is inverted (up-side-
down) and the same methodology of the top-valley agent
is applied to locate candidate cut-point.
the same problems encountered with the top-valley based
agent can be faced with the bottom-hill based agent. The
center
age is reflected into the normal (uninverted) image as the
center
done by applying the following two equations:
Hence, almost
??????
? of the deepest valley in the inverted im-
??????
? of the highest bottom-hill. It is formally
?
??????
?
?
?
??????
The selected candidate cut-point is marked by replacing
its feature point marker (“2” or “3”) by “x”, and its degree
of condifence is determined.
?, and
?
??????
?
?
?
?
?
??????
?
Fig. 2.An example image for which the bottom-hill agent alone failed
(a) original scanned image (b) after thinning (c) with feature-points
(d) after initial segmentation (e) final parts after restoration
As it is the case with top-valley based agent, the success
ofthebottom-hillbasedagentdependsonthedegreeofcon-
fidence in the reported candidate cut-point. The example
Proceedings of the Seventh International Conference on Document Analysis and Recognition (ICDAR 2003)
0-7695-1960-1/03 $17.00 © 2003 IEEE

Page 3

shown in Figure 2 once again illustrates the importance of
thenegotiationprocessbetweenmultipleagentsinresolving
the conflict. Finally, it is worth mentioningthat the imagein
Figure 2 has been successfully segmented by the top-valley
based agent, and the image shown in Figure1 has been suc-
cessfullyseparatedbythebottom-hillbasedagent,asshown
in Figure 3 and Figure 4, respectively.
Fig. 3. The image shown in Figure 1 successfully separated with negotia-
tion (a)original scanned image (b)after thinning (c)with feature-points
(d) after initial segmentation (e) final recognizable digits
Fig. 4. The image shown in Figure 2 successfully separated with negotia-
tion (a)original scanned image (b)after thinning (c)with feature-points
(d) after initial segmentation (e) final recognizable digits
3. Negotiation and Conflict Resolution
After each agent derives its input requiredfor the negoti-
ate process, namely the two candidate cut points marked as
“x” and “y” on the thinned connected image, the two agents
cooperate together to resolve the conflict, if any. Although,
every agent concentrates on its reported cut-point with the
highestconfidence,eachagentkeepstrackoftheconfidence
of each other candidate cut-pointsorted in descendingorder
by confidence. The latter candidate cut-points are utilized
only in case the other agent does not agree on the cut-point
with the highest confidence. In this case, a cut-point is se-
lected as actual cut point only if the sum of its confidence
values reported by both agents is maximum.
Everyagent starts by declaringits detected candidatecut-
point with the highest confidence. There are three possible
cases to consider. The two agents may report: 1) the same
candidate cut-point, 2) two candidate cut-points with the
same
didate cut-points with different
coordinate. The last case is the worst where negotiationand
conflict resolution is employed. The first two cases will in-
crease the degree of confidence in the candidate cut-point
and the image is separated vertically along the reported
coordinate. However, here the degree of confidence of each
agent in its proposed candidate cut-points is important.
As the degree of confidence increases, the agent gets
higher priority during the negotiation process. If the depth
? coordinateand different
? coordinate,or 3) two can-
? coordinate and different
?
?
of the reported valley or the height of the reported hill is
zero then the degree of confidence also reaches zero giving
little chance to the corresponding agent during the negotia-
tion process.
Remains are touching cases that cause the conflict. The
two agents negotiate to resolve this conflict based on the
calculated degrees of confidencein each of the two reported
candidate cut-points.
1. Let
2. If
the actual cut point is reported as having
??
.
3. ElseIf
the actual cut point is reported as having
?
?and
?
?be the two reported candidate cut-points.
????????????
?
??
????????????
?
? then
?
??????
?
?
?
?
?
?
?
???? ????????
?
?
?? ??? ????????
?
? then
?
??????
?
?
?
?
?
?
??
? ??? ????????
?
?
? ???????????
?
?
?
?.
4. ElseIf
the actual cut point is reported as having
? ???????????
?
?
?? ???????????
?
? then
?
??????
?
?
?
?
?
?
??
? ??? ????????
?
?
? ???????????
?
?
?
?.
Notice the
confidence related fraction is either added or subtracted. It
is added in case the other candidate cut-point (?
lies on the right-hand-side of the candidate cut-point (?
? sign in
? and
? above, where the degree of
? or
?
?)
?or
?
equation; it is subtracted otherwise.
?) of the
? coordinate used in the right-hand-side of the
Fig. 5.Some example original scanned touching pairs and their corre-
sponding separated digits
The negotiation and conflict resolution process has
been successfully applied to different images with single-
touching, multiple-touching, long-touching segment and
overlap. For instance, the image shown in Figure 3 has the
degree of confidence in cut-point
valley based agent too high compared to the degree of con-
fidence in cut-point
agent. Therefore,the actual cut-point is selected close to
from the side of
hand, the image shown in Figure 4 has the degree of con-
fidence in cut-point
?
? reported by the top-
?
?reported by the bottom-hill based
?
?
?
?as shown in Figure 3-d. On the other
?
?too high compared to the degree of
Proceedings of the Seventh International Conference on Document Analysis and Recognition (ICDAR 2003)
0-7695-1960-1/03 $17.00 © 2003 IEEE

Page 4

confidence in cut-point
is selected close to
ure 4-d. Some other successfully segmented and restored
illustrating examples are shown in Figure 5.
?
?. Therefore, the actual cut-point
?
?from the side of
?
?as shown in Fig-
4. Restoration
Segmenting the image into two parts by a vertical line
through the already selected actual cut-point will lead to
several situations to consider: 1) A single connected part
on each side. 2) A single connected part on one side and
a single connected part with some small isolated strokes on
the other side. 3) Two or more isolated parts on one side
and a single connected part on the other side. Each of these
three cases is handled in a different way by applying a dif-
ferent heuristic. Case 1: Algorithm 4.1 is applied to move
stroke(s) from each part to the other. Moved stroke(s) help
in restoringthe two digits. Case 2: The small strokes will be
moved to the other side, as they complete the other part of
the image. Then, Algorithm 4.1 is applied. Case 3: Slices
of the connected part are moved to the other side until some
featurepoint(s)areencounteredonthe connectedside orthe
other side becomes a single connected piece. Then, Algo-
rithm 4.1 is applied to investigate the possibility to restore
each part of a segmented image into a recognizable digit by
moving necessary strokes between the two parts. However,
Algorithm 4.1 utilizes what we call artificial end-points as
described next in Definition 4.1.
Definition 4.1(Artificial end-points)
An end-point
one of the two parts of a segmented image,
part, and
“1” or “2”.
? is said to be artificial if and only if:
? lies in
? faces the other
? is not marked as “3”, i.e.,
? is marked as either
Algorithm 4.1 (Restoration)
Input: The two parts obtained after initial segmentation.
Output: The two parts (hopefully digits) after restoration.
Steps:
Consider each of the two parts after initial segmentation;
Locate artificial end-points in each part;
If there exist a single artificial end-point
? in any of the two parts then
? There is a corresponding artificial end-point
other part such that
?
?in the
?????
?
?
?
?
?
?
?
?;
? Locate stroke
tered feature point
? connecting
? to the first encoun-
? along the contour of
?’s part;
? Locate stroke
feature point
If
?
?connecting
?
?to the first encountered
?
?along the contour of
?
?’s part;
? is an end feature point (marked “3”) then
? Move
? to
?
?’s part;
ElseIf
?
?is an end feature point then
? Move
?
?to
?’s part;
ElseIf (?????????
????? ????
?
?) and
(????????
?
?
????????? ?) then
? Move
?
?to
?’s part;
ElseIf (?????????
? ???????? ?) then
? Move
? to
?
?’s part;
EndIf
Else
? Consider artificial end-points in each part in top-down order;
For each two consecutive artificial end-points
in the same part do
? and
?
? Fill the gap between
corresponding stroke from the other part;
If (there are, not necessary in the same part, two
or more feature points
? and
? by moving a
? and
? between
? and
?) and (neither
? nor
? is marked “3”) then
? The stroke that lies between
must be duplicated in both parts;
? and
? (inclusive)
EndIf
EndFor
EndIf
EndAlgorithm.
Fig. 6.An example image with long touching part (a) original scanned
image (b) after thinning (c) with feature-points (d) after initial segmen-
tation (e) final digits after restoration
The process described above is illustrated in Figure 6,
where Algorithm 4.1 successfully moved stroke(s) right
and/or left between the two parts. As a result, two recog-
nizable digits were reported.
4.1. Noise Elimination
Noise may be present between connected digits like
bridging stroke(s). After successfully separating touching-
digit pairs into two parts, our approach successfully elimi-
nates extra strokes by applying some rules. In other words,
there are cases where there are some extra strokes located
on the rightmost side of the left part or on the leftmost side
of the right part. Such strokes are located by apply the fol-
lowing two steps.
1. Project each part of the image on the horizontal x-axis
to get a histogram showing the number of 1’s (dark pixels)
along each column of the thinned image.
2. Eliminate any stroke, located on the rightmost side of
the left part or on the leftmost side of the right part, with
small number of 1’s per column compared to the maximum
number of 1’s in a single column along the whole con-
sidered part, and it is the only stroke across the height of
its part.
Applying this to the connected image shown in Figure 7-
a leads to the elimination of the extra stroke and hence
the corresponding image shown in Figure 7-b is obtained.
These types of extra strokes are not consideredin [2], which
Proceedings of the Seventh International Conference on Document Analysis and Recognition (ICDAR 2003)
0-7695-1960-1/03 $17.00 © 2003 IEEE

Page 5

Fig. 7. An example image (a) with noise (b) after eliminating the noise
considered only special extra strokes and mainly those in-
volving digits “0”, “8”, “6” and “9”. Our approach is more
general to cover many other cases.
5. Experimental Results
Experimentswere conductedusing our own 4095 images
written by 150 different persons. The images are all hand-
written numeral strings with possible touching of differ-
ent categories, including3293 images with single-touching,
411 images with multiple-touching, 235 images with long
touching and 156 images with overlapping. All our images
were scanned on a 600 dpi scanner, and stored as Bitmap
files.
We classified the outcome of the testing process into two
categories, correctly segmented images and erroneous im-
ages. This classification leads to two rates, namely correct
rateanderrorrate. Outof
were correctly segmented and
a result, the achieved correct rate is 97.8%.
????imagesusedinthetest,
????
?? failures were reported. As
Fig. 8. Examples of successfully segmented touching pairs
Fig. 9.An example of erroneously separated touching pair (a) original
scanned pair (b) after thinning (c) with feature-points (d) after initial
segmentation (e) final parts
Similar to our approach, a lot of work has been done
on the recognition of touching in handwritten numerals,
e.g. [4], [5], [7], [8], [10]. However, these techniques are
unstable specially when applied to numerals with multiple-
touching, unclear cut-point and single long touching part.
Further, some of the approaches described in the literature,
e.g., [3] differentiate only between correct rate and error
rate. Other approaches, e.g., [2], [4], [6] consider two rate:
rejection and acceptance, with the correct rate derived from
the acceptance rate. The best rate reported from the latter
approaches is 96%, with 7.8% reject rate [2].
Compared to the other approachesdescribed in the litera-
ture, our approach successfully segmented difficult images.
For instance, the works described in [4] and [6] fail when
segmentingimages that containa longtouchingpart like the
image shown in Figure 6. Further, we also successfully seg-
mented images that have been reported as erroneous or re-
jected images in the best reportedapproachdescribedin [2].
Some example images are shown in Figure 8. However,
our approachfailed for some images with overlap,when the
overlap results in a hole as illustrated in Figure 9.
6. Conclusions and Future Work
In this paper, we elaborated on the importance of utiliz-
ing multi-agents in recognizingtouching in handwritten nu-
meral strings. We mainly introduced two agents that we
have developed to work on the thinned image. Every agent
looks at the image from a different perspective and detects
a candidate cut-point, which may lead to correct segmenta-
tion based on a computed degree of confidence. In general,
the derived two candidate cut-points are not the same. The
difference requires the two agents to negotiate and resolve
the conflict and decide on the actual cut-point. The experi-
ments conducted were successful and encouraging. We re-
alized that each of the two agents was not able to achieve
on its own the same recognition rate achieved after the co-
operation and negotiation strategy is applied. Finally, our
approach successfully removes extra strokes, if any.
REFERENCES
[1]Bul O., Polat F. and Alhajj R., “Multi-Agent Reinforcement Learning
Using Function Approximation,” IEEE-SMC, Vol.30, No.4, pp.485-
497, 2000.
Chen Y.K. and Wang J.F., “Segmentation of Single- or Multiple
Touching Handwritten Numeral String Using Background and Fore-
ground Analysis,” IEEE-PAMI,, Vol.22, No.11, pp.1304-1317, 2000.
Cheriet M., Huang Y.S. and Seun C.Y., “Background Region-Based
Algorithm for the Segmentation of Connected Digits,” Proc. ofICPR,
pp.619-622, 1992.
Chi Z., Suters M. and Yan H., “Separation of Single- and Dou-
ble Touching Handwritten Numeral Strings,” Optical Eng., Vol.34,
pp.159-165, 1995.
Lee E. and Choi H.I., “Recognition of Touching Character Using Par-
tial Prototypes,” Proc. of ICCPOL, Mar. 1999.
Lu Z., Chi Z. and Shi P., “A Background Thinning-Based Ap-
proach for Separating and Recognizing Connected Handwritten Digit
Strings,” Pattern Recognition, Vol.32, No.6, pp.921-933, 1999.
Shridhar M. and Badreldin A., “Recognition of Isolated and Sim-
ply Connected Handwritten Numerals,” Pattern Recognition, Vol.19,
pp.1-12, 1986.
Suters M. and Yan H., “Connected handwritten digit separation using
external boundary curvature,” Journal of Electronic Imaging,, Vol.3,
No.3, pp.251-256, 1994.
Yu D.G. and Yan H., “Separation of Single-Touching Handwritten
Numeral Strings Based on Structural Features,” Pattern Recognition,
Vol.31, pp.1835-1847, 1998.
[10] Wang X., Govindaraju V. and Srihari S.N., “Holistic Recognition of
Handwritten Character Pairs,” Pattern Recognition, Vol.33, pp.1967-
1973, 2000.
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
Proceedings of the Seventh International Conference on Document Analysis and Recognition (ICDAR 2003)
0-7695-1960-1/03 $17.00 © 2003 IEEE