A GraphSpectral Approach to Correspondence Matching
ABSTRACT Not Available

Article: Partial shape recognition by submatrix matching for partial matching guided image labeling
[Show abstract] [Hide abstract]
ABSTRACT: We propose a new partial shape recognition algorithm by submatrix matching using a proximitybased shape representation. Given one or more example object templates and a number of candidate object regions in an image, points with local maximum curvature along contours of each are chosen as feature points to compute distance matrices for each candidate object region and example template(s). A submatrix matching algorithm is then proposed to determine correspondences for evaluation of partial similarity between an example template and a candidate object region. The method is translation, rotation, scale and reflection invariant. Applications of the proposed partial matching technique include recognition of partially occluded objects in images as well as significant acceleration of recognition/matching of full (nonoccluded) objects for object based image labeling by learning from examples. The speed up in the latter application comes from the fact that we can now search only those combinations of regions in the neighborhood of potential partial matches as soon as they are identified, as opposed to all combinations of regions as was done in our prior work [Xu et al., Object formation and retrieval using a learningbased hierarchical contentdescription, Proceedings of the ICIP, Kobe, Japan 1999]. Experimental results are provided to demonstrate both applications.Pattern Recognition 10/2005; 38(10):15601573. · 2.58 Impact Factor
Page 1
A Graphspectral Approach to Correspondence Matching
Antonio RoblesKelly
?
Edwin. R. Hancock
Department of Computer Science, University of York,York YO1 5DD, UK.
?arobkell,erh
? @minster.cs.york.ac.uk
Abstract
This paper describes a spectral method for graph
matching. We adopt a graphical models viewpoint in which
the graph adjacency matrix is taken to represent the transi
tion probability matrix of a Markov chain. The nodeorder
ofthesteady staterandom walkassociated withthis Markov
chain is determined by the coefficent order of the lead
ing eigenvector of the adjacency matrix. We match nodes
in different graphs by aligning their sequence order in the
steadystatewalk. Themethodproceedsfrom thenodeswith
the largest leading eigenvector coefficient. We develop a
brushfiresearchmethod to assign correspondencesbetween
nodes using the rankorder of the eigenvector coefficients
in firstorder neighbourhoods of the graphs. We demon
strate the utility of the new graphmatchingmethod on both
synthetic and real graphs.
1Introduction
Spectral graph theory is a field of mathematics that aims
to characterise the global structural properties of graphs
using the eigenvalues and eigenvectors of the adjacency
matrix [3]. In the computer vision literature there have
been a number of attempts to use graph spectral proper
ties for matching and recognition. For instance Umeyama
has an eigendecomposition method that matches graphs
of the same size [11]. Both Scott and LonguetHiggins
[9] and Shapiro and Brady [10] have developed methods
for pointcorrespondence which use the eigenvectors of the
point proximity matrix. Horaud and Sossa[6] have adopted
a purely structural approach to the recognition of line
drawings which is based on the inmanental polynomials for
the Laplacian matrix of the lineconnectivity graph. Shoko
ufandeh, Dickinson and Siddiqi [1] have shown how graphs
can be encoded using local topological spectra for shape
recognition from large databases.
Although formally elegant, the main limitation of these
graphspectral methods is their inability to cope with graphs
of different sizes. This means that they can not be used
when significant levels of structural corruption are present.
In this paper our aim if to use results from spectral graph
?Supported by CONACYT, under grant No. 146475/151752.
theory concerning random walks on graphs [8, 4, 3]. to
develop a robust method for graphmatching. We match
graphsbyaligningthesteadystaterandomwalksassociated
with their adjacency matrices. The alignment is performed
using a brushfire search procedure which uses the rank or
der of the coefficients of the leading eigenvector. This pro
cedure has features in common with the method of Shapiro
and Brady which finds correspondences between weighted
graphs so as to minimise the Euclidean distance between
the leading eigenvector [10]. This method is notoriously
susceptible to differences in the sizes of the graphs being
matched. Our method, on the other hand, searches for cor
respondences by combining the rankorder of the nodes and
edge consistency constraints. As we shall demonstrate, this
gives us improved robustness to size difference.
2Random Walks on Graphs
The relationship between the leading eigenvector of the
adjacencymatrix and thesteady state randomwalk has been
exploited in a number areas including routeing theory and
information retrieval.
In a weighted graph
and edgeset
diagonal elements of the transition probability matrix
the weights associated with the edges. In this paper, we
exploit a graphspectral property of the transition matrix
to develop a graph matching method. This requires that we
have the eigenvalues and eigenvectors of the matrix
hand. To find the eigenvectors of the transition probability
matrix,
The unit eigenvector
found by solving the system of linear equations
????????????????? with indexset
?
??????????????????? ???!?"?$#%?'&(???)?+*?,?. , the off
?
are
?
?
to
? , we first solvepolynomial equation
??0/21435?6?87 .
94: associated with the eigenvalue
14: is
; ??95:<?
where
1
:
9
: and satisfies the condition
Consider a random walk on the graph
commences at the node
of edgeconnected nodes
95=
:
9>?@? .
? . The walk
?BA and proceeds via the sequence
?????BAC????DE?!?GF6?H;I;J;J
?J?
then the probability of visiting the nodes in the sequence
above is
can be represented using the transition probability matrix
whose element with row
ther, let
:
?!?
:?K
AH?L#>? . Suppose that the transition probability asso
ciated with the move between the nodes
If the random walk can be represented by a Markov chain,
??M and
?GN
is
?OM?PN .
?RQS?T?U?J?
A
?.VXW YZW
M?[\A
?^]`_ba"c
P
]`_ . This Markov chain
?d and column
e
is
?OM?PN . Fur
fhg?? ?i? be the probability of visiting the node in
10514651/02 $17.00 (c) 2002 IEEE
Page 2
dexed
? after tsteps of the random walk and let
be the vector of probabilities. After
time steps
and
then
fhg8?
??fhgj?i?C?j??fhgj??kE?l?H;I;J;b?i=m
f
g
??? ?n=o?
g
fhp . If
14: are the eigenvalues of
?
95: are the corresponding eigenvectors of unit length,
?@?
WYZW
q
:
[\A
1
:
9
:
9
=
:
As a result, after
probability matrix
m applications of the Markov transition
?
g
?
WYZW
q
:
[\A
1
g
:
95:?9
=
:
If the row and columns of the matrix
?
sum to unity, then
1
A
?r? . Furthermore, from spectral graph theory [3] pro
Markov chain approaches it steady state, i.e.
all but the first term in the above series become negligible.
Hence,
ing eigenvector of the transition probability matrix is the
steadystateoftheMarkovchain. Foramorecomplete proof
of this result see the book by Varga [12] or the review by
Lov´ asz [8]. As a result, if we visit the nodes of the graph in
the order defined by the magnitudes of the coefficients of
the leading eigenvector of the transition probability matrix,
then the path is the steady state Markov chain. In this paper
we aim to perform graph matching using brushfire search
along the path specified by the magnitude order of the com
ponents of the leading eigenvector.
To proceed, suppose that the leading eigenvector for
the datagraph adjacency matrix is denoted by
vided that the graph
smallest eigenvalue
?
is not a bipartite graph, then the
1
WY?W>s
/n? . As a result, when the
mutwv
, then
xIyJzUg {}X?
g
?~9AH95=
A . This establishes that the lead
9
?
??9
two graphs as “data” and “model” is a matter of convention.
Here we take the data graph tobe the graph which possesses
the largest leading eigenvalue, i.e.
Our aim is to use the sequence of nodes defined by
the rank order of the magnitudes of the components of
the leading eigenvector as a means of locating corre
spondences. The rank order of the nodes in the data
graph is given by the list of sorted nodeindices
?i?C?j??;J;I;J;I;I??9
???
???
= while that for the model graph is
denoted by
ated eigenvalues are
9
?
??9
?i?G?l?H;I;J;I;I;9
???
? ?)?)= . The associ
1
and
1
. The designation of the
15
s
15
.
?
?J?BAC????D6?!?GFE?H;I;J;I;J???
WY??W
? where
9
?J?BA??
s
9
?I??DC?
s
9
?J?GF??
s
;J;I;
s
9
?I?
WYGW
? . The subscript
of the nodeindex
?G8#
?
is hence the rankorder of the eigenvector component
9
rithm we will need to determine the rankorder of specified
nodes. Accordingly, we define the operator
which returns the rankorder
setusing the coefficients of the eigenvector
?I?G.? . The rankordered list of modelgraph nodes is
?????)AB?)?`D???FE?H;I;J;I;I?)?
WY
W
? where
9
? ?)Al?
s
9
? ??DG?
s
9
? ??FG?
s
;I;I;
s
9
???
WYG?W
? . To develop our matching algo
2?J?G??R??9
?
of the node index
?G in the
%?
9
.
3Correspondence Matching
The idea underpinning our graphmatching algorithm is
to use the rank order provided by the components of the
leading eigenvector to locate correspondence matches. We
pose this as a brushfire search which is driven from the rank
order of the nodes in the datagraph. In a nutshell, the idea
is totraversetherankorderedlistof datagraphnodes, com
mencing with the node of largest coefficient and terminat
ing with the node of smallest coefficient.
We commence by placing the first ranked node from the
datagraph in correspondence with the first ranked node
of the model graph, i.e.
our brushfire search, by considering the firstneighbours
of the datagraph node
may be assigned to these nodes and which satisfy the edge
connectivity constraints provided by the model graph are
the firstneighbours of the modelgraph node
data graph, the set of firstneighbour nodes is
R?J?BAl????)A . We proceed with
?
A . The candidate matches which
?
A . For the
]ic
?
????J?J?6?!?BAl?u#0?
ficients of the leading eigenvectors of the associated adja
cency matrix. We place nodes of the same rank order in
correspondence with oneanother. The assignment rule is
 and the set of candidate correspondences
from the modelgraph is
We rank the nodes in the two sets according to the coef
]ic`
??G?j?J??????R?J?BAl??#,?
 .
?U#>
]ic
R?I???Z??O2?I?E??
])c
??9
?Z?2?????
:c
??9
?`
(1)
We propagate this procedure by visiting each node in
the datagraph in the order specified by the rankedlist
This is an iterative process which spreads like a brushfire
from the seed node
ranked node, i.e.
of the nodes visited and the correspondences assigned we
maintain two lists. The first of these is the set of data
graph nodes
to be assigned. The second is the list of available model
graph nodes
spondence with the datagraph nodes. The algorithm pro
ceeds as follows. First, we find the set of firstneighbours
of the datagraph node
dences. This set is given by
are following a chain of edgeconnected nodes, the data
graph node
dence match since it is one of the firstneighbours of the
node
the algorithm. We would like to preserve edgeconnectivity
constraints while assigning correspondences to the unvis
ited first neighbours of
nodes in the modelgraph which are connected to the as
signed correspondence of the node
]`¤
.
?BA . Suppose that we have reached the
g ¡
?G , in the datagraph. To keep track
¢
?J?
? to which correspondences have yet
¢
???
? which have yet to be placed in corre
?G which remain without correspon
£
]`¤
?¥¢
]?¤§¦
]?¤ . Since we
?G will already have been assigned a correspon
??
K
A which was visited in the previous iteration of
?
.Hence, we find the set of
?
. This set is given by
E
?????l???!R?J?
?j??i?>#¨?
 . The set of nodes which
preserve the edge consistency constraints provided by the
modelgraph and which are available for assignment to the
10514651/02 $17.00 (c) 2002 IEEE
Page 3
nodes of the datagraph is
We assign correspondences from the set of modelgraph
nodes
sis of the rankorder of the coefficients of the leading eigen
vector of the adjacency matrix. However, the two sets may
be of different cardinality. If the set
than the set
eigenvector coefficients may be discarded. If, on the other
hand, the set
than the set
assigning correspondences. To do this we introduce null or
dummy correspondences. We therefore pad the set
with
The ranks of these nodes are
£
E
]?¤E
?¢
?J?G.?
¦
]?¤B.
£
]
¤
to the set of dategraph nodes
£
]
¤ on the ba
£
]
¤ is of smaller size
£
E
]
¤
, then the nodes with low rank leading
£
]
¤
is null (i.e. empty) or of smaller size
£
]`¤ , then we must find an alternative way of
£
]?¤
?
]`¤
?©?£
]?¤
?E/ª?£
E
]?¤E
? dummy nodes denoted by
« .
?£
E
]?¤6
?l¬? ,
?£
E
]?¤E
?l¬®k ,.....,
?£
E
]?¤E
??¬~?
]?¤ . The resulting set of padded modelgraph
nodes is
¯
£
E
]?¤
?~£
E
]`¤°
«
W±
²
¤
W
K
W±
³´b²
¤Eµ
W
(2)
The correspondences for the nodes belonging to the set
are assigned as follows
£
]?¤
?2#£
date the two lists of nodes available for correspondence.
The updated set of datagraph nodes which have yet to
be assigned correspondences is
while the set of nodes available for assignment is
]?¤h¶
R?J?"?Z?8?\2?J?6??£
]`¤
??9
?Z?2?????
¯
£
E
]
¤
??9
??·
(3)
Once the correspondences have been assigned, then we up
¢
?J?G¸\A??®?w¢
?J?G.?h/
£
]`¤
¢
?J?G¸\A??Z?¢
Experiments
?I?G?R/?£
]`¤ .
This process is repeated until all of the nodes in the data
graph have been assigned correspondences, i.e.
¢
?J?G.?u?
¹.
4
We have conducted some experiments with the CMU
house sequence. This sequence consists of a series of im
ages of a model house which have been captured from dif
ferent viewpoints. To construct graphs for the purposes of
matching, we have first extracted corners from the images
using the corner detector of Luo, Cross and Hancock [2].
The graphs used in our experiments are the Delaunay tri
angulations of these points. The Delaunay triangulations
of the example images are shown in Figure 1a. We have
matched pairs of graphs representing increasingly different
views of the model house. To do this, we have matched
the first image in the sequence, with each of the subsequent
images. In Figure 1 b, c and d we show the sequence of
correspondence matches. In each case the lefthand graph
contains 34 nodes, while the righthand graphs contain 30,
32 and 34 nodes. From the Delaunay graphs it is clear that
therearesignificant structuraldifferencesinthegraphs. The
(a)
(b)
(c)
(d)
Figure 1. Delaunay triangulations and sequence of corre
spondences
numbers of correctly matched nodes in the sequence are re
spectively 24, 22 and 21 nodes. By comparison, the more
complicatediterativeEMalgorithm of LuoandHancock [7]
gives 29, 23 and 11 correct correspondences. As the differ
ence in viewing direction increases, the fraction of correct
correspondences decreases from 80% for the closest pair of
images to 60% for the most distant pair of images.
We have conducted some comparison with a number of
10514651/02 $17.00 (c) 2002 IEEE
Page 4
alternative algorithms. The first of these share with our
method the feature of using matrix factorisation to locate
correspondences and have been reported by Umeyama [11]
and Shapiro and Brady [10]. Since these two algorithms
can not operate with graphs of different size, we have taken
pairs of graphs with identical numbers of nodes from the
CMU sequence; these are the second and fourth images
which both contain 32 nodes. Here the Umeyama method
and the Shapiro and Brady method both give 6 correct cor
respondences, while both the Luo and Hancock [7] method
and our own give 22 correct correspondences.
Finally, we have conducted some experiments with syn
thetic data to measure the sensitivity of our matching
method to structural differences in the graphs and to pro
vide comparison with alternatives. Here we have gener
ated random pointsets and have constructed their Delau
nay graphs. We have simulated the effects of structural er
rors by randomly deleting nodes and retriangulating the re
maining pointset. In Figure 2 we show the fraction of cor
rect correspondences as a function of the fraction of nodes.
We plot two performance curves for our new method. The
curve labeled ”Markov worst case” shows the result if any
of the nodes are deleted from the graph. The curve labeled
”Markov best case” shows the result obtained if the seed
node is protected. This latter result is much better than the
former. In the case of random deletion, the fraction of cor
rectmatches falls to80% when thefraction of deleted nodes
is 10%. By contrast, the Shapiro and Brady method gives
on a few percentage of correct correspondences at this level
of corruption. Also shown on the plot are the performance
curves for the Wilson and Hancock [13] and Gold and Ran
garajan [5] methods. In the case of random node deletion,
our method gives performance that is better than the Gold
and Rangarajanmethod but worse than the Wilsonand Han
cock method. If the seed node is protected, then the method
is comparable with the Wilson and Hancock method. The
main conclusion to be drawn from this study is that our
method may form the basis of a robust algorithm if a more
sophisticated search procedure is used.
0
10
20
30
40
50
60
70
80
90
100
0.10.2 0.3 0.40.50.6 0.7
FRACTION OF CORRECT CORRESPONDANCES VS NUMBER OF NODES DELETED
MARKOV WORST CASE
MARKOV BEST CASE
QUADRATIC
DISCRETE RELAXATION
NONQUADRATIC ASSIGNMENT
Figure 2. Sensitivity study results.
5Conclusions
In this paper we have described a spectral method for
correspondence matching. The method makes use of a
brushfire search procedure to find correspondences using
the rankorder of the coefficientsof the leading eigenvector
of the adjacency matrix. The search procedure commences
from the node of largest coefficient and proceeds via first
order neighbourhoods to assign correspondences on the ba
sis of local rank order. The method proves to be robust to
structural differences in the graphs being matching and out
performs the method of Shapiro and Brady.
References
[1] K. Siddiqi A. Shokoufandeh, S. J. Dickinson and S. W.
Zucker. Indexing using a spectral encoding of topological
structure. In Proceedings of the Computer Vision and Pat
tern Recognition, 1998.
[2] Luo Bin and E. R. Hancock. Procrustes alignment with the
em algorithm.In 8th International Conference on Com
puter Analysis of Images and Image Patterns, pages 623–
631, 1999.
[3] Fan R. K. Chung. Spectral Graph Theory. American Math
ematical Society, 1997.
[4] M. Doob D. Cvetkovi´ c and H. Sachs.
Graphs:Theory and Application. Academic Press, 1980.
Spectra of
[5] S. Gold and A. Rangarajan. A graduated assignment algo
rithm for graph matching. IEEE TPAMI, 18(4), 1996.
[6] R. Horaud and H. Sossa. Polyhedral object recognition by
indexing. Pattern Recognition, 1995.
[7] Bin Luo and E. R. Hancock. Structural graph matching us
ing the EM algorithm and singular value decomposition. To
appear in IEEE Trans. on Pattern Analysis and Machine In
telligence, 2001.
[8] L. Lov´ asz. Random walks on graphs: a survey. Bolyai Soci
ety Mathematical Studies, 2(2):1–46, 1993.
[9] G. Scott and H. LonguetHiggins. An algorithm for associat
ing the features of two images. In Proceedings of the Royal
Society of London, number 244 in B, 1991.
[10] L. S. Shapiro and J. M. Brady. A modal approach to feature
based correspondence. In British Machine Vision Confer
ence, 1991.
[11] S. Umeyama. An eigen decomposition approach to weighted
graph matching problems. PAMI, 10(5):695–703, September
1988.
[12] R. S. Varga. Matrix Iterative Analysis. Springer, second
edition, 2000.
[13] R.C. Wilson, A.N. Evans, and E.R. Hancock. Relational
matching by discrete relaxation. IVC, 13(5):411–421, June
1995.
10514651/02 $17.00 (c) 2002 IEEE