LDC: enabling search by partial distance in a hyper-dimensional space

Abstract

Recent advances in research fields like multimedia and bioinformatics have brought about a new generation of hyper-dimensional databases which can contain hundreds or even thousands of dimensions. Such hyper-dimensional databases pose significant problems to existing high-dimensional indexing techniques which have been developed for indexing databases with (commonly) less than a hundred dimensions. To support efficient querying and retrieval on hyper-dimensional databases, we propose a methodology called local digital coding (LDC) which can support k-nearest neighbors (KNN) queries on hyper-dimensional databases and yet co-exist with ubiquitous indices, such as B+-trees. LDC extracts a simple bitmap representation called digital code(DC) for each point in the database. Pruning during KNN search is performed by dynamically selecting only a subset of the bits from the DC based on which subsequent comparisons are performed. In doing so, expensive operations involved in computing L-norm distance functions between hyper-dimensional data can be avoided. Extensive experiments are conducted to show that our methodology offers significant performance advantages over other existing indexing methods on both real life and synthetic hyper-dimensional datasets.

Full text

Page 1

LDC: Enabling Search By Partial Distance In A Hyper-Dimensional
Space
Nick Koudas1
Beng Chin Ooi2
Heng Tao Shen2
Anthony K. H. Tung2
1AT&T Labs Research
Shannon Laboratory Florham Park, USA
2Department of Computer Science
National University of Singapore, Singapore
Abstract
Recent advances in research fields like multimedia
and bioinformatics have brought about a new genera-
tion of hyper-dimensional databases which can contain
hundreds or even thousands of dimensions. Such hyper-
dimensionaldatabasesposesignificantproblemstoexisting
high-dimensional indexing techniques which have been de-
veloped for indexing databases with (commonly) less
than a hundred dimensions. To support efficient query-
ing and retrieval on hyper-dimensional databases, we pro-
pose a methodology called Local Digital Coding (LDC)
which can support k-nearest neighbors (KNN) queries on
hyper-dimensional databases and yet co-exist with ubiq-
uitous indices, such as B+-trees. LDC extracts a sim-
ple bitmap representation called Digital Code(DC) for each
point in the database. Pruning during KNN search is per-
formed by dynamically selecting only a subset of the
bits from the DC based on which subsequent compar-
isons are performed. In doing so, expensive operations in-
volved in computing L-norm distance functions between
hyper-dimensional data can be avoided. Extensive ex-
periments are conducted to show that our methodology
offers significant performance advantages over other ex-
isting indexing methods on both real life and synthetic
hyper-dimensional datasets.
1. Introduction
Hyper-dimensional1databases are databases which
contain hundreds or even thousands of dimensions. Re-
cent advances in several research fields including mul-
timedia, bioinformatics, data mining on audio, images
and text, as well as networking, have resulted in such
databases which pose significant challenges to existing
high-dimensional indexing techniques, that are usually
capable of handling databases (commonly) up to tens of
dimensions.
The problem of indexing and searching in a hyper- di-
mensional database is a challenging one, due to three
main reasons:
•First, according to several studies (e.g., [3]), the ex-
pected minimal distance between any two points in
a hyper-dimensional space is very large (becoming
1The term hyper-dimensional is used to differentiate the problem
we are addressing from the present norm of 30- to 50- (high) di-
mensional space
larger with increasing dimensionality) while the dif-
ference between the minimal and maximal distance
to a point is expected to be small (becoming smaller
with increasing dimensionality). These two charac-
teristics of a hyper-dimensional space mean that the
search radius for a k-nearest neighbor query is ex-
pected to be large. This in turn results in a large
number of ‘false positives’ since most points are ex-
pected to have almost equal distance to the query
point. This phenomenon leads to significant deterio-
ration of the query performance in most existing in-
dexing methods.
•Second, due to the extremely high dimensional-
ity, the fanout for most indexes built on a hyper-
dimensional space is typically very small, resulting
in an increase in the height of the indexes (e.g., in
a 200 dimensional space, we can’t expect more that
ten entries in an 8K page if 4 bytes are needed for
each dimension).
•Finally, the computation of the distance (e.g.,
Euclidean distance) between two points in a
hyper-dimensional space, becomes processor inten-
sive as the dimensionality increases. This implies
that the processor time is expected to become a sig-
nificant portion of the overall query response time
for a hyper-dimensional database. Proposed tech-
niques for optimizing the performance of most
indexing techniques do not take this into considera-
tion.
In this paper, we propose an effective methodology
called Local Digital Coding (LDC) for finding k-nearest
neighbors in a hyper-dimensional space. LDC is devel-
oped to address the problems mentioned above and pro-
vide a substantial reduction on both I/O and processor
time when searching on hyper-dimensional datasets con-
sisting of hundreds of dimensions. It is compatible with
ubiquitous indices, such as B+-trees and thus can be eas-
ily deployed.
Given a cluster of points in a high-dimensional data
space LDC transforms each point into a bitmap which we
refer to as the point’s Digital Code (DC). Each dimen-
sion of the point is represented by a single bit in its DC.
The DC of a point is generated by comparing the coordi-
nates of the point with the coordinates of the cluster cen-
ter the point belongs to. A bit is set to 1, if the value of
the dimension it corresponds to, is larger than the value
of the corresponding dimension of the cluster center, and
0 otherwise.

Page 2

Since there is a bit in the DC for each dimension, in-
dexing a D-dimensional space will result in DCs with
D bits. The data points in a cluster can thus be sep-
arated into 2Dpartitions with points in each partition
sharing the same DC. Based on LDC, we propose a novel
searching algorithm, called Searching on-the-fly by
PArtial-distance (SPA). Given the DCs of both the
query point and a partition, SPA dynamically selects a
subset from the DCs (say n bits) to perform matching.
A partition is pruned off if the number of matching bits
in the two DCs is less than m bits. The intuition behind
such an approach is that the points in the pruned parti-
tion are on different sides of some cutting planes with re-
spect to the query point and thus are too far away to be
in the answer set.
A summary of our contributions is as follows:
• We present LDC, an efficient methodology for index-
ing a hyper-dimensional space. LDC has the follow-
ing advantages over existing methods for handling
hyper-dimensional data:
–LDC is optimized to take both processor and
I/O time into account. This is important since
processor time for comparing vectors becomes
significant in a hyper-dimensional space.
–LDC’s method of spatial inference for pruning
the search space is based on cutting-planes in-
stead of direct distance computation. Because
of this, the proposed algorithm, SPA, that per-
forms KNN search, has significantly less ’false
positives’ compared to methods using direct
distance computation.
–LDC adopts a bit representation instead of
storing the actual dimensional values, thus, its
storage requirements are the smallest possible
(without considering the option of using com-
pression methods) for representing the location
of any D-dimensional partition. This effectively
reduces the number of I/Os that are required to
search the index.
We identify the parameters that affect the perfor-
mance of LDC and present the results of a detailed
analysis identifying how these parameters can be
tuned for certain objectives given a query point.
We present the results of a detailed experimental
evaluation of our methodology supporting our claims
that LDC is capable of handling hyper-dimensional
databases more efficiently, compared to other appli-
cable methods.
•
•
The rest of the paper is organized as follows. Section
2 reviews related work. LDC is presented in Section 3.
In Section 4, the novel SPA algorithm is introduced. An
extensive performance study on our method is presented
in Section 5. Finally, Section 6 concludes the paper.
2. Related Work
Indexing techniques have been the focus of exten-
sive research both in low as well as high-dimensional
databases [4]. With the demand for even higher-
dimensional databases, consisting of hundreds of or more
dimensions, earlier high-dimensional indexes face sig-
nificant challenges. Indexing techniques have been
designed typically for 30-50 dimensions, and fail to im-
prove the performance of sequential scan [11] due to
the known ‘dimensionality curse’. To tackle this phe-
nomenon recent proposals adopt one of the three
approaches: (1) Dimensionality reduction, (2) Data ap-
proximation, and (3) One dimensional Transforma-
tion.
Dimensionality reduction methods [9, 6] map the high
dimensional space into a low dimensional space which can
be indexed efficiently using existing multi-dimensional in-
dexing techniques. The main idea is to condense the orig-
inal space into a few dimensions along which the infor-
mation is maximized. Such methods report approximate
nearest neighbors however, since dimensionality reduc-
tion incurs information loss.
Representations of the original data points us-
ing smaller, approximate representations have also been
proposed, as a means of aiding high dimensional in-
dexing and searching. Such proposals include, the
VA-file [11], the IQ-tree [1] and the A-tree [10]. The
VA-file (Vector Approximation file) represents the orig-
inal data points by much smaller vectors. The main
drawback of the VA-file however, is that it defaults in as-
sessing the full distance between the approximate vec-
tors, which imposes a significant overhead, especially
if the underlying dimensionality is very large. More-
over, the VA-file does not adapt gracefully to highly
skewed data. The IQ-tree was proposed recently. It
maintains a flat directory which contains the mini-
mum bounding rectangles of the approximate data
representations.
One dimensional transformations provide another di-
rection for high-dimensional indexing. Such techniques
include the Pyramid technique[2] and iDistance[12].
They suffer however, from the fact that any meaning-
ful search operation involves assessing distances be-
tween the full high dimensional representation of the
data points; thus, pruning during search becomes prob-
lematic as the dimensionality increases. High dimen-
sional search techniques, assuming specific storage mod-
els, have been recently proposed in [7].
While LDC exploits bit representation, it is fundamen-
tally different from the VA-file in three ways: 1) LDC uses
one bit for each dimension, while the VA-file uses multi-
ple bits. 2) Pruning in LDC is achieved dynamically by
selecting a subset of DC for bitwise operation, while the
VA-file involves full distance computation between ap-
proximation vectors. 3) LDC is adaptive to skewed data
distributions while the VA-file has been shown to be ef-
fective mainly for uniform datasets [10]. This is further
confirmed in our experiments.
3. Local Digital Coding (LDC)
In this section, we start by presenting the LDC
methodology. Since LDC can easily co-exist with B+-
tree indices, we refer to a B+-tree employing LDC as an
LDC tree. Then, we will discuss how to efficiently con-
struct an LDC tree.
3.1. Structure of the LDC tree
The structure of the LDC tree can be essentially sepa-
rated into three levels. Figure 1 presents an example. The

Page 3

first level of the LDC tree is a B+-tree which is formed by
simply transforming each point in the hyper-dimensional
space into a one-dimensional value. Several methods can
be utilized to perform this transformation; we choose the
method proposed in [12], due to its simplicity. In [12],
the whole data space is first grouped into clusters with
each point in the data space belonging to one of the clus-
ters.
Key Key
...
...
...
Data Points
Data PointsData Points
... ...
...
Key
DC... DC...
Key
DC DC
Figure 1. The overall structure of an LDC tree.
Assuming each cluster is assigned a unique ID, the
one-dimensional value for a data point P in cluster i is
computed as:
key(P) = i ∗ c + d(P,Oi)
where Oiis the center of cluster i, c a constant and d(.)
a distance function, assessing the distance between two
points in the hyper- dimensional space. The minimum
and maximum value of Key() among all points in a clus-
ter, define a range of values, which we refer to as the range
of cluster i. Constant c, which we refer to as the scaling
constant is chosen as to increase the range of each clus-
ter, so that the ranges of different clusters are disjoint
and can thus be distinguished easily. This way, the range
of each cluster i, is [i∗c,(i+1)∗c]. An additional auxil-
iary array is used to store the center and range for each
cluster. Given a D-dimensional point P, the cluster with
center O, that P belongs to, can be determined by com-
puting Key(P) and checking membership of the value
Key(P) in a cluster’s range.
The second level of the LDC tree is the Digital Code
(DC) level. The basic idea of a DC is to produce an ef-
fective single bit representation for each dimension of a
data point. Given a D-dimensional data space, we en-
code each data point into a D-dimensional DC represen-
tation by applying the algorithm in Figure 2. Given a D-
dimensional point P and a cluster center O, initially, the
DC is set to 0 (line 1). For each dimension, if its value
is equal or greater than the value of the corresponding
dimension of the cluster center, we set the correspond-
ing bit of the DC representing that dimension to 1 and
0 otherwise (lines 3 and 4). The DCs are stored sepa-
rately from the index in order to enable fast sequential
scanning of DCs in the SPA algorithm.
The third level of the tree is the data level where the
actual data points are stored.
3.2. Constructing the LDC Tree
The LDC tree is built in three steps. In the first step,
the data space is partitioned into clusters. A plethora of
Local Digital Coding Algorithm
LDC(P,O)
Input: P, O;
Output: DC;
1. unsigned int DC=0;
2. for i=0 to D-1
3.if (P[i] ≥ O[i])
4.DC | = 1 << i;
5. return DC;
Figure 2. Local Digital Coding
proposals exist for clustering high dimensional data sets
[8, 13]. For hyper-dimensional datasets, this process can
be computationally expensive even as a pre-processing
step. A variety of approaches exist for clustering high di-
mensional datasets approximately, trading accuracy for
speed [5]; such techniques can be readily utilized in our
setting, significantly improving the performance of this
preprocessing step.
For simplicity we adopt the following approach for par-
titioning the data set into clusters. We first select the edge
points as done in [12] and then group the data points
into clusters by assigning them to the nearest edge point.
While doing so, the centroid of each cluster can be ap-
proximated. This is done by estimating the median of the
cluster on each dimension through the construction of a
histogram along each dimension. The centroid of each
cluster is then used as the cluster center.
The Key() value for each point (relative to its clus-
ter center) is then computed. A single B+-tree index is
then built on these values. The second step links each dis-
tance value in the leaf nodes of the B+-tree to the cor-
responding point’s DC representation in order. This way
the DC level of LDC is populated. Pages containing DC’s
are sequentially stored so that a sequential scan on DCs
can be performed.
Finally, the third step links each DC to the correspond-
ing data point in order. This way, for each point in the
data space its corresponding DC value is under its cor-
responding Key() value in the leaf nodes of the B+-tree,
and above the actual coordinates of the point.
Dynamic maintenance operations arising from inser-
tions/deletions on an LDC Tree can be easily performed.
When a new point is added in the dataset, its cluster
membership is first determined by identifying the near-
est cluster center. Then its one-dimensional distance and
DC representation is computed, followed by the stan-
dard insertion operation in B+-trees. Correspondingly,
the point’s linked DC and data are inserted into the cor-
rect position in the DC level and data level of the tree
respectively. Deletion operations are quite similar. After
the point’s one-dimensional distance value is computed,
the standard deletion operation in a B+-tree is then per-
formed, followed by removing the corresponding linked
DC and data point.
4. KNN Search in an LDC Tree
We now turn our attention on hyper-dimensional
searching and show how a k-nearest neighbor (KNN)
search can be effectively performed on the LDC tree.

Page 4

Notation
Q,P,O
Q[i], P[i], O[i]
DCQ, DCP
DCQ[i], DCP[i]
Description
Query, data point, cluster center
The ithdimension value for Q,P,O
The DC for Q and P
The corresponding bit for the ithdi-
mension in the DC of Q and P
A dimension ranking array
The ithelement of the dimension
ranking array
Dimensionality of the database
The set ofdimensions
database, |DIM| = D
A proper subset of DIM
Number of dimensions
Pruning length
Candidate pruning length
Partial distance
Threshold for search space or
Maximal number of candidates al-
lowed
rank
rank[i]
D
DIM
forthe
DIM′
s, t
m
n
pd
Θ
Φ
n
m
Table 1. A Table of Notations.
Without loss of generality and to simplify our pre-
sentation we will assume that the distance function
adopted in the search is the Euclidean distance. Thus,
in the remainder of the paper d(.) refers to the Eu-
clidean distance between a pair of points. We summa-
rize the notation used in Table 1 for quick reference.
We first present the theoretical foundation of our algo-
rithm.
4.1. Partial Distance
We begin by formally defining partial distance:
Definition 4.1 (Partial Distance pd(Q,P,DIM′))
Let Q and P be two points in a D-dimensional space and let
Q[i], P[i] denote the values of dimension i in Q and P re-
spectively. We denote the set of D dimensions as DIM.
Given DIM′⊂ DIM, the partial distance between Q and
P is defined as
pd(Q,P,DIM′) =
?
?
i∈DIM′
(Q[i] − P[i])2
Thus, the partial distance between two D-dimensional
points P and Q is in fact the Euclidean distance com-
puted on a subset of the D dimensions. In what follows,
when the parameters of partial distance are clearly im-
plied by the context, we will denote partial distance sim-
ply using pd. We thus have the following corollary:
Corollary 4.1
tance between Q and P, then pd(Q,P,DIM′) ≤ d(Q,P).
Let d(Q,P) denote the Euclidean dis-
Next, we derive a lower bound on the distance of a
query point Q and a data point P, from their correspond-
ing DC representations:
Theorem 4.1
point. We use DCQand DCP to denote the DCs that are
computed with respect to a cluster center O. In addition,
Let Q be a query point and P be a data
we use DCQ[i] and DCP[i] to denote the corresponding bit
in dimension i in both DCs. Let DIM′be a set of dimen-
sions such that ∀i ∈ DIM′,DCQ[i] ?= DCP[i], then
d(P,Q) ≥
?
?
i∈DIM′
(Q[i] − O[i])2
Proof:
Since DCQand DCP differ on the ithbit, this implies
that Q and P are located on different sides of a plane that is
parallel to the ith dimension and passes through O. We can
thusinferthat|Q[i]−O[i]| ≤ |Q[i]−P[i]|foralli ∈ DIM′.
Thus, we conclude that
?
?
i∈DIM′
(Q[i] − O[i])2≤ pd(Q,P,DIM′)
Corollary 4.1 completes the proof.
We now provide an upper bound on the number
of mismatches that can occur between DCQand DCP
(computed with respect to O) if the distance between Q
and P is less than a pruning distance - pd. To do so, we
first introduce the notion of a dimension ranking ar-
ray.
Definition 4.2 (Dimension Ranking Array)
Q be a query point and O be a cluster center. The dimen-
sion ranking array for Q and O is an array, rank, of D
positions such that:
Let
1.Each dimension in the data space is uniquely repre-
sented by one element rank[i] in the array.
2.(Q[rank[0]]−O[rank[0]])≥ Q[rank[1]]−O[rank[1]])
≥ ..... ≥ Q[rank[D − 1]] − O[rank[D − 1]])
Thus, each element rank[i] in the dimension ranking
array for O and Q is the dimension in which |Q[rank[i]]−
O[rank[i]]| is the ithlargest among all the D dimensions.
Theorem 4.2
dimensional points which are computed with respect to a
cluster center O. Let rank be the dimension ranking array
for Q and O. Given a pruning distance pd, we use s to de-
note the largest number such that
Let DCQand DCPbe the DCs of two D-
?
?
?
?
D-1
?
i=D-s
(Q[rank[i]] − O[rank[i]])2≤ pd
Ifd(P,Q) ≤ pd,thenthenumberofbitmismatchesbetween
DCQand DCPcannot be more than s.
Proof:
Assume DCQand DCPhave t mismatches, t ≥ s and
let DIM′represent the set of dimensions in which the t
mismatches between DCQand DCP occurred. According
to Theorem 4.1,
d(P,Q) ≥
?
?
i∈DIM′
(Q[i] − O[i])2
To minimize the R.H.S of the inequality, we should pick
DIM′tobethesetoftdimensionsinwhich|Q[i]−O[i]|are

Page 5

the smallest i.e. the dimensions represented by rank[D−t]
to rank[D − 1]. Since s is the largest value such that
?
?
?
?
D−1
?
i=D−s
(Q[rank[i]] − O[rank[i]])2≤ pd
and t > s, we can thus conclude that
?
?
?
?
D−1
?
i=D−t
(Q[rank[i]] − O[rank[i]])2> pd
By combining the inequalities, we will have d(P,Q) >
pd.
Figure 3 illustrates the relationship between D, s and
t on the ranking array. Notice that given a pruning dis-
tance pd, Theorem 4.2 enables pruning of candidate data
points. Given a query Q, a cluster center O and a prun-
ing distance pd, one can derive the suitable value of s ac-
cording to Theorem 4.2 and all DCs, with more than s
mismatches to the query DC, can be pruned away. Such
a strategy for pruning has two main drawbacks: (a) it in-
volves performing bitwise operations on vectors of D bits,
for a possibly very large D and (b) it provides no way to
control the amount of actual pruning performed.
To overcome these drawbacks, we will compare the
DCs on only those bits that correspond to the top (largest)
n dimensions in the dimension ranking array. Comparing
on only n < D dimensions effectively reduces the number
of bits examined, offering additional savings in processor
time. Assuming n is sufficiently large, we can still deter-
mine a value m based on Theorem 4.2, such that hav-
ing more than m mismatches among the n bits will mean
that d(P,Q) > pd. Comparing on the n largest dimen-
sions, for a specific value of pd, assures that n is as small
as possible. Moreover, as we will see, it enables a flexi-
ble strategy that allows the user to control the amount
of pruning performed.
We note that selecting a good value for n can be diffi-
cult. A large value of n will not bring substantial savings
in processor time. Even more importantly, a small value
of n could result in a low probability of pruning off candi-
date data points. For example, in the extreme case where
n = 1 and m = 0, only 1 bit of the DCs is compared
and there is only a 50% chance of pruning off a candi-
date DC if we assume a uniform distribution of the data
points. In the next section, we will illustrate the relation-
ship between the probability of pruning a partition and
the selection of n and m.
4.2. Selecting the values m and n
In our analysis on the relationship between n and m,
we shall assume that data are uniformly distributed in
the hyper-dimensional space.
Lemma 4.1
vides the portion of the hyper-dimensional space spanned
by the points belonging to a cluster into 2Dpartitions.
Proof: Since the ithbit of a DC represents the two sides of
a plane that is orthogonal to the ithdimension and passes
through a cluster center O, having D bits indicates that
there are D orthogonal planes passing through O. Each of
A D-dimensional DC representation di-
rank[0]
rank[1]
rank[2]
rank[D-1]
rank[D-2]
m
t
s
D
n
Figure 3. Dimensions Ranking Array.
O1
11
00
01
10
Q1
r
DC for Q1 is 11
x
y
dy
dx
max
Figure 4. Searching space in a 2-d space
thesemutuallyindependentplanesdividestheoriginaldata
space into half which gives us 2Dpartitions.
Figure 4 shows an example in 2-dimensional space. It
is evident that the cluster space is divided into four par-
titions by two orthogonal planes. The partitions are iden-
tified by their DCs as 11, 10, 01, and 00 respectively.
Lemma 4.2
set of any n dimensions. If comparisons are made only on
the n bits of the DCs that correspond to DIM′, then the av-
erage number of partitions which share exactly m common
bits with DCQis
LetDCQbeaqueryDCandDIM′beasub-
?
n
m
?
∗ 2(D−n)
Proof: We first notice that each axis-aligned hyperplane,
in expectation, will partition the points in the cluster into
half. Hence, each bit in a data point’s DC has equal proba-
bility of matching or not matching the corresponding bit of
DCQ.TheprobabilitythataDChasexactlymcommonbits
?
Sinceweconsideronlynbits,therewillbe2ndifferentcom-
binations of the n bits. The number of DCs associated with
with DCQis thus
n
m
?
∗(1
2)m∗(1
2)n−m=
?
n
m
?
∗
1
2n.