Buffered Bloom filters on solid state storage

Abstract

Bloom Filters are widely used in many applications includ-ing database management systems. With a certain allowable error rate, this data structure provides an efficient solution for membership queries. The error rate is inversely pro-portional to the size of the Bloom filter. Currently, Bloom filters are stored in main memory because the low locality of operations makes them impractical on secondary storage. In multi-user database management systems, where there is a high contention for the shared memory heap, the limited memory available for allocating a Bloom filter may cause a high rate of false positives. In this paper we are proposing a technique to reduce the memory requirement for Bloom filters with the help of solid state storage devices (SSD). By using a limited memory space for buffering the read/write requests, we can afford a larger SSD space for the actual Bloom filter bit vector. In our experiments we show that with significantly less memory requirement and fewer hash functions the proposed technique reduces the false positive rate effectively. In addition, the proposed data structure runs faster than the traditional Bloom filters by grouping the inserted records with respect to their locality on the filter.

Full text

Page 1

Buffered Bloom Filters on Solid State Storage
Mustafa Canim
University of Texas at Dallas
Richardson, TX, USA
canim@utdallas.edu
George A. Mihaila
IBM Watson Research Center
Hawthorne, NY, USA
mihaila@us.ibm.com
Bishwaranjan
Bhattacharjee
IBM Watson Research Center
Hawthorne, NY, USA
bhatta@us.ibm.com
Kenneth A. Ross
Columbia University
New York, NY, USA
kar@cs.columbia.edu
Christian A. Lang
IBM Watson Research Center
Hawthorne, NY, USA
langc@us.ibm.com
ABSTRACT
Bloom Filters are widely used in many applications includ-
ing database management systems. With a certain allowable
error rate, this data structure provides an efficient solution
for membership queries.The error rate is inversely pro-
portional to the size of the Bloom filter. Currently, Bloom
filters are stored in main memory because the low locality
of operations makes them impractical on secondary storage.
In multi-user database management systems, where there is
a high contention for the shared memory heap, the limited
memory available for allocating a Bloom filter may cause a
high rate of false positives. In this paper we are proposing
a technique to reduce the memory requirement for Bloom
filters with the help of solid state storage devices (SSD). By
using a limited memory space for buffering the read/write
requests, we can afford a larger SSD space for the actual
Bloom filter bit vector. In our experiments we show that
with significantly less memory requirement and fewer hash
functions the proposed technique reduces the false positive
rate effectively. In addition, the proposed data structure
runs faster than the traditional Bloom filters by grouping
the inserted records with respect to their locality on the
filter.
1.INTRODUCTION
The Bloom filter is a probabilistic data structure opti-
mized for very efficient membership queries [4]. It provides
a very compact representation of sets. This efficiency comes
at the price of reduced precision: specifically, the Bloom fil-
ter may incorrectly indicate that an element is present in
the original set, thus occasionally generating false positives
(but no false negatives). The false positive rate is directly
proportional to the compression ratio. Thus, in order to
lower the false positive rate, a larger amount of memory is
required.
Bloom filters are used in a wide variety of application ar-
eas, such as databases [1], distributed information retrieval [20],
network computing [5], and bioinformatics [15].
these applications require large Bloom filters to reduce the
false positive rate. In memory limited environments dedi-
cating large memories for Bloom filters may not be feasible.
A naive approach to tackle this issue would be to put the
Bloom filter on secondary storage, either explicitly or by
using memory-mapped I/O. Each probe would then require
Some of
one or more random I/O operations, which would become
a severe performance bottleneck. Therefore, simply plac-
ing the filters on disk will not provide a sufficient quality of
service particularly for time critical applications.
A less naive approach can be used when probes come in
bulk. Instead of one large filter, we store δ disjoint filters.
A record is hashed to one of the δ filters during the build
or probe, and the access is performed on the smaller filter.
However, instead of performing the access immediately, we
instead store the access request in a main-memory buffer.
Only when a buffer becomes full (or when we flush a buffer
periodically) do the accesses actually get applied to the cor-
responding filter partition. Because all of the accesses to
the component filter get applied at once, we improve the
access locality, amortizing the I/O cost and cache-miss cost
of retrieving the filter into RAM/cache over a large number
of accesses. This idea is motivated by a similar idea used by
Zhou and Ross for buffering accesses to B-trees to improve
locality of reference [23].
As we shall see experimentally, even this buffering ap-
proach performs poorly when the secondary storage device
is a hard disk. The random I/O-cost is sufficiently large that
it dominates the overall cost and limits throughput. How-
ever, more recent secondary storage devices based on solid
state flash memory have demonstrated much better random
I/O performance. We therefore study whether such devices
have the potential to provide cost-effective solutions for ac-
cessing large Bloom filters.
To perform this study, we have chosen to use what is to-
day a high-end flash memory device, namely the Fusion-IO
card [10]. See Table 1 for the specifications of this device.
As of May 2010, the retail price1of this device is about $38
per gigabyte. A higher-capacity MLC device by the same
vendor has 60% of the throughput, but a price of $15 per gi-
gabyte. Note that such flash devices are in a relatively early
stage of development, and prices are expected to drop as
they become more widely adopted. Industry experts seem
to agree (see e.g., [11]) that within the next few years the
price per Gigabyte of SSDs will be more than an order of
magnitude below the price of main memory, and approach-
ing the price of hard disks.
1All quoted prices were obtained from the Dell web-site.
For comparison purposes, the retail price of server RAM
is about $82 per gigabyte using 8GB DIMMs, or $25 per
gigabyte using 4GB DIMMs.

Page 2

The main contribution of the paper is an SSD-aware Bloom
filter algorithm that reduces the main memory usage by a
factor of at least four without any performance or quality
loss.
1.1Motivating example
To motivate the need for Bloom filters utilizing flash stor-
age, consider the following example based on the TPC-H
schema. Suppose a user poses the following query
Select *
From Lineitem L, Orders O
Where L.Orderkey=O.Orderkey
And L.Quantity>1
And O.Orderstatus=’P’
In relational algebra, we might write this query as
σC1(L) 1J σC2(O)
where C1 and C2 are the local selection conditions on each
table, and J is the join condition on Orderkey. Suppose that
the selectivities of C1 and C2 are θ1 and θ2 respectively, and
that C1 and C2 are statistically independent over the join
of L and O. Note that Orderkey is a foreign key from L to O.
If this database was stored in a distributed fashion, with
L and O stored at different sites, then data transmission can
be reduced by employing a semijoin or Bloom filter [18]. For
example, one could ship
P = πOrderkey(σC2(O))
or a Bloom filter of P from the site hosting O to the site
hosting L in order to reduce the amount of data from L
that needs to be transmitted over the network. When the
network is the bottleneck, such data reduction improves per-
formance significantly.
Now imagine that both O and L are stored in a com-
plex storage subsystem. This subsystem contains a variety
of resources, including multiple persistent devices of various
kinds, including hard disk drives and solid state drives. The
subsystem contains a limited amount of processing power
and memory, sufficient to manage and optimize the stor-
age system, but not designed to perform computations that
require extensive CPU or memory resources. The storage
subsystem serves as the storage layer for many processing
nodes, connected to the storage subsystem via a shared high-
bandwidth channel such as infiniband [12].
Even a high-bandwidth channel can be a performance bot-
tleneck. Since the storage subsystem abstracts a large num-
ber of devices, and since data is requested by many process-
ing nodes, there can be contention for the channel. In such a
situation, it would be advantageous to preprocess data in the
storage subsystem to reduce the bandwidth required. One
straightforward way to reduce bandwidth is to compress the
data.
Another way to reduce bandwidth is to filter out useless
data so that it is not transmitted in the first place.
example of this kind of approach is the Netezza TwinFin
system [19]. That system uses FPGAs in the storage layer
to apply local selections such as C1 and C2 above so that
only rows satisfying the local selections are transmitted to
compute nodes. The data volume is reduced by a factor of
θ1 for L and θ2 for O.
There still remain opportunities for further data reduc-
tion. In the query mentioned above, the join induces a fur-
An
ther data reduction because the local conditions on the two
tables are independent. In fact, the fraction of each table
that contributes to an output row for this query is θ1θ2. It
would be unrealistic to expect the storage subsystem to per-
form the full join. In the absence of special clustering prop-
erties, the join operation would require substantial CPU and
memory resources. What is more, this join might be part of
a larger query plan in which these tables are not joined di-
rectly, and performing the full join of these two tables would
be suboptimal.
Nevertheless, we argue that the storage subsystem could
effectively optimize the bandwidth requirements by applying
a Bloom filter to L and O before transmitting the data. Two
Bloom filters would be created, one for L and one for O,
representing the set of Orderkey values satisfying the local
selection conditions. These filters would then be used to
further reduce the data in the other table before data is
transmitted across the data channel to the compute nodes.
The challenge in such a system is performing the Bloom
filter reduction in a low-resource environment such as a stor-
age subsystem. We expect to have a small amount of RAM
relative to the total disk capacity, and much of that RAM
could be devoted to other functions such as caching. On
the other hand, the storage subsystem does have plentiful
persistent storage. We therefore propose an approach that
leverages the persistent storage, particularly solid state stor-
age that has high random-I/O rates. Our approach allows
efficient Bloom filter reduction to scale to much larger con-
figurations than could be handled with RAM alone.
In the remainder of this paper, we first recap the char-
acteristics of traditional Bloom filters (TBF). In section 3,
we introduce our extensions to the TBF that allows it to
be SSD-resident without sacrificing creation and probing
efficiency. The resulting data structure is called “buffered
Bloom filter” (BBF). Finally, section 4 presents detailed ex-
periments for a variety of BBF usage scenarios. Section 5
discusses related work and section 6 concludes the paper.
2.TRADITIONAL BLOOM FILTERS
To introduce the traditional Bloom filters we use the frame-
work and notations described in [9, 17, 13]. A Bloom filter is
a method to represent a set S = {R1,R2,...,Rn} to support
membership queries. Each element of S, Ri, is any item that
can be an argument of a hash function. It was invented by
Burton Bloom in 1970 [4] and has been used in numerous
applications since then.
A Bloom filter consists of a vector of β bits, initially all
set to 0. To update the filter, k independent hash functions
h1,h2,...,hk all with range {1,...,β} are used. For each ele-
ment R ∈ S, the bit locations hi(R), i ∈ {1,2,...,k}, are set
to 1. Note that a particular bit location could be set to 1
several times since the hash results of different elements may
collide. To check if an element e is in S, the bit locations
hi(e), i ∈ {1,2,...,k}, are checked. If any of them is not set
to 1, we conclude that e is not an element of S. Otherwise,
we assume that e is an element of S, with a certain prob-
ability that the assumption is wrong. If we conclude that
the element is in the set although it is not, that element is
called a false positive.
2.1The Locality Problem
The probability of cache misses has a significant impact
on the execution time of TBF. Each bit location on the filter

Page 3

has the same access probability during the execution of both
build and probe phases. In the worst case the distance be-
tween two consecutive bit locations that are accessed could
be as large as the size of the filter. For small datasets for
which the filter size is smaller than the CPU cache, cache
misses are not expected to occur since the filter is small
enough to fit in the cache. For larger datasets the filter size
should be increased so as to keep the false positive rate be-
low an acceptable threshold. As the filter size increases, the
probability of cache misses will grow proportionally. If the
cache size is c bytes and the filter size is f bytes, then the
probability that an accessed bit of the filter is not cached is
f−c
f. As the cache miss probability increases, the execution
cost of the Bloom filter would increase gradually.
Figure 1: Effect of filter size on the cost of bit op-
erations
To observe the impact of cache misses on the performance
of Bloom filters, we implemented the build phase and probe
phase of the Bloom filter. For both of these phases we have
used 100 million records and a single hash function. We run
the experiment with different filter sizes in the range [4KB,
64MB] on a platform with 2MB CPU cache (L2 cache). In
another experiment setup, we measured how much time is
spent in the other steps of the algorithms and subtracted
that amount from the previous measurements to isolate the
cost of bit operations. The experiment results are given in
Figure 1. The x axis represents the Bloom filter size which is
stored in the memory2. The vertical axis on the left indicates
the average cost of a single bit operation for different filter
sizes. The vertical axis on the right indicates the probabil-
ity of cache misses that is computed using the formula
For instance, if the filter size is 16MB and the CPU cache is
2MB, 14 out of 16 bit accesses are expected to cause a cache
miss. Note that, as the filter size increases, the probabil-
ity of cache miss increases as well and rapidly converges to
1. Moreover, the experimental results are precisely parallel
with the analytical observations.
In the following section we introduce buffered Bloom fil-
ters (BBF) which prevent cache misses effectively regardless
of the size of the dataset. Even though the total size of
the filter exceeds the CPU cache size, the cache misses are
prevented by partitioning the Bloom filter into sub-Bloom
filters. The memory partition in each sub-Bloom filter on
f−c
f.
2In Figure 1 the step size has a finer granularity between
1MB and 4MB
which the bit operations are performed does not exceed the
cache size. This, in turn, prevents the cache misses effec-
tively and yields a significant performance gain.
3.BUFFERED BLOOM FILTERS
Due to the locality problem in the Bloom filters, storing
the filter in lower storage layers such as conventional hard
drives would entail excessive random access cost. SSDs al-
leviate this problem by eliminating mechanical head move-
ments.Based on this observation, one can suggest moving
the filter to the SSD without any modification in the tra-
ditional Bloom filter algorithm. However, this will yield a
significant performance loss since random access speed of
the SSDs is about one order of magnitude slower than that
of memories. In this paper, we propose a modified version
of the traditional Bloom filter so as to implement it on a
solid state disk. To compensate for the performance loss,
we propose the idea of deferring the read and write opera-
tions during the execution of both build and probe phases.
With this improvement the number of SSD page accesses is
reduced by up to three orders of magnitude depending on
the buffer size. As a second improvement, we propose divid-
ing the Bloom filter into virtual sub-Bloom filters to improve
the locality of the bit operations. This modification reduces
the CPU cache misses and helps to compensate for the cost
of IO operations.
Figure 2: Structure of Buffered Bloom Filter
The structure of the BBF is depicted in Figure 2. The
major difference between TBF and BBF is that the latter
has a hierarchical structure. TBF consists of a single layer,
stored in the main memory, which keeps the filter. In con-
trast, there are two layers in the BBF. The upper layer is
called buffer layer and the lower layer is called filter layer.
The buffer layer is stored in the main memory while the
filter layer is stored in the SSD. As the name implies, the
buffer layer is used to buffer the auxiliary information per-
taining to the deferred reads and writes. The filter layer, on
the other hand, is used to store the filter (bit vector). The
filter is divided into virtual pages such that each page keeps
the bit vector of a sub-Bloom filter. If the filter is divided
into δ pages, the buffered Bloom filter can be considered as
a combination of δ separate traditional Bloom filters. Each
sub-Bloom filter has a dedicated buffer block; therefore, the
number of buffer blocks is equal to the number of filter pages.
3.1 The Build Phase
The build phase of the BBF is described in Algorithm 1.
Similar to the TBF, the input parameters include the list
of records, BuildArray, the size of the build array, nb, and
the number of hash functions, k.

Page 4

Algorithm 1 Buffered Bloom Filter - Build Phase:
1: Given BuildArray, nb, k
2: for i = 0 to nb do
3:
Bi ← h0(BuildArray[i])
4:
for j = 1 to k do
5:Ωj ← hj(BuildArray[i])
6:addOffsetValueToBufBlock(Ωj, Bi)
7:
if isBufferBlockFull(Bi) then
8:retrievedBlock ← readPageFromSSD(Bi)
9:
for each offsetVal in BufferBlock[Bi] do
10:retrievedBlock[offsetVal] ← 1
11: removeOffsetValFromBufBlock(offsetVal, Bi)
12:
end for
13: writeUpdatedPageBackToSSD(retrievedBlock)
14:
end if
15:
end for
16: end for
While processing the records of the build array, an initial
hash function, h0, is used to determine the sub-Bloom filter
to which the records will be assigned. For each record Ri in
the BuildArray, the block number Bi is equal to h0(Ri). If
the filter is divided into δ virtual pages, Bi would be a value
in the range [0, δ − 1].
Once the block number is determined (line 3), subsequent
hash functions, h1,..,hk, are used to find the offset values
for Ri, that is, Ωj = hj(Ri)∀j ∈ (1,...,k), in each iteration
of the for loop starting at line 4. The range of Ωj is [0,β]
where β is the size of the filter page (i.e. total number of
bits in a filter page). If Ωj is equal to x, the xthbit of the
filter page is expected to be set to 1. However, instead of
updating the Bithfilter page immediately, the offset value,
Ωj, is placed to the available position at the Bithbuffer
block (line 6). For each record, k offset values are computed
and stored in the buffer blocks. Note that in TBF, for any
record, x would be any bit location in the filter. In contrast,
in BBF the range of x is limited to the size of a filter page. If
the same record is processed in the probe phase, the record
will be forwarded to this sub-Bloom filter to check the bit
values corresponding to the hash values of the record.
At any given time each buffer block can be either in “Grow-
ing Phase” or “Shrinking phase”. Initially, all buffer blocks
are assumed to be in the growing phase. As new records are
processed, the buffer blocks are filled with the offset values
(line 6). Once a block gets full, the growing phase of the
block ends and the shrinking phase starts by retrieving the
corresponding page from the SSD to the memory (line 8).
In the shrinking phase, for each offset value waiting in the
buffer block, the corresponding bit position of the retrieved
page is set to one (line 10) and the offset value is removed
from the buffer block (line 11). After processing all of the
offset values in the buffer block, the page is written back to
the SSD (line 13). Once the offset values are removed, the
buffer block gets ready for the next growing phase.
The growing and shrinking phases of the buffer blocks
continue as long as new records are read from the stream.
Each buffer block enters the growing phase and the shrinking
phase consecutively. At any time, at most one buffer block
could be in the shrinking phase. The growing phase does not
require any disk accesses. All of the updates in this phase
are performed on the main memory. On the other hand,
the shrinking phase requires retrieval of an SSD page and
writing this updated page back to the SSD.
3.2The Probe Phase
The probe phase of the BBF is described in Algorithm 2.
The input parameters include the list of records, ProbeArray,
the number of records in the probe list, np and the number
of hash functions, k.
Initially, all buffer blocks are assumed to be in the growing
phase. Similar to the build phase, for each record Ri in
ProbeArray, the initial hash function is used to determine
the sub-Bloom filter to which the record will be assigned. If
Bi is equal to h0(Ri), Ri is placed to the available position
at the Bithbuffer block (line 4).
full, the shrinking phase starts and the corresponding page
is retrieved from the SSD to the memory (line 6). For each
record waiting in the buffer block, subsequent offset values
are computed using the hash functions. If all of the bits
in the retrieved page corresponding to the computed hash
values are set to 1, the record is output as positive. If any
of these bits is set to 0, the record is output as negative
indicating that the record ID does not exist in the build list.
The same procedure is applied as long as new probe records
are queried.
If the buffer block gets
Algorithm 2 Buffered Bloom Filter - Probe Phase:
1: Given ProbeArray, np, k
2: for i = 0 to np do
3:
Bi ← h0(ProbeArray[i])
4: addRecordIDToBufBlock(ProbeArray[i], Bi)
5:
if isBufferBlockFull(Bi) then
6:retrievedBlock ← readSSDpage(Bi)
7:
for each recordID in BufferBlock[Bi] do
8:
for j = 1 to k do
9:Ωj ← hj(recordID)
10:
if retrievedBlock[Ωj] = 0 then
11:outputNegative(recordID)
12:break
13:
else
14:
if j = k then
15: outputPositive(recordID)
16:
end if
17:
end if
18:
end for
19:removeRecordIDFromBufBlock(recordID, Bi)
20:
end for
21:
end if
22: end for
3.3Probability of False Positives in BBF
In this section we show that PFP in BBF is independent
of the number of sub-Bloom filters and therefore is equal to
PFP in TBF.
Let b be the number of bits in a filter page and δ be the
number of buffer blocks ( i.e. the Bloom filter is divided into
δ sub-Bloom filters). In the shrinking phase of a buffer block
Bi, after processing an element Ri, the probability that a
single bit of the filter page is not set to 1 by any of the k
functions is:
„
1 −1
b
«k
(1)
Assuming that the hash function, h0, used to assign the
records to the buffer blocks is perfectly random, the ex-
pected number of records assigned to each buffer block will
be:
υ = nb×1
δ
(2)

Page 5

After inserting υ elements to the Bithbuffer block, the prob-
ability that a single bit of the filter page is not set to 1 by
any of the k functions is:
„
1 −1
b
«k×υ
(3)
Hence, at the end of the build phase, the probability that
a certain bit in the Bithfilter page is set to 1 by at least one
hash function is:
1 −
„
1 −1
b
«k×υ
(4)
Now, PFP, the probability of getting a false positive for an
element assigned to Bi is:
PFP =

1 −
„
1 −1
b
«k×υ!k
≈
“
1 − e
−k×υ
b
”k
(5)
Let β be the total number of bits in the filter. Then δ, the
number of buffer blocks, is equal toβ
of bits in a filter page. Substituting for δ in equation 2, we
get υ = nb×
positives is now given by the following equation:
b, where b is the number
b
β. The formula for the probability of false
PFP ≈

1 − e
−k×nb×b
b
β
!k
=
„
1 − e
−k×nb
β
«k
(6)
4. EXPERIMENTS
In this section, we first give the hardware and software
specifications of the system where we conducted the exper-
iments. In Section 4.2, we then discuss the starvation prob-
lem in response time critical applications.
tion 4.3 we compare the performance of the proposed algo-
rithm running on an SSD versus an HDD. In Section 4.4, we
examine the impact of the buffer size on the performance of
the BBF. We conclude this section by presenting the obser-
vations on the impact of the ratio of probe to build records.
4.1Hardware & Software Specifications
All experiments are conducted on a 64 bit Fedora 8 (Linux
kernel 2.6.24) operating system. IBM DB2 V.9 is used as
the DBMS software. The system that is used to run all the
experiments has an Intel(R) Core(TM)2 Quad CPU Q6600
@ 2.40GHz with 2MB L2 cache per each core and 4GB mem-
ory. Hardware specifications for the solid state disk are given
in Table 1.
Then, in Sec-
Table 1: Hardware specifications of the SSD
Brand:Fusion IO
NAND TypeSingle Level Cell (SLC)
Storage capacity:80GB
Interface:PCI-Express x4
Write Bandwidth:550 MB/s (random 16K)
Read Bandwidth:700 MB/s (random 16K)
IOPS: 88,000 (70/30 random 4k mix)
Access Latency:50µs Read
Wear Leveling:24yrs (@ 5-TB write-erase/day)
4.2Starvation Avoidance
Bloom filters are used in numerous applications. In some
of these the filter is unaltered over time and is mostly used
for processing the incoming stream of queries. Using Bloom
filters as spell checkers in word processors is an example
where the filter remains almost the same as the dictionary
words are not expected to change frequently [16]. In some
applications the response time could be critical. Once the
probe requests of the client application(s) are inserted into
a request queue, the applications may expect to get the re-
sult of the query within an acceptable time period. This is
not a concern for the traditional Bloom filter as the probing
requests are processed and output in the same order of inser-
tion into the request queue. In contrast to TBF, the buffered
Bloom filter does not guarantee that the output order of the
requests will be similar to the order of the request queue.
Therefore, the data structure does not guarantee that the
queries will be processed within a certain amount of time.
Even though a perfectly random hash function is used to
distribute the probe requests to the buffer blocks, the pro-
cessing latency of the request will depend on the distribution
of the probes. If a specific record is probed frequently, the
buffer block corresponding to this record will fill up faster
than other buffer blocks. This, in turn, may cause some of
the probes to wait for a long time while some of them are
processed and output faster. To avoid starvation the buffer
blocks should be flushed if a certain timeout period, tmax,
is exceeded even though the block has not been filled up.
A dedicated timer is assigned to each buffer block to keep
track of time since the last flushing. Once a probe record is
inserted into an empty buffer block, the timer is reset. After
a while the buffer block is flushed, provided that the block
is not flushed within tmax unit of time.
Figure 3: The impact of tmax on the average waiting
time of the probe requests
One can suggest choosing tmax as small as possible in or-
der to reduce the average waiting time of the probe requests.
However setting tmax below a certain threshold would in-
crease the average waiting time of the probe requests. If
tmax is very small, fewer records will be processed while
flushing the buffer blocks. Due to the cost of I/O opera-
tions, this will decrease the overall throughput even though
the waiting time of the records in the buffer blocks is re-
duced. As a side effect, the waiting time of probe requests
in the request queue will be much longer. Therefore tmax
should be chosen deliberately so as to minimize the waiting
time of the records in both the request queue and the buffer
blocks.
We conduct an experiment in order to analyse the impact
of tmax on the average waiting time of the probe requests.
In this experiment 50 million records are inserted into the