||Sign In/My Account | View Cart|
|O'Reilly Open Source Convention: July 26-30, Portland, OR.|
Using Bloom Filtersby Maciej Ceglowski
April 08, 2004
Anyone who has used Perl for any length of time is familiar with the lookup hash, a handy idiom for doing existence tests:
As useful as the lookup hash is, it can become unwieldy for very large lists or in cases where the keys themselves are large. When a lookup hash grows too big, the usual recourse is to move it to a database or flat file, perhaps keeping a local cache of the most frequently used keys to improve performance.
Many people don't realize that there is an elegant alternative to the lookup hash, in the form of a venerable algorithm called a Bloom filter. Bloom filters allow you to perform membership tests in just a fraction of the memory you'd need to store a full list of keys, so you can avoid the performance hit of having to use a disk or database to do your lookups. As you might suspect, the savings in space comes at a price: you run an adjustable risk of false positives, and you can't remove a key from a filter once you've added it in. But in the many cases where those constraints are acceptable, a Bloom filter can make a useful tool.
For example, imagine you run a high-traffic online music store along the lines of iTunes, and you want to minimize the stress on your database by only fetching song information when you know the song exists in your collection. You can build a Bloom filter at startup, and then use it as a quick existence check before trying to perform an expensive fetching operation:
In this example, there's a 1% chance that the test will give a false positive, which means the program will perform the expensive fetch operation and eventually return a null result. Still, you've managed to avoid the expensive query 99% of the time, using only a fraction of the memory you would have needed for a lookup hash. As we'll see further on, a filter with a 1% error rate requires just under 2 bytes of storage per key. That's far less memory than you would need for a lookup hash.
Bloom filters are named after Burton Bloom, who first described them in a 1970 paper entitled Space/time trade-offs in hash coding with allowable errors. In those days of limited memory, Bloom filters were prized primarily for their compactness; in fact, one of their earliest applications was in spell checkers. However, there are less obvious features of the algorithm that make it especially well-suited to applications in social software.
Because Bloom filters use one-way hashing to store their data, it is impossible to reconstruct the list of keys in a filter without doing an exhaustive search of the keyspace. Even that is unlikely to be of much help, since the false positives from an exhaustive search will swamp the list of real keys. Bloom filters therefore make it possible to share information about what you have without broadcasting a complete list of it to the world. For that reason, they may be especially valuable in peer-to-peer applications, where both size and privacy are important constraints.
A Bloom filter consists of two components: a set of
All of the hash functions in a Bloom filter are configured so that their range matches the length of the bit vector. For example, if a vector is 200 bits long, the hash functions return a value between 1 and 200. It's important to use high-quality hash functions in the filter to guarantee that output is equally distributed over all possible values -- "hot spots" in a hash function would increase our false-positive rate.
To enter a key into a Bloom filter, we run it through each one of the
As an example, let's take a look at a Bloom filter with three hash functions and a bit vector of length 14. We'll use spaces and asterisks to represent the bit vector, to make it easier to follow along. As you might expect, an empty Bloom filter starts out with all the bits turned off, as seen in Figure 1.
Let's now add the string
Then we turn on the bits at the corresponding positions in the vector -- in this case bits 3, 11, and 12, as shown in Figure 2.
To add another key, such as
And again turn on the appropriate bits in the vector, as shown with highlights in Figure 3.
Notice that the bit at position 11 was already turned on -- we had set
it when we added
Checking to see whether a key already exists in a filter is exactly analogous to adding a new key. We run the key through our set of hash functions, and then check to see whether the bits at those offsets are all turned on. If any of the bits is off, we know for certain the key is not in the filter. If all of the bits are on, we know the key is probably there.
I say "probably" because there's a certain chance our key might be a false
positive. For example, let's see what happens when we test our filter for the
And then examine the bits at those offsets, as shown in Figure 4.
All of the bits at positions 3, 8, and 12 are on, so our filter will report
As you might expect, the false-positive rate depends on the bit vector
length and the number of keys stored in the filter. The roomier the bit vector,
the smaller the probability that all
c = ( 1 - e(-kn/m) )k
When using Bloom filters, we very frequently have a desired false-positive rate in mind and we are also likely to have a rough idea of how many keys we want to add to the filter. We need some way of finding out how large a bit vector is to make sure the false-positive rate never exceeds our limit. The following equation will give us vector length from the error rate and number of keys:
You'll notice another free variable here:
To give you a sense of how error rate and number of keys affect the storage size of Bloom filters, Table 1 lists some sample vector sizes for a variety of capacity/error rate combinations.
You can find further lookup tables for various combinations of error rate, filter size, and number of hash functions at Bloom Filters -- the math.
Copyright © 2000-2004 O’Reilly Media, Inc. All Rights Reserved.