FastBit is an extremely efficient indexing technology for accelerating database
queries on massive datasets. FastBit enhances conventional bitmap indexing
technology by employing advanced compression, encoding, and binning methods.
Previously, bitmap indexes were only considered useful for categorical data with a
small number of possible values. Now, many applications (especially scientific
applications) require indexing over data with a very large number of possible values.
Because of specialized enhancements, FastBit is fast enough to support real-time
queries for scientific data exploration applications, such as visual analytics. In many
applications, FastBit can search data 10–100 times faster than other products.
As computers become ever more powerful, they
collect and produce more bytes of data. Making
sense of all the bytes is becoming a central challenge
of many scientific endeavors. Often, a small fraction
of the data records holds the key to insight; there-
fore, an efficient tool to locate and retrieve key data
records is essential. In the past 30 years, database
management systems have emerged in industry as
the most prevalent tools for such tasks.
A database management system imposes certain
structures on the data records, typically as tables
with rows and columns, and requires all questions
(queries) to be in a machine processable language,
such as the Structured Query Language (SQL). The
system optimizes the query-answering process by
implementing auxiliary data structures to acceler-
ate common types of questions. These acceleration
techniques are known as database indexes. FastBit
is one such database indexing system, designed pri-
marily for answering queries efficiently.
FastBit can search huge databases much more
quickly than the fastest commercially available
database management system (figure 1). For exam-
ple, researchers from the University of Hamburg,
Germany used FastBit to accelerate their drug dis-
covery software by 140–250 times. Engineers at a
major Internet company found t