Abstract:The amount of data managed in today's Cloud systems has reached an unprecedented scale. In order to speed up query processing, an effective mechanism is to build indexes on attributes that are used in query predicates. However, conventional indexing schemes fail to provide a scalable service: as the size of these indexes are proportional to the data size, it is not space efficient to build many indexes. As such, it becomes more crucial to develop effective index to provide scalable database services in the Cloud. In this paper, we propose a compact bitmap indexing scheme for a large-scale data store. The bitmap indexing scheme combines state-of-the-art bitmap compression techniques, such as WAH encoding and bit-sliced encoding. To further reduce the index cost, a novel and query efficient partial indexing technique is adopted, which dynamically refreshes the index to handle updates and process queries. The intuition of our indexing approach is to maximize the number of indexed attributes, so that a wider range of queries, including range and join queries, can be efficiently supported. Our indexing scheme is light-weight and its creation can be seamlessly grafted onto the MapReduce processing engine without incurring significant running cost. Moreover, the compactness allows us to maintain the bitmap indexes in memory so that performance overhead of index access is minimal. We implement our indexing scheme on top of the underlying Distributed File System (DFS) and evaluate its performance on an in-house cluster. We compare our index-based query processing with HadoopDB to show its superior performance. Our experimental results confirm the effectiveness, efficiency and scalability of the indexing scheme.

Compact Pyramid Tree:Effective Index Structure in High-Dimensional Data Spaces

A New Linkless Hierarchical Structure Based on Perfect Hashing

Indexing High-Dimensional Data in Dual Distance Spaces

Making the Pyramid Technique Robust to Query Types and Workloads

A Clustered Dwarf Structure to Speed Up Queries on Data Cubes

PK-Tree: A Spatial Index Structure for High Dimensional Point Data

Indexing high-dimensional data in dual distance spaces: a symmetrical encoding approach

LIFT:An Index Structure for High Dimensional Data

Contorting High Dimensional Data for Efficient Main Memory KNN Processing

An Efficient and Compact Indexing Scheme for Large-Scale Data Store.

MC-Tree: Dynamic Index Structure for Partially Clustered Multi-Dimensional Database

Packing R-trees with Space-filling Curves

Packing R-trees with Space-filling Curves: Theoretical Optimality, Empirical Efficiency, and Bulk-loading Parallelizability

A Dynamic Pyramid Rtree Index for Massive GeoSpatial Files

Composite Distance Transformation for Indexing and K -Nearest-neighbor Searching in High-Dimensional Spaces

Research on Dynamic Indexing Structure for Multi-Dimensional Vectors

Novel High-Dimensional Indexing Structure Based on Dual-Distance Metric

Effective Data Stream Cube Structure

Indexing high-dimensional data for efficient in-memory similarity search

RR-tree: A New Relational Approach of Multi-dimension Index

Effectively Indexing the Uncertain Space