Abstract:The amount of data managed in today's Cloud systems has reached an unprecedented scale. In order to speed up query processing, an effective mechanism is to build indexes on attributes that are used in query predicates. However, conventional indexing schemes fail to provide a scalable service: as the size of these indexes are proportional to the data size, it is not space efficient to build many indexes. As such, it becomes more crucial to develop effective index to provide scalable database services in the Cloud. In this paper, we propose a compact bitmap indexing scheme for a large-scale data store. The bitmap indexing scheme combines state-of-the-art bitmap compression techniques, such as WAH encoding and bit-sliced encoding. To further reduce the index cost, a novel and query efficient partial indexing technique is adopted, which dynamically refreshes the index to handle updates and process queries. The intuition of our indexing approach is to maximize the number of indexed attributes, so that a wider range of queries, including range and join queries, can be efficiently supported. Our indexing scheme is light-weight and its creation can be seamlessly grafted onto the MapReduce processing engine without incurring significant running cost. Moreover, the compactness allows us to maintain the bitmap indexes in memory so that performance overhead of index access is minimal. We implement our indexing scheme on top of the underlying Distributed File System (DFS) and evaluate its performance on an in-house cluster. We compare our index-based query processing with HadoopDB to show its superior performance. Our experimental results confirm the effectiveness, efficiency and scalability of the indexing scheme.

The Chunk-Locality Index: An Efficient Query Method for Climate Datasets

Orthogonal Locality Preserving Indexing

Document Clustering Using Locality Preserving Indexing

An Efficient and Compact Indexing Scheme for Large-Scale Data Store.

Enhanced Locality Sensitive Clustering in High Dimensional Space

A Study of Performance Optimization Method for Massive Spaito-temporal Data Based on Spatio-temporal Partition Clustering

An Efficient High Dimensional Cluster Method and Its Application in Global Climate Sets.

Scalable Top-K Spatial Keyword Search

Efficient B-tree Based Indexing for Cloud Data Processing.

2L-LSH: A Locality-Sensitive Hash Function-Based Method For Rapid Point Cloud Indexing

Research on the Analysis and Statistic of Geographical Conditions Based on the Strategy of "grid Index + MapReduce"

Study on Spatio-Temporal Indexing Model of Geohazard Monitoring Data Based on Data Stream Clustering Algorithm

Efficient Bulk Loading to Accelerate Spatial Keyword Queries

Using Bitmap Index to Accelerate Accessing Large Scale Scientific Data on Demand

An Indexing Framework for Efficient Retrieval on the Cloud.

An Improved Method of Locality Sensitive Hashing for Indexing Large-Scale and High-Dimensional Features

An Efficient Indexing Structure for Multidimensional Categorical Range Aggregation Query

CLSH: Cluster-based Locality-Sensitive Hashing

An Efficient Graph-Based Spatio-Temporal Indexing Method for Task-Oriented Multi-Modal Scene Data Organization

A High Performance Compression Method For Climate

Data Independent Method of Constructing Distributed LSH for Large-Scale Dynamic High-Dimensional Indexing