Abstract:The amount of data managed in today's Cloud systems has reached an unprecedented scale. In order to speed up query processing, an effective mechanism is to build indexes on attributes that are used in query predicates. However, conventional indexing schemes fail to provide a scalable service: as the size of these indexes are proportional to the data size, it is not space efficient to build many indexes. As such, it becomes more crucial to develop effective index to provide scalable database services in the Cloud. In this paper, we propose a compact bitmap indexing scheme for a large-scale data store. The bitmap indexing scheme combines state-of-the-art bitmap compression techniques, such as WAH encoding and bit-sliced encoding. To further reduce the index cost, a novel and query efficient partial indexing technique is adopted, which dynamically refreshes the index to handle updates and process queries. The intuition of our indexing approach is to maximize the number of indexed attributes, so that a wider range of queries, including range and join queries, can be efficiently supported. Our indexing scheme is light-weight and its creation can be seamlessly grafted onto the MapReduce processing engine without incurring significant running cost. Moreover, the compactness allows us to maintain the bitmap indexes in memory so that performance overhead of index access is minimal. We implement our indexing scheme on top of the underlying Distributed File System (DFS) and evaluate its performance on an in-house cluster. We compare our index-based query processing with HadoopDB to show its superior performance. Our experimental results confirm the effectiveness, efficiency and scalability of the indexing scheme.

LIGHT: A Query-Efficient Yet Low-Maintenance Indexing Scheme over DHTs

LHT: A Low-Maintenance Indexing Scheme over DHTs

m-LIGHT: Indexing Multi-Dimensional Data over DHTs

A Lightweight Multidimensional Index for Complex Queries over DHTs

Multi-Dimensional Queries in Dht-Based Peer-To-Peer Systems

P2P Data Indexing and Querying

Multi-dimensional Complex Query Processing over DHT

An Efficient and Compact Indexing Scheme for Large-Scale Data Store.

An effective single-hop distributed hash table with high lookup performance and low traffic overhead

Efficient Range Query Processing over DHTs Based on the Balanced Kautz Tree.

Just-in-time Query Retrieval over Partially Indexed Data on Structured P2P Overlays

Research on Complex Query Processing in Schema-based P2P System

Supporting Efficient Recursive Query Processing in DHTs

Facilitating Effective Resource Publishing and Searching in DHT Networks

Toward an Accurate Snapshot of DHT Networks.

An Efficient Peer-to-peer Indexing Tree Structure for Multidimensional Data.

Supporting complex query with structured overlays in schema-based p2p system

Load balance of the fast similarity search using locality sensitive hashing in structured P2P networks

Secure P2P Topology Based on a Multidimensional DHT Space Mapping

SDI: a Swift Tree Structure for Multi-Dimensional Data Indexing in Peer-to-peer Networks

LDHT: Locality-aware Distributed Hash Tables