Abstract:The amount of data managed in today's Cloud systems has reached an unprecedented scale. In order to speed up query processing, an effective mechanism is to build indexes on attributes that are used in query predicates. However, conventional indexing schemes fail to provide a scalable service: as the size of these indexes are proportional to the data size, it is not space efficient to build many indexes. As such, it becomes more crucial to develop effective index to provide scalable database services in the Cloud. In this paper, we propose a compact bitmap indexing scheme for a large-scale data store. The bitmap indexing scheme combines state-of-the-art bitmap compression techniques, such as WAH encoding and bit-sliced encoding. To further reduce the index cost, a novel and query efficient partial indexing technique is adopted, which dynamically refreshes the index to handle updates and process queries. The intuition of our indexing approach is to maximize the number of indexed attributes, so that a wider range of queries, including range and join queries, can be efficiently supported. Our indexing scheme is light-weight and its creation can be seamlessly grafted onto the MapReduce processing engine without incurring significant running cost. Moreover, the compactness allows us to maintain the bitmap indexes in memory so that performance overhead of index access is minimal. We implement our indexing scheme on top of the underlying Distributed File System (DFS) and evaluate its performance on an in-house cluster. We compare our index-based query processing with HadoopDB to show its superior performance. Our experimental results confirm the effectiveness, efficiency and scalability of the indexing scheme.

Adaptively incremental dictionary compression method for column-oriented database

Exploiting Multiple Similarity Spaces for Efficient and Flexible Incremental Update of Mobile Apps

Adaptive Compression Algorithm Selection Using LSTM Network in Column-oriented Database

Data Compression for Analytics over Large-scale In-memory Column Databases

A Compression Algorithm for English-Text Based on Dynamic Dictionary

DEC: An Efficient Deduplication-Enhanced Compression Approach

A Novel Approach to Compress Centralized Text Data using Indexed Dictionary

An Efficient and Compact Indexing Scheme for Large-Scale Data Store.

Upscaledb: Efficient Integer-Key Compression in a Key-Value Store using SIMD Instructions

CompressDB: Enabling Efficient Compressed Data Direct Processing for Various Databases

LeCo: Lightweight Compression Via Learning Serial Correlations

Efficient and Scalable Data Evolution with Column Oriented Databases

Efficient Immediate-Access Dynamic Indexing

Compressed Data Direct Computing for Databases

Blitzcrank: Fast Semantic Compression for In-Memory Online Transaction Processing

Ddelta: A Deduplication-Inspired Fast Delta Compression Approach

An Efficient Compression Method of Relational Database

Storage and Operation of Massive Data Compression Based on Predication Index

Dynamic Dictionary with Subconstant Wasted Bits per Key

Multi-Scale Structured Dictionary Learning for 3-D Point Cloud Attribute Compression