Abstract:The amount of data managed in today's Cloud systems has reached an unprecedented scale. In order to speed up query processing, an effective mechanism is to build indexes on attributes that are used in query predicates. However, conventional indexing schemes fail to provide a scalable service: as the size of these indexes are proportional to the data size, it is not space efficient to build many indexes. As such, it becomes more crucial to develop effective index to provide scalable database services in the Cloud. In this paper, we propose a compact bitmap indexing scheme for a large-scale data store. The bitmap indexing scheme combines state-of-the-art bitmap compression techniques, such as WAH encoding and bit-sliced encoding. To further reduce the index cost, a novel and query efficient partial indexing technique is adopted, which dynamically refreshes the index to handle updates and process queries. The intuition of our indexing approach is to maximize the number of indexed attributes, so that a wider range of queries, including range and join queries, can be efficiently supported. Our indexing scheme is light-weight and its creation can be seamlessly grafted onto the MapReduce processing engine without incurring significant running cost. Moreover, the compactness allows us to maintain the bitmap indexes in memory so that performance overhead of index access is minimal. We implement our indexing scheme on top of the underlying Distributed File System (DFS) and evaluate its performance on an in-house cluster. We compare our index-based query processing with HadoopDB to show its superior performance. Our experimental results confirm the effectiveness, efficiency and scalability of the indexing scheme.

FILT: Optimizing KV-Embedded File Systems Through Flat Indexing.

Exploiting Flat Namespace to Improve File System Metadata Performance on Ultra-fast, Byte-addressable NVMs

A Flattened Metadata Service for Distributed File Systems.

FlatFS: Flatten Hierarchical File System Namespace on Non-volatile Memories

LocoFS: a loosely-coupled metadata service for distributed file systems

LearnedKV: Integrating LSM and Learned Index for Superior Performance on SSD

FPGA-Accelerated Compactions for LSM-based Key-Value Store.

HIndex-FLSM: Fragmented Log-Structured Merge Trees Integrated with Heat and Index

MiF: Mitigating the Intra-file Fragmentation in Parallel File System.

InfiniFS: An Efficient Metadata Service for Large-Scale Distributed Filesystems

SepStore: Data Storage Accelerator for Distributed File Systems by Separating Small Files from Large Files.

Reducing Rename Overhead In Full-Path-Indexed File System

<i>SA-LSM</i>: Optimize Data Layout for LSM-tree Based Storage using Survival Analysis

The Design and Implementation of UniKV for Mixed Key-Value Storage Workloads

LightFS: A Lightweight Host-CSD Coordinated File System Optimizing for Heavy Small File Accesses

An Efficient and Compact Indexing Scheme for Large-Scale Data Store.

An Efficient Design and Implementation of LSM-tree Based Key-Value Store on Open-Channel SSD.

HIFFS: A Hybrid Index for Flash File System.

High Velocity Kernel File Systems with Bento

FlashKV

User‐level parallel file system: Case studies and performance optimizations