Abstract:The amount of data managed in today's Cloud systems has reached an unprecedented scale. In order to speed up query processing, an effective mechanism is to build indexes on attributes that are used in query predicates. However, conventional indexing schemes fail to provide a scalable service: as the size of these indexes are proportional to the data size, it is not space efficient to build many indexes. As such, it becomes more crucial to develop effective index to provide scalable database services in the Cloud. In this paper, we propose a compact bitmap indexing scheme for a large-scale data store. The bitmap indexing scheme combines state-of-the-art bitmap compression techniques, such as WAH encoding and bit-sliced encoding. To further reduce the index cost, a novel and query efficient partial indexing technique is adopted, which dynamically refreshes the index to handle updates and process queries. The intuition of our indexing approach is to maximize the number of indexed attributes, so that a wider range of queries, including range and join queries, can be efficiently supported. Our indexing scheme is light-weight and its creation can be seamlessly grafted onto the MapReduce processing engine without incurring significant running cost. Moreover, the compactness allows us to maintain the bitmap indexes in memory so that performance overhead of index access is minimal. We implement our indexing scheme on top of the underlying Distributed File System (DFS) and evaluate its performance on an in-house cluster. We compare our index-based query processing with HadoopDB to show its superior performance. Our experimental results confirm the effectiveness, efficiency and scalability of the indexing scheme.

Organization and management of massive terrain dataset using rectangular bintree

Partition Based Real-Time Rendering Method for Large-Area Terrain Data

Tile quadtree and filling curve realizing massive terrain dataset management

Multi-screen Tiled Displayed, Parallel Rendering System for a Large Terrain Dataset

A New BSP Model for Large Scale Meshes with Point Index

Compression of Multi-Resolution Terrain Data Based on Binary Tree

3D Visualization of Massive Terrain Data Based on Grid Partition

Method for Geometric Data Management and Real-time Rendering of Large-scale Forest Scene

An effective visualization method for large-scale terrain dataset

An Management and Visualization System for Massive Remote Sensing Image: Research and Implementation.

A Dynamic Pyramid Rtree Index for Massive GeoSpatial Files

FAST RENDERING OF MASSIVE TEXTURED TERRAIN DATA

Application and Research of Three Geographic Information System Based on Mass Storage

Multi-resolution organization and management of massive 3D seismic data volume.

Extended algorithm of real-time rendering for 3D terrain in the large volume of data

Huge texture mapping for real-time visualization of large-scale terrain

Efficient BVH-based Collision Detection Scheme with Ordering and Restructuring.

An Efficient and Compact Indexing Scheme for Large-Scale Data Store.

Research on Large Virtual Octree Model for True Three Dimensional Geo-Science Modeling

Study of Seamless Organization and Storage Structure for Massive Spatio-Temporal Data

Management of Massive Spatial Data Based on Feature-Oriented Object Level Model