Abstract:We study the problem of optimizing data storage and access costs on the cloud while ensuring that the desired performance or latency is unaffected. We first propose an optimizer that optimizes the data placement tier (on the cloud) and the choice of compression schemes to apply, for given data partitions with temporal access predictions. Secondly, we propose a model to learn the compression performance of multiple algorithms across data partitions in different formats to generate compression performance predictions on the fly, as inputs to the optimizer. Thirdly, we propose to approach the data partitioning problem fundamentally differently than the current default in most data lakes where partitioning is in the form of ingestion batches. We propose access pattern aware data partitioning and formulate an optimization problem that optimizes the size and reading costs of partitions subject to access patterns. We study the various optimization problems theoretically as well as empirically, and provide theoretical bounds as well as hardness results. We propose a unified pipeline of cost minimization, called SCOPe that combines the different modules. We extensively compare the performance of our methods with related baselines from the literature on TPC-H data as well as enterprise datasets (ranging from GB to PB in volume) and show that SCOPe substantially improves over the baselines. We show significant cost savings compared to platform baselines, of the order of 50% to 83% on enterprise Data Lake datasets that range from terabytes to petabytes in volume.

An Email Server Optimized for Storage Issues

Design of a Mail Server for Bandwidth Saving in Cloud Environment

Storage Model Based on Distributed Cache for Message Oriented Middleware

A Novel Scalable Architecture of Cloud Storage System for Small Files Based on P2P

Performance Analysis and Framework Optimization of Open Source Cloud Storage System

Research and Implementation of a Scalable Distributed Mail System

DCStore: A Deduplication-Based Cloud-of-Clouds Storage Service

Serverless Architecture for Bulk Email Management

Towards Optimizing Storage Costs on the Cloud

An Indexing Framework for Efficient Retrieval on the Cloud.

Cloud Storage System for Small File Based on P2P

Scheme for Improving the Performance of Multimedia Storage Systems

A Novel Solution of Distributed File Storage for Cloud Service

ASSER: an Efficient, Reliable, and Cost-Effective Storage Scheme for Object-Based Cloud Storage Systems

AmazingStore: Available, Low-Cost Online Storage Service Using Cloudlets

PC-Cluster based Storage System Architecture for Cloud Storage

An Optimized Replica Distribution Method in Cloud Storage System

Research on Data Storage Technology in Cloud Computing Environment

Skewly replicating hot data to construct a power-efficient storage cluster.

A Novel Optimization Method to Improve De-duplication Storage System Performance

Hybrid Caching for Cloud Storage to Support Traditional Application