Abstract:We study the problem of optimizing data storage and access costs on the cloud while ensuring that the desired performance or latency is unaffected. We first propose an optimizer that optimizes the data placement tier (on the cloud) and the choice of compression schemes to apply, for given data partitions with temporal access predictions. Secondly, we propose a model to learn the compression performance of multiple algorithms across data partitions in different formats to generate compression performance predictions on the fly, as inputs to the optimizer. Thirdly, we propose to approach the data partitioning problem fundamentally differently than the current default in most data lakes where partitioning is in the form of ingestion batches. We propose access pattern aware data partitioning and formulate an optimization problem that optimizes the size and reading costs of partitions subject to access patterns. We study the various optimization problems theoretically as well as empirically, and provide theoretical bounds as well as hardness results. We propose a unified pipeline of cost minimization, called SCOPe that combines the different modules. We extensively compare the performance of our methods with related baselines from the literature on TPC-H data as well as enterprise datasets (ranging from GB to PB in volume) and show that SCOPe substantially improves over the baselines. We show significant cost savings compared to platform baselines, of the order of 50% to 83% on enterprise Data Lake datasets that range from terabytes to petabytes in volume.

Cost Optimisation for Distributed Data Warehouses

Towards the Building of a Dense-Region-based OLAP System

Cost-Based Optimization Of Logical Partitions For A Query Workload In A Hadoop Data Warehouse

Requirement-Based Data Cube Schema Design

Design Of Distributed Data Warehouse For Sales Decision Of Large-Scale Clothing Enterprise

Cost-Intelligent Data Analytics in the Cloud

New Distributed Spatial Query Optimization Approach by Using Query Analyzer

A heuristic approach to cost-efficient fragmentation and allocation of complex value databases.

A Heuristic Approach to Cost-Efficient Derived Horizontal Fragmentation of Complex Value Databases.

Query optimization for massively parallel data processing.

Towards Optimizing Storage Costs on the Cloud

Reduce The Wastage of Data During Movement in Data Warehouse

Optimized Design of Materialized Views in a Real-Life Data Warehousing Environment

Moving Big Data to The Cloud: An Online Cost-Minimizing Approach

Distributed coordination of internet data centers for energy cost minimization

Cost-effective Data Analytics Across Multiple Cloud Regions

Query Optimisation As Part of Distribution Design for Complex Value Databases

Hierarchically Distributed Data Warehouse

To store or not: Online cost optimization for running big data jobs on the cloud

Stochastic programming on cost optimization in geographically distributed Internet data centers

Extended Derivation Cube Based View Materialization Selection in Distributed Data Warehouse