Abstract:With the widespread use of shared-nothing clusters of servers, there has been a proliferation of distributed object stores that offer high availability, reliability and enhanced performance for MapReduce-style workloads. However, relational workloads cannot always be evaluated efficiently using MapReduce without extensive data migrations, which cause network congestion and reduced query throughput. We study the problem of computing data placement strategies that minimize the data communication costs incurred by typical relational query workloads in a distributed setting. Our main contribution is a reduction of the data placement problem to the well-studied problem of {\sc Graph Partitioning}, which is NP-Hard but for which efficient approximation algorithms exist. The novelty and significance of this result lie in representing the communication cost exactly and using standard graphs instead of hypergraphs, which were used in prior work on data placement that optimized for different objectives (not communication cost). We study several practical extensions of the problem: with load balancing, with replication, with materialized views, and with complex query plans consisting of sequences of intermediate operations that may be computed on different servers. We provide integer linear programs (IPs) that may be used with any IP solver to find an optimal data placement. For the no-replication case, we use publicly available graph partitioning libraries (e.g., METIS) to efficiently compute nearly-optimal solutions. For the versions with replication, we introduce two heuristics that utilize the {\sc Graph Partitioning} solution of the no-replication case. Using the TPC-DS workload, it may take an IP solver weeks to compute an optimal data placement, whereas our reduction produces nearly-optimal solutions in seconds.

An Irredundant Decomposition of Data Flow with Affine Dependences

A Comprehensive Dataflow-Mapping Optimization for Fully-Pipelined Execution in Spatial Programmable Architecture

Rubick: A Unified Infrastructure for Analyzing, Exploring, and Implementing Spatial Architectures Via Dataflow Decomposition

Polyhedral-based data reuse optimization for configurable computing

Efficient Path-Sensitive Data-Dependence Analysis

Rubick: A Synthesis Framework for Spatial Architectures Via Dataflow Decomposition.

Improved Affine Partition Algorithm for Compile-Time and Runtime Performance.

Memory Transfer Decomposition: Exploring Smart Data Movement Through Architecture-Aware Strategies.

Optimizing Parallel Performance of Streamline Visualization for Large Distributed Flow Datasets

High performance flow field visualization with high-order access dependencies.

Falcon: A Fused Approach to Path-Sensitive Sparse Data Dependence Analysis

A Mapping Method for Reconfigurable Array Based on Decoupled DataFlow

Automatic multidimensional memory partitioning for FPGA-based accelerators (abstract only).

Efficient Memory Partitioning for Parallel Data Access in Multidimensional Arrays

A Fast and Power-Efficient Memory-Centric Architecture for Affine Computation.

Polyhedral Model Based Data Locality Optimization for Embedded Applications

Distributed Data Placement via Graph Partitioning

Static Generation of Efficient OpenMP Offload Data Mappings

A Quantitative Approach for Adopting Disaggregated Memory in HPC Systems

Rethinking Control Flow in Spatial Architectures: Insights into Control Flow Plane Design

Improving Communication Patterns in Polyhedral Process Networks