Abstract:As the heterogeneity of the high-performance computing platform and the scale of data-parallel applications increased significantly, data partition becomes a key issue. Recent works use computation performance model to optimize the data partition algorithm generally. However, these methods cannot take the communication overhead into account, resulting in incompatibility for the applications with high communication ratio or unbalanced communication topology. In this paper, a new heterogeneous-aware data partition algorithm, HaDPA, is proposed. Firstly, the computation and communication overhead are predicted by suitable computation and communication performance models given a partition topology. Then, the search tree is constructed, and the hierarchical deep first search with branch and bound is designed to obtain the optimal solution, which makes up the whole HaDPA process with the constructing of optimizing model. Finally, to verify the performance of the algorithm, Matrix multiplication and axial compressor rotor applications are tested on TianHe-2A supercomputer. Experimental results show that HaDPA can effectively reduce the execution time of data parallel applications. What's more, the impact factors of performance improvement are analyzed and explained. Regression model proofs that the communication to computation ratio matters more to the data-partition on heterogeneous HPC platforms. Besides, compared with HPOPTA, the HaDPA improvement ratio increases with a higher communication ratio and a lower heterogeneity of hardware platform.

Research on Automatic Partitioning of Appended Data in Parallel OLTP Systems

Data Based Application Partitioning and Workload Balance in Distributed Environment

Time-Based Trajectory Data Partitioning for Efficient Range Query.

Research on Mass Process-Planning Data Storing and Parallel Accessing in Large-Scale Manufacturing Enterprise

Research on Mass Process-Planning Data Storing and Parallel Accessing in Large-Scale Manufacturing Enterprise.

Distributed storage and processing of real-time data in smart grid dispatching and control systems

The Research of Distributed Shared Memory Technology in Power System

Enhancing Storage Efficiency and Performance: A Survey of Data Partitioning Techniques

A Data Partitioning Method for Parallel Digital Terrain Analysis

An Adaptive Model For Building Service-Partition System

A Method of Spatial Data Partition for Efficient Parallel Computing of Topological Relations

A Grid-Aided and STR-Tree-based Algorithm for Partitioning Vector Data

Fine-Grained Data Partitioning Framework For Distributed Database Systems

A Study of Performance Optimization Method for Massive Spaito-temporal Data Based on Spatio-temporal Partition Clustering

HaDPA: A Data-Partition Algorithm for Data Parallel Applications on Heterogeneous HPC Platforms

A Two-Level Parallel Decomposition Approach for Transient Stability Constrained Optimal Power Flow

Coexistence of Multiple Partition Plan Based Physical Database Design.

Data Structure Directed Thread Partitioning Method and Execution Model

Effective Data Distribution And Reallocation Strategies For Fast Query Response In Distributed Query-Intensive Data Environments

Improving performance of parallel transaction processing systems by balancing data load on line