Abstract:The use of large-scale machine learning and data mining methods is becoming ubiquitous in many application domains ranging from business intelligence and bioinformatics to self-driving cars. These methods heavily rely on matrix computations, and it is hence critical to make these computations scalable and efficient. These matrix computations are often complex and involve multiple steps that need to be optimized and sequenced properly for efficient execution. This paper presents new efficient and scalable matrix processing and optimization techniques for in-memory distributed clusters. The proposed techniques estimate the sparsity of intermediate matrix-computation results and optimize communication costs. An evaluation plan generator for complex matrix computations is introduced as well as a distributed plan optimizer that exploits dynamic cost-based analysis and rule-based heuristics to optimize the cost of matrix computations in an in-memory distributed environment. The result of a matrix operation will often serve as an input to another matrix operation, thus defining the matrix data dependencies within a matrix program. The matrix query plan generator produces query execution plans that minimize memory usage and communication overhead by partitioning the matrix based on the data dependencies in the execution plan. We implemented the proposed matrix processing and optimization techniques in Spark, a distributed in-memory computing platform. Experiments on both real and synthetic data demonstrate that our proposed techniques achieve up to an order-of-magnitude performance improvement over state-of-the-art distributed matrix computation systems on a wide range of applications.

Memory-Based High-Performance Optimization for High Concurrent Data-Intensive Problems

Improving the Memory Efficiency of In-Memory MapReduce Based HPC Systems.

Network Optimization for Distributed Memory File System on High Performance Computers

Performance Optimization for In-Memory File Systems on NUMA Machines

High Performance Memory Management for a Multi-core Architecture.

Performance Optimization of In-Memory File System in Distributed Storage System

Efficient Scheduling with Intensive In-Memory File Accesses Considering Bandwidth Constraint on Memory Bus.

Memory System Optimization for Graph Processing: a Survey

A Self-Organizing Distributed Memory Cache For Data Sharing Applications In Cluster Environment

An adaptive memory tuning strategy with high performance for Spark.

Efficient Memory Partitioning for Parallel Data Access in Multidimensional Arrays

“Anti-Caching”-based elastic memory management for Big Data

In-memory Distributed Matrix Computation Processing and Optimization

Optimizing Pipelined Execution For Distributed In-Memory Olap System

In-memory big data analytics under space constraints using dynamic programming.

Using Locality-Enhanced Distributed Memory Cache to Accelerate Applications on High Performance Computers

Parallel Program Performance Optimization for Data-Intensive Applications on Multi-Core Clusters

Improving In-Memory File System Reading Performance by Fine-Grained User-Space Cache Mechanisms

Analysis and Optimization of Massive Data Processing on High Performance Computing Architecture

Towards Efficient NVDIMM-based Heterogeneous Storage Hierarchy Management for Big Data Workloads

Efficient Distributed Data Structures for Future Many-core Architectures