Abstract:This paper introduces a programming interface called PARRAY (or Parallelizing ARRAYs) that supports system-level succinct programming for heterogeneous parallel systems like GPU clusters. The current practice of software development requires combining several low-level libraries like Pthread, OpenMP, CUDA and MPI. Achieving productivity and portability is hard with different numbers and models of GPUs. PARRAY extends mainstream C programming with novel array types of distinct features: 1) the dimensions of an array type are nested in a tree, conceptually reflecting the memory hierarchy; 2) the definition of an array type may contain references to other array types, allowing sophisticated array types to be created for parallelization; 3) threads also form arrays that allow programming in a Single-Program-Multiple-Codeblock (SPMC) style to unify various sophisticated communication patterns. This leads to shorter, more portable and maintainable parallel codes, while the programmer still has control over performance-related features necessary for deep manual optimization. Although the source-to-source code generator only faithfully generates low-level library calls according to the type information, higher-level programming and automatic performance optimization are still possible through building libraries of sub-programs on top of PARRAY. The case study on cluster FFT illustrates a simple 30-line code that 2x outperforms Intel Cluster MKL on the Tianhe-1A system with 7168 Fermi GPUs and 14336 CPUs.

On Parallel Array P Systems

Online Scheduling on Parallel Machines: A Survey

Programming Heterogeneous Systems with Array Types

Towards Polytypic Parallel Programming

A Proof System for Disjoint Parallel Quantum Programs

Development of parallel programs on shared data-structures -- Revised version

PARRAY

A Compositional Framework for Developing Parallel Programs on Two-Dimensional Arrays

1 Reasoning about Parallel antum Programs

Parallel Power Flow Algorithm Based on MPI and P-Q Analysis Method

P Systems with Proteins Working in the Minimally Parallel Mode

On the Design and Analysis of Parallel and Distributed Algorithms

L O ] 2 6 O ct 2 01 8 Reasoning about Parallel Quantum Programs

A Survey of Parallel Object-Oriented Language

Parallelization in Calculational Forms.

A Comprehensive Exploration of Languages for Parallel Computing

Parallelizing Prolog on Shared-Memory Multiprocessors

A Class of Parallel Algorithms for Solving Large Sparse Linear Systems on Multiprocessors

On Communication Patterns of Parallel Objects

Parallelization Via Context Preservation.

Ptype System: A Featherweight Parallelizability Detector