Abstract:This paper introduces a programming interface called PARRAY (or Parallelizing ARRAYs) that supports system-level succinct programming for heterogeneous parallel systems like GPU clusters. The current practice of software development requires combining several low-level libraries like Pthread, OpenMP, CUDA and MPI. Achieving productivity and portability is hard with different numbers and models of GPUs. PARRAY extends mainstream C programming with novel array types of distinct features: 1) the dimensions of an array type are nested in a tree, conceptually reflecting the memory hierarchy; 2) the definition of an array type may contain references to other array types, allowing sophisticated array types to be created for parallelization; 3) threads also form arrays that allow programming in a Single-Program-Multiple-Codeblock (SPMC) style to unify various sophisticated communication patterns. This leads to shorter, more portable and maintainable parallel codes, while the programmer still has control over performance-related features necessary for deep manual optimization. Although the source-to-source code generator only faithfully generates low-level library calls according to the type information, higher-level programming and automatic performance optimization are still possible through building libraries of sub-programs on top of PARRAY. The case study on cluster FFT illustrates a simple 30-line code that 2x outperforms Intel Cluster MKL on the Tianhe-1A system with 7168 Fermi GPUs and 14336 CPUs.

Parallel Array-rewriting P Systems

PRAM: A Novel Model For 2-Dimensional Reconfigurable Arrays

One-dimensional Array Grammars and P Systems with Array Insertion and Deletion Rules

Label Languages of 8-directional Array P System

A General Framework for Parallel Planar Mesh Generation

Programming Heterogeneous Systems with Array Types

A High Level Compiling Implementation of PARLOG Based on Extended Graph Rewriting.

Regular Abstractions for Array Systems

Minimization Strategies for Maximally Parallel Multiset Rewriting Systems

Parallel Hyperedge Replacement Grammars

Parallel Algorithms for Reversal Distance of Permutations on PRAM and LARPBS

Confluence of algebraic rewriting systems

Development of parallel programs on shared data-structures -- Revised version

PARRAY

Towards Polytypic Parallel Programming

Analysing Parallel Complexity of Term Rewriting

Periodic Parallelogram Polyominoes

A Parameterized ParaIIeIism Memory TempIate for Affine Array Subscript AppIication

Parallel Summation in P-Recursive Extensions

Row-Polymorphic Types for Strategic Rewriting

Language properties and Grammar of Parallel and Series Parallel Languages