Abstract:Managing servers integration to realize distributed data computing framework is an important concern. Regardless of the underlying architecture and the actual distributed system’s complexity, such framework gives programmers an abstract view of systems to achieve variously data-intensive applications. However, some state-of-the-art frameworks need too much library dependencies and parameters configuration, or lack extensibility in application programming. Moreover, general framework’s precise design is a nontrivial work, which is fraught with challenges of task scheduling, message communication and computing efficiency, etc. To address these problems, we present a general, scalable and programmable parallel computing framework called SunwayMR, which only needs GCC/G++ environment. We argue it from the following aspects: (1) Distributed data partitioning, message communication and task organization are given to support transparent application execution on parallel hardware. By searching threads table of each node, the task gets an idle thread (with preferred node IP address) for executing data partition. A novel communication component, SunwayMRHelper, is employed to merge periodical results synchronously. Through identifying whether current node is master or slave, SunwayMR deals with the periodical task’s results differently. (2) As for optimizations, a simple fault tolerance is given to resume data-parallel applications, and thread-level stringstream is utilized to boost computing. To ensure ease-of-use, open Application Programming Interface (API) excerpts can be invoked by various of applications with fewer handwritten code than OpenMPI/MPI. We conduct extensively experimental studies to evaluate the performance of SunwayMR over real-world datasets. Results indicate that SunwayMR (runs on 16 computational nodes) outperforms Spark in various applications, and has good scaling with data sizes, nodes and threads.

Arachne: a portable threads system supporting migrant threads on heterogeneous network farms

Heterothread: Hybrid Thread Level Parallelism on Heterogeneous Multicore Architectures.

Runtime Support for Performance Portability on Heterogeneous Distributed Platforms

A partially reconfigurable architecture supporting hardware threads

Optimizing Memory Access Traffic Via Runtime Thread Migration for On-Chip Distributed Memory Systems

CPC: programming with a massive number of lightweight threads

MPI+Threads: runtime contention and remedies

The Weaves Reconfigurable Programming Framework

Tasking framework for Adaptive Speculative Parallel Mesh Generation

A Machine-Independent port of the MPD language run time system to NetBSD

The Hardware Thread Interface Design and Adaptation on Dynamically Reconfigurable SoC

SunwayMR: A Distributed Parallel Computing Framework with Convenient Data-Intensive Applications Programming.

Xar-Trek: Run-time Execution Migration among FPGAs and Heterogeneous-ISA CPUs

SYNPA: SMT Performance Analysis and Allocation of Threads to Cores in ARM Processors

Novel task migration framework on configurable heterogeneous MPSoC platforms

Heterogeneous Cluster Application Communication Optimization and Computer Big Data Management

Improving Energy Efficiency by Phase-Grained Migration for Asymmetric Multicore

The Approach to Support the P&M Multithreaded Program Execution Model on Workstation Cluster

Arbor -- a morphologically-detailed neural network simulation library for contemporary high-performance computing architectures

Automatic parallelization of fine-grained metafunctions on a chip multiprocessor

FairThreads: mixing cooperative and preemptive threads in C