Abstract:Traditional High-Performance Computing (HPC) based big-data applications are usually constrained by having to move large amount of data to compute facilities for real-time processing purpose. Modern HPC systems, represented by High-Throughput Computing (HTC) and Many-Task Computing (MTC) platforms, on the other hand, intend to achieve the long-held dream of moving compute to data instead. This kind of data-aware scheduling, typically represented by Hadoop MapReduce, has been successfully implemented in its Map Phase, whereby each Map Task is sent out to the compute node where the corresponding input data chunk is located. However, Hadoop MapReduce limits itself to a one-map-to-one-reduce framework, leading to difficulties for handling complex logics, such as pipelines or workflows. Meanwhile, it lacks built-in support and optimization when the input datasets are shared among multiple applications and/or jobs. The performance can be improved significantly when the knowledge of the shared and frequently accessed data is taken into scheduling decisions.To enhance the capability of managing workflow in modern HPC system, this paper presents CloudFlow, a Hadoop MapReduce based programming model for cloud workflow applications. CloudFlow is built on top of MapReduce, which is proposed not only being data aware, but also shared-data aware. It identifies the most frequently shared data, from both task-level and job-level, replicates them to each compute node for data locality purposes. It also supports user-defined multiple Map- and Reduce functions, allowing users to orchestrate the required data-flow logic. Mathematically, we prove the correctness of the whole scheduling framework by performing theoretical analysis. Further more, experimental evaluation also shows that the execution runtime speedup exceeds 4X compared to traditional MapReduce implementation with a manageable time overhead. (C) 2014 Elsevier B.V. All rights reserved.

An Advanced MapReduce: Cloud MapReduce, Enhancements and Applications

The performance of MapReduce: an in-depth study

The Performance of MapReduce

Multimedia Applications and Security in MapReduce: Opportunities and Challenges

Distributed data management using MapReduce

Energy-efficient Multi-task Scheduling Based on MapReduce for Cloud Computing

Cloud Computing: Data-Intensive Computing and Scheduling

Security and Privacy Aspects in MapReduce on Clouds: A Survey

Location-Aware MapReduce in Virtual Cloud

GENMR: Generalized Query Processing through Map Reduce In Cloud Database Management System

Dynamic Token Based Improving Mapreduce Performance in Cloud Computing

MAP-JOIN-REDUCE: Toward Scalable and Efficient Data Analysis on Large Clusters

LLMapReduce: Multi-Level Map-Reduce for High Performance Data Analysis

Scheduling Data Intensive Workloads through Virtualization on MapReduce based Clouds

Service replication with mapreduce in clouds

A Survey of Big Data Machine Learning Applications Optimization in Cloud Data Centers and Networks

Practical Verifiable Computation–A MapReduce Case Study

Privacy-Preserving Layer over MapReduce on Cloud

Cloudflow: A Data-Aware Programming Model for Cloud Workflow Applications on Modern Hpc Systems

A MapReduce Cluster Deployment Optimization Framework with Geo-distributed Data.

Skyline Computation under MapReduce Framework