Abstract:With the globalization of service, organizations continuously produce large volumes of data that need to be analysed over geo-dispersed locations. Traditionally central approach that moving all data to a single cluster is inefficient or infeasible due to the limitations such as the scarcity of wide-area bandwidth and the low latency requirement of data processing. Processing big data across geo-distributed datacenters continues to gain popularity in recent years. However, managing distributed MapReduce computations across geo-distributed datacenters poses a number of technical challenges: how to allocate data among a selection of geo-distributed datacenters to reduce the communication cost, how to determine the Virtual Machine (VM) provisioning strategy that offers high performance and low cost, and what criteria should be used to select a datacenter as the final reducer for big data analytics jobs. In this paper, these challenges is addressed by balancing bandwidth cost, storage cost, computing cost, migration cost, and latency cost, between the two MapReduce phases across datacenters. We formulate this complex cost optimization problem for data movement, resource provisioning and reducer selection into a joint stochastic integer nonlinear optimization problem by minimizing the five cost factors simultaneously. The Lyapunov framework is integrated into our study and an efficient online algorithm that is able to minimize the long-term time-averaged operation cost is further designed. Theoretical analysis shows that our online algorithm can provide a near optimum solution with a provable gap and can guarantee that the data processing can be completed within pre-defined bounded delays. Experiments on WorldCup98 web site trace validate the theoretical analysis results and demonstrate that our approach is close to the offline-optimum performance and superior to some representative approaches.

A MapReduce Cluster Deployment Optimization Framework with Geo-distributed Data.

A Hierarchical Hadoop Framework to Process Geo-Distributed Big Data

Processing Geo-Dispersed Big Data in an Advanced Mapreduce Framework

Cost-Aware Big Data Processing Across Geo-Distributed Datacenters

The performance of MapReduce: an in-depth study

The Performance of MapReduce

Optimizing MapReduce for Highly Distributed Environments

Improving Performance for Geo-Distributed Data Process in Wide-Area.

QoS-Aware Data Placement for MapReduce Applications in Geo-Distributed Data Centers

Analysis and Optimization of Massive Data Processing on High Performance Computing Architecture

A Survey on Geographically Distributed Big-Data Processing using MapReduce

Software Design and Implementation for MapReduce Across Distributed Data Centers

MapReduce Across Distributed Clusters for Data-intensive Applications

Adaptive priority-based data placement and multi-task scheduling in geo-distributed cloud systems

Traffic-Aware Geo-Distributed Big Data Analytics with Predictable Job Completion Time

MapReduce Task Scheduling in Heterogeneous Geo-Distributed Data Centers

G-Hadoop: MapReduce across distributed data centers for data-intensive computing

Storage Optimization for Energy-Saving Based on Hypergraph in Cloud Data Center

Implementation of MapReduce parallel computing framework based on multi-data fusion sensors and GPU cluster

Energy-efficient Geo-Distributed Big Data Analytics

Optimized Big Data K-means Clustering Using MapReduce