Abstract:In the past twenty years, we have witnessed an unprecedented production of data worldwide that has generated a growing demand for computing resources and has stimulated the design of computing paradigms and software tools to efficiently and quickly obtain insights on such a Big Data. State-of-the-art parallel computing techniques such as the MapReduce guarantee high performance in scenarios where involved computing nodes are equally sized and clustered via broadband network links, and the data are co-located with the cluster of nodes. Unfortunately, the mentioned techniques have proven ineffective in geographically distributed scenarios, i.e., computing contexts where nodes and data are geographically distributed across multiple distant data centers. In the literature, researchers have proposed variants of the MapReduce paradigm that obtain awareness of the constraints imposed in those scenarios (such as the imbalance of nodes computing power and of interconnecting links) to enforce smart task scheduling strategies. We have designed a hierarchical computing framework in which a context-aware scheduler orchestrates computing tasks that leverage the potential of the vanilla Hadoop framework within each data center taking part in the computation. In this work, after presenting the features of the developed framework, we advocate the opportunity of fragmenting the data in a smart way so that the scheduler produces a fairer distribution of the workload among the computing tasks. To prove the concept, we implemented a software prototype of the framework and ran several experiments on a small-scale testbed. Test results are discussed in the last part of the paper.

HOG: Distributed Hadoop MapReduce on the Grid

Multi-discipline computations and collaborative visualization on the campus grid

A Grid Service-Based Engineering Computation Architecture.

A Public Grid Computing Framework Based on a Hierarchical Combination of Middleware

Vhadoop: A Scalable Hadoop Virtual Cluster Platform for MapReduce-Based Parallel Machine Learning with Performance Consideration

DataMPI: Extending MPI to Hadoop-Like Big Data Computing

A Hierarchical Hadoop Framework to Process Geo-Distributed Big Data

Applying Computational Grid Technology to Power System

Evaluating Large Graph Processing in MapReduce Based on Message Passing

The performance of MapReduce: an in-depth study

The Performance of MapReduce

A MapReduce Cluster Deployment Optimization Framework with Geo-distributed Data.

HOGWO: a fog inspired optimized load balancing approach using hybridized grey wolf algorithm

A security framework in G-Hadoop for big data computing across distributed Cloud data centres

Hogmap: Using Sdns To Incentivize Collaborative Security Monitoring

A MapReduce Computing Framework Based on GPU Cluster.

OEHadoop: Accelerate Hadoop Applications by Co-Designing Hadoop with Data Center Network

A High Performance Query Analytical Framework for Supporting Data-Intensive Climate Studies

A Grid Computing Infrastructure Based on Ogsa

An Effective Framework for Distributed Geospatial Query Processing in Grids

Horde: A Parallel Programming Framework for Clusters