Abstract:In the past twenty years, we have witnessed an unprecedented production of data worldwide that has generated a growing demand for computing resources and has stimulated the design of computing paradigms and software tools to efficiently and quickly obtain insights on such a Big Data. State-of-the-art parallel computing techniques such as the MapReduce guarantee high performance in scenarios where involved computing nodes are equally sized and clustered via broadband network links, and the data are co-located with the cluster of nodes. Unfortunately, the mentioned techniques have proven ineffective in geographically distributed scenarios, i.e., computing contexts where nodes and data are geographically distributed across multiple distant data centers. In the literature, researchers have proposed variants of the MapReduce paradigm that obtain awareness of the constraints imposed in those scenarios (such as the imbalance of nodes computing power and of interconnecting links) to enforce smart task scheduling strategies. We have designed a hierarchical computing framework in which a context-aware scheduler orchestrates computing tasks that leverage the potential of the vanilla Hadoop framework within each data center taking part in the computation. In this work, after presenting the features of the developed framework, we advocate the opportunity of fragmenting the data in a smart way so that the scheduler produces a fairer distribution of the workload among the computing tasks. To prove the concept, we implemented a software prototype of the framework and ran several experiments on a small-scale testbed. Test results are discussed in the last part of the paper.

HaRD: a heterogeneity-aware replica deletion for HDFS

An Experimental Evaluation of Performance of A Hadoop Cluster on Replica Management

A Request Skew Aware Heterogeneous Distributed Storage System Based on Cassandra

A Heterogeneity-aware Data Distribution and Rebalance Method in Hadoop Cluster

ERMS: an Elastic Replication Management System for HDFS

Heterogeneous Replicas for Multi-dimensional Data Management

An Efficient Replicated System for the Metadata of HDFS

Optimize Multidimensional Arrays Queries with Heterogeneous Replica Method

Hadoop High Availability Through Metadata Replication

Rapid Replica Copy Algorithm Based on Popularity in Hadoop

Replica-aware Data Recovery Performance Improvement for Hadoop System with NVM

Heterogeneity-aware Data Regeneration in Distributed Storage Systems

Acb-R: An Adaptive Clustering-Based Data Replication Algorithm On A P2p Data-Store

MapReduce Performance Optimizing through Replica Placement Strategy

BT-Duper: A Binomial-Tree Based Data Replication Offloading Method with Native RDMA Primitives

A Data-Aware Remote Procedure Call Method for Big Data Systems

Boosting Degraded Reads in Heterogeneous Erasure-Coded Storage Systems

Optimizing Hadoop Block Placement Policy and Cluster Blocks Distribution

Accelerating Data Regeneration for Distributed Storage Systems with Heterogeneous Link Capacities

A Hierarchical Hadoop Framework to Process Geo-Distributed Big Data

Triple-H: A Hybrid Approach to Accelerate HDFS on HPC Clusters with Heterogeneous Storage Architecture