Abstract:Data-intensive applications involving the analysis of large datasets often require large amounts of compute and storage resources, for which data locality can be crucial to high throughput and performance. We propose a "data diffusion" approach that acquires compute and storage resources dynamically.. replicates data in response to demand, and schedules computations close to data. As demand increases, more resources are acquired, thus allowing faster response to subsequent requests that refer to the same data; when demand drops, resources are released. This approach can provide the benefits of dedicated hardware without the associated high costs, depending on workload and resource characteristics. To explore the feasibility of data diffusion, we offer both a theoretical and an empirical analysis. We define an abstract model for data diffusion, introduce new scheduling policies with heuristics to optimize real-world performance, and develop a competitive online cache eviction policy. We also offer many empirical experiments to explore the benefits of dynamically expanding and contracting resources based on load, to improve system responsiveness while keeping wasted resources small. We show performance improvements of one to two orders of magnitude across three diverse workloads when compared to the performance of parallel file systems with throughputs approaching 80 Gb/s on a modest cluster of 200 processors. We also compare data diffusion with a best model for active storage, contrasting the difference between a pull-model found in data diffusion and a push-model found in active storage.

Dynamic Data Migration Policies for Query-Intensive Distributed Data Environments

Distributed Model Based on Data Partition and Load Balance Algorithm

Data Based Application Partitioning and Workload Balance in Distributed Environment

Effective Data Distribution And Reallocation Strategies For Fast Query Response In Distributed Query-Intensive Data Environments

DP: Dynamic Prepage in Postcopy Migration for Fixed-Size Data Load.

A Load Balancing Strategy Based on Data Correlation in Cloud Computing

MBA:A Market-Based Approach to Data Allocation and Dynamic Migration for Cloud Database

On Data Staging Strategies for Mobile Accesses to Cloud Services

A Policy of Task Allocation Base on Distributed Cluster Computing Towards Cloud

An Environment-Aware Market Strategy for Data Allocation and Dynamic Migration in Cloud Database

Performance Influence of Live Migration on Multi-Tier Workloads in Virtualization Environments

Cost-minimizing Dynamic Migration of Content Distribution Services into Hybrid Clouds

Title Cost-minimizing dynamic migration of content distributionservices into hybrid clouds

Optimal Operator State Migration for Elastic Data Stream Processing

On Service Migrations in the Cloud for Mobile Accesses

Towards efficient load distribution in big data cloud

Optimal Asynchronous Dynamic Policies in Energy-Efficient Data Centers

The Quest For Scalable Support Of Data-Intensive Workloads In Distributed Systems

A Strategy for Queuing Theory-Based Performance and Energy Management in Heterogeneous Data Centers

Host load Prediction-based GMDH-EA and MMTP for Virtual Machines Load Balancing in Cloud Environment

A Workload-Driven Approach To Dynamic Data Balancing In Mongodb