Abstract:Multi-level buffer cache hierarchies are now commonly seen in most client/server cluster configurations, especially in today's big data application deployment. However, multi-level caching policies deployed so far typically use independent cache replacement algorithms in each level, which has two major drawbacks: (1) File blocks may be redundantly cached on multiple levels, reducing the actual aggregate cache usable size; (2) Less accurate replacement decisions at lower level caches due to weakened locality. Inefficient cache resource usage may result in noticeable performance degradation for big data applications. To address these problems, we propose new adaptive multi-level exclusive caching policies that can dynamically adjust replacement and placement decisions in response to changing access patterns. (1) First, to capture locality information in multi-level cache hierarchies, we propose a Reuse Distance based Adaptive Replacement Caching (ReDARC) algorithm that adopts reuse distance as the means of locality measure and adaptively balances between the Small Reuse Distance (SRD) set and Large Reuse Distance (LRD) set. (2) Second, to achieve exclusive caching and make global caching decisions, we propose an Adaptive Level-Aware Caching Algorithm (ALACA) that works collaboratively with ReDARC. The ALACA algorithm uses an adaptive probabilistic PUSH technique that allows lower caches to push blocks to higher caches and appropriately decide blocks' caching locations with the ReDARC algorithm. In this way, we achieve multi-level exclusive caching with significant cache performance improvement. Our trace-driven simulation experiments show that the policies we proposed achieve a reduction of the client average response time of 8 percent to 56 percent over other multi-level cache schemes. (C) 2017 Elsevier B.V. All rights reserved.

Improving Cache Management with Redundant RDDs Eviction in Spark

LCRC: A Dependency-Aware Cache Management Policy for Spark

An Improved Memory Cache Management Study Based on Spark

LCS: an Efficient Data Eviction Strategy for Spark.

Resource-Aware Cache Management for In-Memory Data Analytics Frameworks.

Efficient Cache Resource Aggregation Using Adaptive Multi-Level Exclusive Caching Policies

A Cache Sharing Mechanism Based on RD MA.

Reo: Enhancing Reliability and Efficiency of Object-based Flash Caching

Intermediate Data Caching Optimization for Multi-Stage and Parallel Big Data Frameworks

Red: an Efficient Replacement Algorithm Based on REsident Distance for Exclusive Storage Caches

A New Scheme for Cache Optimization Based on Cluster Computing Framework Spark

Efficient Shuffle Management for DAG Computing Frameworks Based on the FRQ Model

Intermediate data placement and cache replacement strategy under Spark platform

A Novel Cache Replacement Policy Via Dynamic Adaptive Insertion And Re-Reference Prediction

Adaptive memory reservation strategy for heavy workloads in the Spark environment

Performance Improvement of DAG-Aware Task Scheduling Algorithms with Efficient Cache Management in Spark

Accelerating Big Data Applications on Tiered Storage System with Various Eviction Policies.

Energy-Efficient Data Caching Framework for Spark in Hybrid DRAM/NVM Memory Architectures.

DCache: A Distributed Cache Mechanism for HDFS based on RDMA

Efficient shuffle management with SCache for DAG computing frameworks.

Lifetime-Based Memory Management for Distributed Data Processing Systems.