Abstract:The rise of the Internet of Things (IoT) and Industry 2.0 has spurred a growing need for extensive data computing, and Spark emerged as a promising Big Data platform, attributed to its distributed in-memory computing capabilities. However, practical heavy workloads often lead to memory bottleneck issues in the Spark platform. This results in resilient distributed datasets (RDD) eviction and, in extreme cases, violent memory contentions, causing a significant degradation in Spark computational efficiency. To tackle this issue, we propose an adaptive memory reservation (AMR) strategy in this article, specifically designed for heavy workloads in the Spark environment. Specifically, we model optimal task parallelism by minimizing the disparity between the number of tasks completed without blocking and the number completed in regular rounds. Optimal memory for task parallelism is determined to establish an efficient execution memory space for computational parallelism. Subsequently, through adaptive execution memory reservation and dynamic adjustments, such as compression or expansion based on task progress, the strategy ensures dynamic task parallelism in the Spark parallel computing process. Considering the cost of RDD cache location and real-time memory space usage, we select suitable storage locations for different RDD types to alleviate execution memory pressure. Finally, we conduct extensive laboratory experiments to validate the effectiveness of AMR. Results indicate that, compared to existing memory management solutions, AMR reduces the execution time by approximately 46.8%.

Dynamic Memory-Aware Scheduling in Spark Computing Environment

Design Of A Dynamic Memory Access Scheduler

Adaptive memory reservation strategy for heavy workloads in the Spark environment

MCS: Memory Constraint Strategy for Unified Memory Manager in Spark.

An adaptive memory tuning strategy with high performance for Spark.

Memory optimization of Spark parallel computing framework

DynIMS: A Dynamic Memory Controller for In-memory Storage on HPC Systems

An Adaptive Tuning Strategy on Spark Based on In-Memory Computation Characteristics

A New Scheme for Cache Optimization Based on Cluster Computing Framework Spark

An Improved Memory Cache Management Study Based on Spark

Share Memory Aware Scheduler

An Optimal Locality-Aware Task Scheduling Algorithm Based on Bipartite Graph Modelling for Spark Applications

An Energy-Aware Scheduling Algorithm for Big Data Applications in Spark

Design and Implementation of Scheduling Pool Scheduling Algorithm Based on Reuse of Jobs in Spark

Comparative Analysis of Energy-Efficient Scheduling Algorithms for Big Data Applications.

Performance Improvement of DAG-Aware Task Scheduling Algorithms with Efficient Cache Management in Spark

A Spark Scheduling Strategy for Heterogeneous Cluster

Efficient shuffle management with SCache for DAG computing frameworks.

In-memory big data analytics under space constraints using dynamic programming.

MespaConfig: Memory-Sparing Configuration Auto-Tuning for Co-Located In-Memory Cluster Computing Jobs

Memory Access Scheduling Based on Dynamic Multilevel Priority in Shared DRAM Systems