DAG-aware harmonizing job scheduling and data caching for disaggregated analytics frameworks
Yulai Tong,Jiazhen Liu,Hua Wang,Mingjian He,Ke Zhou,Rongfeng He,Qin Zhang,Cheng Wang
DOI: https://doi.org/10.1016/j.future.2024.03.005
IF: 7.307
2024-03-07
Future Generation Computer Systems
Abstract:Modern data analytics frameworks often integrate with external storage services, which can lead to storage bottlenecks. Existing caching and prefetching solutions utilize high-level information from data analytics frameworks to forecast future data accesses. They employ these predictions to prefetch data into the cache and manage the cache contents. However, this approach overlooks a fundamental opportunity: rather than caching data given a prediction of job execution, influencing the job execution order can enable more effective caching and prefetching. With this key insight, we introduce a novel system called Tripod , designed to synchronize job scheduling and data caching for analytics frameworks. Leveraging the higher-level information provided by analytics frameworks, Tripod explores the best-suited job execution order, guided by developed heuristics, to facilitate prefetching and caching. To fully exploit the potential of Tripod , we also introduce a novel caching strategy named CAP . This strategy not only acknowledges the job scheduling order but also offers fine-grained control over object prefetching and eviction. Our evaluation, conducted using standard analytic benchmarks (TPC-H and TPC-DS), demonstrates that Tripod achieves a speedup of up to 1.7x on state-of-the-art approaches. Moreover, when employing CAP to make caching decisions, the performance can further be improved (as much as 1.5x).
computer science, theory & methods