Abstract:Many resource management techniques for task scheduling, energy and carbon efficiency, and cost optimization in workflows rely on a-priori task runtime knowledge. Building runtime prediction models on historical data is often not feasible in practice as workflows, their input data, and the cluster infrastructure change. Online methods, on the other hand, which estimate task runtimes on specific machines while the workflow is running, have to cope with a lack of measurements during start-up. Frequently, scientific workflows are executed on heterogeneous infrastructures consisting of machines with different CPU, I/O, and memory configurations, further complicating predicting runtimes due to different task runtimes on different machine types. This paper presents Lotaru, a method for locally predicting the runtimes of scientific workflow tasks before they are executed on heterogeneous compute clusters. Crucially, our approach does not rely on historical data and copes with a lack of training data during the start-up. To this end, we use microbenchmarks, reduce the input data to quickly profile the workflow locally, and predict a task’s runtime with a Bayesian linear regression based on the gathered data points from the local workflow execution and the microbenchmarks. Due to its Bayesian approach, Lotaru provides uncertainty estimates that can be used for advanced scheduling methods on distributed cluster infrastructures. In our evaluation with five real-world scientific workflows, our method outperforms two state-of-the-art runtime prediction baselines and decreases the absolute prediction error by more than 12.5%. In a second set of experiments, the prediction performance of our method, using the predicted runtimes for state-of-the-art scheduling, carbon reduction, and cost prediction, enables results close to those achieved with perfect prior knowledge of runtimes.

Lotaru: Locally predicting workflow task runtimes for resource management on heterogeneous infrastructures

Ponder: Online Prediction of Task Memory Requirements for Scientific Workflows

Reshi: Recommending Resources for Scientific Workflow Tasks on Heterogeneous Infrastructures

KS+: Predicting Workflow Task Memory Usage Over Time

Cross-Domain Workloads Performance Prediction Via Runtime Metrics Transferring

Modular Resource Centric Learning for Workflow Performance Prediction

Workload Failure Prediction for Data Centers

Leveraging Reinforcement Learning for Task Resource Allocation in Scientific Workflows

Runtime Variation in Big Data Analytics

Sizey: Memory-Efficient Execution of Scientific Workflow Tasks

A mathematical programming approach for resource allocation of data analysis workflows on heterogeneous clusters

Optimizing Resource Allocation for Data-Parallel Jobs Via GCN-Based Prediction

The Case for Task Sampling based Learning for Cluster Job Scheduling

Modeling and predicting execution time of scientific workflows in the Grid using radial basis function neural network

Dynamic Job Shop Scheduling Based on Order Remaining Completion Time Prediction

TimeLink: enabling dynamic runtime prediction for Flink iterative jobs

Predicting batch queue job wait times for informed scheduling of urgent HPC workloads

Machine learning approaches to predict the execution time of the meteorological simulation software COSMO

Evaluating Distributed Execution of Workloads

Forecasting Workload in Cloud Computing: Towards Uncertainty-Aware Predictions and Transfer Learning

End-to-End Predictions-Based Resource Management Framework for Supercomputer Jobs