Abstract:Although High Performance Computing (HPC) users understand basic resource requirements such as the number of CPUs and memory limits, internal infrastructural utilization data is exclusively leveraged by cluster operators, who use it to configure batch schedulers. This task is challenging and increasingly complex due to ever larger cluster scales and heterogeneity of modern scientific workflows. As a result, HPC systems achieve low utilization with long job completion times (makespans). To tackle these challenges, we propose a co-scheduling algorithm based on an adaptive reinforcement learning algorithm, where application profiling is combined with cluster monitoring. The resulting cluster scheduler matches resource utilization to application performance in a fine-grained manner (i.e., operating system level). As opposed to nominal allocations, we apply decision trees to model applications' actual resource usage, which are used to estimate how much resource capacity from one allocation can be co-allocated to additional applications. Our algorithm learns from incorrect co-scheduling decisions and adapts from changing environment conditions, and evaluates when such changes cause resource contention that impacts quality of service metrics such as jobs slowdowns. We integrate our algorithm in an HPC resource manager that combines Slurm and Mesos for job scheduling and co-allocation, respectively. Our experimental evaluation performed in a dedicated cluster executing a mix of four real different scientific workflows demonstrates improvements on cluster utilization of up to 51% even in high load scenarios, with 55% average queue makespan reductions under low loads.

PerQueue: Managing Complex and Dynamic Workflows

Exascale Workflow Applications and Middleware: An ExaWorks Retrospective

FireWorks: a dynamic workflow system designed for high‐throughput applications

Workflow environments for advanced cyberinfrastructure platforms

A Workflow Manager for Complex NLP and Content Curation Pipelines

Stochastic Workflow Authorizations With Queueing Constraints.

Complex Workflow Management and Integration of Distributed Computing Resources by Science Gateway Portal for Molecular Dynamics Simulations in Materials Science

HyperQueue: Efficient and ergonomic task graphs on HPC clusters

Common workflows for computing material properties using different quantum engines

Construction of a specialized integrated simulation platform for molecule screening based on scientific computing workflow engine

Employing artificial intelligence to steer exascale workflows with colmena

Interoperable Workflows by Exchanging Grid-Based Data between Quantum-Chemical Program Packages

Paving the Way to Hybrid Quantum-Classical Scientific Workflows

An object-oriented framework to enable workflow evolution across materials acceleration platforms

A GPU-accelerated Molecular Docking Workflow with Kubernetes and Apache Airflow

An Autonomic Performance-Aware Workflow Job Management for Service-Oriented Computing

An Automatized Simulation Workflow for Powder Pressing Simulations Using SimStack

Machine‐learning assisted scheduling optimization and its application in quantum chemical calculations

CMS Workflow Execution using Intelligent Job Scheduling and Data Access Strategies

A HPC Co-Scheduler with Reinforcement Learning

AlabOS: A Python-based Reconfigurable Workflow Management Framework for Autonomous Laboratories