Abstract:ProKube is a proactive container scaling and migration orchestrator to dynamically adjust the resources and container locations with a fair balance between cost and delay for ML inferences in the inherent heterogeneous, resource‐constrained, and distributed edge environments. Deep neural network (DNN) and machine learning (ML) models/ inferences produce highly accurate results demanding enormous computational resources. The limited capacity of end‐user smart gadgets drives companies to exploit computational resources in an edge‐to‐cloud continuum and host applications at user‐facing locations with users requiring fast responses. Kubernetes hosted inferences with poor resource request estimation results in service level agreement (SLA) violation in terms of latency and below par performance with higher end‐to‐end (E2E) delays. Lifetime static resource provisioning either hurts user experience for under‐resource provisioning or incurs cost with over‐provisioning. Dynamic scaling offers to remedy delay by upscaling leading to additional cost whereas a simple migration to another location offering latency in SLA bounds can reduce delay and minimize cost. To address this cost and delay challenges for ML inferences in the inherent heterogeneous, resource‐constrained, and distributed edge environment, we propose ProKube, which is a proactive container scaling and migration orchestrator to dynamically adjust the resources and container locations with a fair balance between cost and delay. ProKube is developed in conjunction with Google Kubernetes Engine (GKE) enabling cross‐cluster migration and/ or dynamic scaling. It further supports the regular addition of freshly collected logs into scheduling decisions to handle unpredictable network behavior. Experiments conducted in heterogeneous edge settings show the efficacy of ProKube to its counterparts cost greedy (CG), latency greedy (LG), and GeKube (GK). ProKube offers 68%, 7%, and 64% SLA violation reduction to CG, LG, and GK, respectively, and it improves cost by 4.77 cores to LG and offers more cost of 3.94 to CG and GK.

Kub: Enabling Elastic HPC Workloads on Containerized Environments

Dynamically Adjusting Scale of a Kubernetes Cluster under QoS Guarantee

Zeus: Improving Resource Efficiency Via Workload Colocation for Massive Kubernetes Clusters

Fine-Grained Scheduling for Containerized HPC Workloads in Kubernetes Clusters

Container orchestration on HPC systems through Kubernetes

Towards Standard Kubernetes Scheduling Interfaces for Converged Computing

HCA Operator: A Hybrid Cloud Auto-scaling Tooling for Microservice Workloads.

ProKube: Proactive Kubernetes Orchestrator for Inference in Heterogeneous Edge Computing

HPC Alongside User-space Kubernetes

Portability and Scalability Evaluation of Large-Scale Statistical Modeling and Prediction Software through HPC-Ready Containers

Running Cloud-native Workloads on HPC with High-Performance Kubernetes

KubeGPU: efficient sharing and isolation mechanisms for GPU resource management in container cloud

A Bi-Metric Autoscaling Approach for N-Tier Web Applications on Kubernetes

On the Optimization of Kubernetes toward the Enhancement of Cloud Computing

A Cost-Efficient Container Orchestration Strategy in Kubernetes-Based Cloud Computing Infrastructures with Heterogeneous Resources

Hybrid Workload Scheduling on HPC Systems

A HPC Co-Scheduler with Reinforcement Learning

A Case For Intra-rack Resource Disaggregation in HPC

Containers Orchestration with Cost-Efficient Autoscaling in Cloud Computing Environments

On the Convergence of Malleability and the HPC PowerStack: Exploiting Dynamism in Over-Provisioned and Power-Constrained HPC Systems

CarbonScaler: Leveraging Cloud Workload Elasticity for Optimizing Carbon-Efficiency