Abstract:Deep neural network (DNN) foundation models are currently exhibiting high prediction accuracy and strong adaptability to broad tasks with remarkably large model scales. They are increasingly becoming the backend support of DNN-driven real-time online services, e.g., Siri and Instagram. Such services require low-latency and cost-efficiency for quality-of-service and commercial competitiveness. When deployed in a cloud environment, these services call for an appropriate selection of cloud configurations (i.e., specific types of VM instances), as well as a considerate device placement plan that places the operations of the model to multiple GPUs via model parallelism for cost-efficiency. Currently, the deployment mainly relies on service providers’ manual efforts, which is not only onerous but also far from satisfactory oftentimes due to the huge joint search space of cloud configurations and device placement plans (for a same service, a poor deployment can incur significantly more costs by tens of times). In this paper, we attempt to efficiently automate the cloud deployment for real-time foundation model inference with minimum costs under the constraint of acceptably low latency. This attempt is enabled by 1) jointly leveraging the Bayesian Optimization and Deep Reinforcement Learning to adaptively unearth the (nearly) optimal cloud configuration and device placement with limited search time, and 2) enhancing the cost-efficiency of the deployment based on the probing-informed block multiplexing mechanism and Tensor Algebra SuperOptimizer. We implement a prototype system based on TensorFlow, conduct extensive experiments on top of Microsoft Azure, and demonstrate the generality and scalability of our solution. Results show that for lightweight DNN models and foundation models, our solution essentially saves inference costs by up to 15% and 47% with 57% and 38% lower search overheads respectively, compared with non-trivial baselines.

Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instances

Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs

PREMA: A Predictive Multi-task Scheduling Algorithm For Preemptible Neural Processing Units

Cost-Driven Off-Loading for DNN-Based Applications Over Cloud, Edge, and End Devices

Adaptive Partitioning and Efficient Scheduling for Distributed DNN Training in Heterogeneous IoT Environment

Pre-DNNOff: On-Demand DNN Model Offloading Method for Mobile Edge Computing

Joint DNN Partition and Resource Allocation Optimization for Energy-Constrained Hierarchical Edge-Cloud Systems

SpotServe: Serving Generative Large Language Models on Preemptible Instances

DDPQN: An Efficient DNN Offloading Strategy in Local-Edge-Cloud Collaborative Environments

DeepVM: Integrating Spot and On-Demand VMs for Cost-Efficient Deep Learning Clusters in the Cloud

Automating Cloud Deployment for Real-Time Online Foundation Model Inference

Rationing Bandwidth Resources for Mitigating Network Resource Contention in Distributed DNN Training Clusters.

Automating Cloud Deployment for Deep Learning Inference of Real-time Online Services

Preemptive Scheduling for Distributed Machine Learning Jobs in Edge-Cloud Networks

BFTrainer: Low-Cost Training of Neural Networks on Unfillable Supercomputer Nodes

HierTrain: Fast Hierarchical Edge AI Learning with Hybrid Parallelism in Mobile-Edge-Cloud Computing

Dynamic DNN Decomposition for Lossless Synergistic Inference

Learning to Dynamically Select Cost Optimal Schedulers in Cloud Computing Environments

A Novel Dynamic Pricing Approach for Preemptible Cloud Services

An Online Approach for DNN Model Caching and Processor Allocation in Edge Computing

Low-latency job scheduling with preemption for the development of deep learning