Abstract:Recent years have witnessed significant achievements in deep learning (DL) technologies. In the meantime, an increasing number of online service operators take advantage of deep learning to provide intelligent and personalized services. Although significant efforts have been put into optimizing the inference efficiency, our investigation shows that for many DL models that process data-intensive requests, the network I/O subsystem also plays an essential role in determining responsiveness. Furthermore, under the latency constraint, uncontrolled network flow processing will impact request batching. Based on the above observation, this paper proposes CoFB, an inference service system that optimizes performance in a holistic way. CoFB improves the load imbalance in the network I/O subsystem with a lightweight flow scheduling scheme that collaborates the network interface card with a dispatcher thread. In addition, CoFB introduces a request reordering and batching policy and an interference-aware concurrent batch throttling strategy for enforcing inference concerning the deadline. We evaluate CoFB on four DL inference services and compare it to two state-of-the-art inference systems: NVIDIA Triton and DVABatch. Experimental results show that CoFB outperforms these two baselines by serving up to 2.69×documentclass[12pt]{minimal}usepackage{amsmath}usepackage{wasysym}usepackage{amsfonts}usepackage{amssymb}usepackage{amsbsy}usepackage{mathrsfs}usepackage{upgreek}setlength{oddsidemargin}{-69pt}egin{document}$$ imes$$end{document} and 1.96×documentclass[12pt]{minimal}usepackage{amsmath}usepackage{wasysym}usepackage{amsfonts}usepackage{amssymb}usepackage{amsbsy}usepackage{mathrsfs}usepackage{upgreek}setlength{oddsidemargin}{-69pt}egin{document}$$ imes$$end{document} higher load under preset tail latency objectives, respectively.

StreamBox: A Lightweight GPU SandBox for Serverless Inference Workflow.

Design and implementation of efficient distributed deep learning model inference architecture on serverless computation

FaST-GShare: Enabling Efficient Spatio-Temporal GPU Sharing in Serverless Computing for Deep Learning Inference

FSD-Inference: Fully Serverless Distributed Inference with Scalable Cloud Communication

GPU-enabled Function-as-a-Service for Machine Learning Inference

CoFB: latency-constrained co-scheduling of flows and batches for deep learning inference service on the CPU–GPU system

Efficient CUDA stream management for multi-DNN real-time inference on embedded GPUs

Dynamic Space-Time Scheduling for GPU Inference

SplitStream: Distributed and workload-adaptive video analytics at the edge

ParvaGPU: Efficient Spatial GPU Sharing for Large-Scale DNN Inference in Cloud Environments

FaaSwap: SLO-Aware, GPU-Efficient Serverless Inference via Model Swapping

PARIS and ELSA: An Elastic Scheduling Algorithm for Reconfigurable Multi-GPU Inference Servers

Responsive ML inference in multi-tenanted environments using AQUA

SPRIGHT: High-Performance eBPF-Based Event-Driven, Shared-Memory Processing for Serverless Computing

LMStream: When Distributed Micro-Batch Stream Processing Systems Meet GPU

Efficient Architecture Paradigm for Deep Learning Inference As a Service.

A holistic approach to build real-time stream processing system with GPU

ESG: Pipeline-Conscious Efficient Scheduling of DNN Workflows on Serverless Platforms with Shareable GPUs

In-Storage Domain-Specific Acceleration for Serverless Computing

Kernel-as-a-Service: A Serverless Interface to GPUs

A Streaming Cloud Platform for Real-Time Video Processing on Embedded Devices