Abstract:Due to the high computational demands inherent in Deep Neural Network (DNN) executions, multi-tier environments have emerged as preferred platforms for DNN inference tasks. Previous research on partitioning strategies for DNN models typically involved leveraging all layers of the DNN to identify optimal splits aimed at reducing latency or cost. However, due to their computational complexity, these approaches face scalability issues, particularly with models containing hundreds of layers. The novelty of our work lies in uniquely identifying specific split points within various DNN models that consistently lead to efficient latency or cost partitioning. Under the assumption that per unit computing cost decreases in higher tiers and that bandwidth is not free, we show that only these specific split points need to be considered to optimize latency or cost. Importantly, these split points are independent of different infrastructure configurations and bandwidth variations. The key contribution of our work is the significant reduction in the computational complexity of DNN partitioning, making our strategy applicable to models with a large number of layers. Introducing DNNSplit, an adaptive strategy, enables dynamic split decisions in varying conditions with the least complexity. Evaluated across nine DNN models varying in size and architecture, DNNSplit exhibits exceptional effectiveness in optimizing latency and cost. Even for a more substantial model containing 517 layers, it identifies only 5 points as potential split points, thereby reducing the partitioning complexity by more than 100x. This makes DNNSplit especially advantageous for managing larger models. DNNSplit also demonstrates significant improvements for multi-tier deployments compared to single-tier execution, including up to 15x latency speedup, 20x cost reduction, and 5x throughput enhancement.

DEEPSPLIT: An Efficient Splitting Method for Neural Network Verification via Indirect Effect Analysis

Improving Neural Network Verification through Spurious Region Guided Refinement.

Bound Tightening using Rolling-Horizon Decomposition for Neural Network Verification

Expediting Neural Network Verification via Network Reduction

Efficient verification of neural networks based on neuron branching and LP abstraction

Tight Neural Network Verification via Semidefinite Relaxations and Linear Reformulations

Efficient Neural Network Verification via Layer-based Semidefinite Relaxations and Linear Cuts

DNNSplit: Latency and Cost-Efficient Split Point Identification for Multi-Tier DNN Partitioning

ZoPE: A Fast Optimizer for ReLU Networks with Low-Dimensional Inputs

Steepest Descent Neural Architecture Optimization: Escaping Local Optimum with Signed Neural Splitting

DelBugV: Delta-Debugging Neural Network Verifiers

SyReNN: A Tool for Analyzing Deep Neural Networks

Energy-Aware Neural Architecture Optimization with Fast Splitting Steepest Descent

Towards Optimal Branching of Linear and Semidefinite Relaxations for Neural Network Robustness Certification

SplitNets: Designing Neural Architectures for Efficient Distributed Computing on Head-Mounted Systems

Online Verification of Deep Neural Networks under Domain Shift or Network Updates

Incremental Verification of Neural Networks

I-SplitEE: Image classification in Split Computing DNNs with Early Exits

SplitPlace: AI Augmented Splitting and Placement of Large-Scale Neural Networks in Mobile Edge Environments

Shared Certificates for Neural Network Verification

Pruning and Slicing Neural Networks using Formal Verification