Abstract:Large-scale Transformer models bring significant improvements for various downstream vision language tasks with a unified architecture. The performance improvements come with increasing model size, resulting in slow inference speed and increased cost for severing. While some certain predictions benefit from the full complexity of the large-scale model, not all of inputs need the same amount of computation to conduct, potentially leading to computation resource waste. To handle this challenge, early exiting is proposed to adaptively allocate computational power in term of input complexity to improve inference efficiency. The existing early exiting strategies usually adopt output confidence based on intermediate layers as a proxy of input complexity to incur the decision of skipping following layers. However, such strategies cannot apply to encoder in the widely-used unified architecture with both encoder and decoder due to difficulty of output confidence estimation in the encoder. It is suboptimal in term of saving computation power to ignore the early exiting in encoder component. To handle this challenge, we propose a novel early exiting strategy for unified visual language models, which allows dynamically skip the layers in encoder and decoder simultaneously in term of input layer-wise similarities with multiple times of early exiting, namely \textbf{MuE}. By decomposing the image and text modalities in the encoder, MuE is flexible and can skip different layers in term of modalities, advancing the inference efficiency while minimizing performance drop. Experiments on the SNLI-VE and MS COCO datasets show that the proposed approach MuE can reduce expected inference time by up to 50\% and 40\% while maintaining 99\% and 96\% performance respectively.

LECO: Improving Early Exiting Via Learned Exits and Comparison-based Exiting Mechanism.

SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for Accelerating BERT Inference.

Elastic DNN Inference with Unpredictable Exit in Edge Computing

Unlocking the Non-deterministic Computing Power with Memory-Elastic Multi-Exit Neural Networks

Learning to Weight Samples for Dynamic Early-Exiting Networks.

DE$^3$-BERT: Distance-Enhanced Early Exiting for BERT based on Prototypical Networks

DynExit: A Dynamic Early-Exit Strategy for Deep Residual Networks

ConsistentEE: A Consistent and Hardness-Guided Early Exiting Method for Accelerating Language Models Inference

To Exit or Not to Exit: Cost-Effective Early-Exit Architecture Based on Markov Decision Process

Joint or Disjoint: Mixing Training Regimes for Early-Exit Models

Dynamic Perceiver for Efficient Visual Recognition

Accelerating BERT Inference for Sequence Labeling Via Early-Exit.

EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism

You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Model

F-PABEE: Flexible-patience-based Early Exiting for Single-label and Multi-label text Classification Tasks

A Simple Hash-Based Early Exiting Approach for Language Understanding and Generation

RAEE: A Robust Retrieval-Augmented Early Exiting Framework for Efficient Inference

Early Exiting with Ensemble Internal Classifiers.

A Global Past-Future Early Exit Method for Accelerating Inference of Pre-trained Language Models.

EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models

DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Models