Abstract:The visual medium (images and videos) naturally contains a large amount of information redundancy, thereby providing a great opportunity for leveraging efficiency in processing. While Vision Transformer (ViT) based models scale effectively to large data regimes, they fail to capitalize on this inherent redundancy, leading to higher computational costs. Mixture of Experts (MoE) networks demonstrate scalability while maintaining same inference-time costs, but they come with a larger parameter footprint. We present Mixture of Nested Experts (MoNE), which utilizes a nested structure for experts, wherein individual experts fall on an increasing compute-accuracy curve. Given a compute budget, MoNE learns to dynamically choose tokens in a priority order, and thus redundant tokens are processed through cheaper nested experts. Using this framework, we achieve equivalent performance as the baseline models, while reducing inference time compute by over two-fold. We validate our approach on standard image and video datasets - ImageNet-21K, Kinetics400, and Something-Something-v2. We further highlight MoNE$'$s adaptability by showcasing its ability to maintain strong performance across different inference-time compute budgets on videos, using only a single trained model.

What problem does this paper attempt to address?

The paper primarily addresses the computational efficiency issues of visual Transformer models (such as Vision Transformer (ViT) and Video Vision Transformer (ViViT)) when processing images and videos by proposing a new solution. Specifically, the paper aims to solve the following problems: 1. **Computational Redundancy**: Due to the significant information redundancy in visual data (images and videos), traditional ViT and ViViT models fail to fully leverage this characteristic to improve computational efficiency, resulting in unnecessary computational costs when processing these models. 2. **Mixture of Experts Network Issues**: Although Mixture of Experts (MoE) networks can effectively scale the model size while maintaining the same inference time cost, they usually increase the number of parameters, thereby increasing storage requirements. Additionally, existing works rely on expert modules with the same number of parameters and computational load, which limits their ability to reduce computational costs without completely skipping certain tokens. To address the above issues, the paper proposes the **Mixture of Nested Experts (MoNE)** framework, whose main contributions include: 1. **Dynamic Resource Allocation**: The MoNE framework can dynamically prioritize tokens based on the computational budget and process redundant tokens through cheaper nested expert modules, thereby achieving dynamic allocation of computational resources. 2. **Significant Computational Efficiency Improvement**: With a fixed number of parameters, MoNE can learn much lower FLOPs (approximately 2.3 times lower) than baseline models while still matching the performance of the baseline models. 3. **High Adaptability**: Experimental validation shows that MoNE is not only applicable to image transformers but also to video transformers, and visualization results indicate that tokens processed by larger expert modules are well associated with regions of interest. In short, the paper addresses the computational inefficiency issues in existing visual Transformer models by introducing the MoNE framework, achieving the goal of significantly reducing computational costs while ensuring performance.

Mixture of Nested Experts: Adaptive Processing of Visual Tokens

Video Relationship Detection Using Mixture of Experts

Mixture of Tokens: Continuous MoE through Cross-Example Aggregation

Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization

Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (moe) Inference

From Sparse to Soft Mixtures of Experts

Residual Mixture of Experts

Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception

Mobile V-MoEs: Scaling Down Vision Transformers via Sparse Mixture-of-Experts

Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models

Experts Weights Averaging: A New General Training Scheme for Vision Transformers

ViMoE: An Empirical Study of Designing Vision Mixture-of-Experts

MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts

MoE Jetpack: From Dense Checkpoints to Adaptive Mixture of Experts for Vision Tasks

Merging Experts into One: Improving Computational Efficiency of Mixture of Experts

MoNDE: Mixture of Near-Data Experts for Large-Scale Sparse Models

FasterMoE

AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference

Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning

Cross-token Modeling with Conditional Computation

Mixture of Parrots: Experts improve memorization more than reasoning