Abstract:In this paper, a deep mixture of diverse experts algorithm is developed to achieve more efficient learning of a huge (mixture) network for large-scale visual recognition application. First, a two-layer ontology is constructed to assign large numbers of atomic object classes into a set of task groups according to the similarities of their learning complexities, where certain degrees of inter-group task overlapping are allowed to enable sufficient inter-group message passing. Second, one particular base deep CNNs with M+1 outputs is learned for each task group to recognize its M atomic object classes and identify one special class of "not-in-group", where the network structure (numbers of layers and units in each layer) of the well-designed deep CNNs (such as AlexNet, VGG, GoogleNet, ResNet) is directly used to configure such base deep CNNs. For enhancing the separability of the atomic object classes in the same task group, two approaches are developed to learn more discriminative base deep CNNs: (a) our deep multi-task learning algorithm that can effectively exploit the inter-class visual similarities; (b) our two-layer network cascade approach that can improve the accuracy rates for the hard object classes at certain degrees while effectively maintaining the high accuracy rates for the easy ones. Finally, all these complementary base deep CNNs with diverse but overlapped outputs are seamlessly combined to generate a mixture network with larger outputs for recognizing tens of thousands of atomic object classes. Our experimental results have demonstrated that our deep mixture of diverse experts algorithm can achieve very competitive results on large-scale visual recognition.

Adaptive Mixtures of Local Experts

A Modified Mixtures of Experts Architecture for Classification with Diverse Features

Customizing Student Networks From Heterogeneous Teachers Via Adaptive Knowledge Amalgamation

Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners

Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM

Specialization in Hierarchical Learning Systems

Learning a Mixture of Granularity-Specific Experts for Fine-Grained Categorization

TLMIX: Twin Leader Mixing Network for Cooperative Multi-Agent Reinforcement Learning.

Deep Mixture of Diverse Experts for Large-Scale Visual Recognition.

Unsupervised Multi-channel Separation and Adaptation

Hierarchical Mixture of Experts: Generalizable Learning for High-Level Synthesis

An Incremental Network with Local Experts Ensemble.

Mixture-of-Variational-Experts for Continual Learning

Hard Mixtures of Experts for Large Scale Weakly Supervised Vision

Enhancing Code-Switching Speech Recognition with LID-Based Collaborative Mixture of Experts Model

Distributed Learning of Mixtures of Experts

Learning More Generalized Experts by Merging Experts in Mixture-of-Experts

Learning Mixtures of Experts with EM

Co-Supervised Learning: Improving Weak-to-Strong Generalization with Hierarchical Mixture of Experts

Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning

Adaptive Compositional Continual Meta-Learning.