Abstract:Mixture-of-Experts (MoE) based sparse architectures can significantly increase model capacity with sublinear computational overhead, which are hence widely used in massively multilingual neural machine translation (MNMT). However, they are prone to overfitting on low-resource language translation. In this paper, we propose a modularized MNMT framework that is able to flexibly assemble dense and MoE-based sparse modules to achieve the best of both worlds. The training strategy of the modularized MNMT framework consists of three stages: (1) Pre-training basic MNMT models with different training objectives or model structures, (2) Initializing modules of the framework with pre-trained couterparts (e.g., encoder, decoder and embedding layers) from the basic models and (3) Fine-tuning the modularized MNMT framework to fit modules from different models together. We pre-train three basic MNMT models from scratch: a dense model, an MoE-based sparse model and a new MoE model, termed as MoE-LGR that explores multiple Language-Group-specifc Routers to incorporate language group knowledge into MNMT. The strengths of these pre-trained models are either on low-resource language translation, high-resource language translation or zero-shot translation. Our modularized MNMT framework attempts to incorporate these advantages into a single model with reasonable initialization and fine-tuning. Experiments on widely-used benchmark datasets demonstrate that the proposed modularized MNMT framwork substantially outperforms both MoE and dense models on high- and low-resource language translation as well as zero-shot translation. Our framework facilitates the combination of different methods with their own strengths and recycling off-the-shelf models for multilingual neural machine translation. Codes are available at https://github.com/lishangjie1/MMNMT.

MOLE: MOdular Learning FramEwork via Mutual Information Maximization

CMNN: Coupled Modular Neural Network.

MMNMT: Modularizing Multilingual Neural Machine Translation with Flexibly Assembled MoE and Dense Blocks

Multitask Learning With Enhanced Modules.

Multimodal Moore-Penrose Inverse-Based Recomputation Framework for Big Data Analysis

LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing

MoleMCL: a multi-level contrastive learning framework for molecular pre-training

KNET: A General Framework for Learning Word Embedding using Morphological Knowledge

Learning Personalized Modular Network Guided By Structured Knowledge

MODL: Multilearner Online Deep Learning

Modularized Morphing of Neural Networks

MINER: Mining the Underlying Pattern of Modality-Specific Neurons in Multimodal Large Language Models

Robust Online Multi-Task Learning with Correlative and Personalized Structures

A Multiobjective Learning and Ensembling Approach to High-Performance Speech Enhancement with Compact Neural Network Architectures

MLAAN: Scaling Supervised Local Learning with Multilaminar Leap Augmented Auxiliary Network

Multiobjective bilevel programming model for multilayer perceptron neural networks

AMOSL: Adaptive Modality-wise Structure Learning in Multi-view Graph Neural Networks For Enhanced Unified Representation

MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Model

Multi-Task Structural Learning using Local Task Similarity induced Neuron Creation and Removal

Self-Supervised Interpretable End-to-End Learning via Latent Functional Modularity

Learning Task-Specific Morphological Representation for Pyramidal Cells via Mutual Information Minimization