Abstract:State-of-the-art encoder-decoder models (e.g. for machine translation (MT) or automatic speech recognition (ASR)) are constructed and trained end-to-end as an atomic unit. No component of the model can be (re-)used without the others, making it impossible to share parts, e.g. a high resourced decoder, across tasks. We describe LegoNN, a procedure for building encoder-decoder architectures in a way so that its parts can be applied to other tasks without the need for any fine-tuning. To achieve this reusability, the interface between encoder and decoder modules is grounded to a sequence of marginal distributions over a pre-defined discrete vocabulary. We present two approaches for ingesting these marginals; one is differentiable, allowing the flow of gradients across the entire network, and the other is gradient-isolating. To enable the portability of decoder modules between MT tasks for different source languages and across other tasks like ASR, we introduce a modality agnostic encoder which consists of a length control mechanism to dynamically adapt encoders' output lengths in order to match the expected input length range of pre-trained decoders. We present several experiments to demonstrate the effectiveness of LegoNN models: a trained language generation LegoNN decoder module from German-English (De-En) MT task can be reused without any fine-tuning for the Europarl English ASR and the Romanian-English (Ro-En) MT tasks, matching or beating the performance of baseline. After fine-tuning, LegoNN models improve the Ro-En MT task by 1.5 BLEU points and achieve 12.5% relative WER reduction on the Europarl ASR task. To show how the approach generalizes, we compose a LegoNN ASR model from three modules -- each has been learned within different end-to-end trained models on three different datasets -- achieving an overall WER reduction of 19.5%.

LegoNet: Efficient Convolutional Neural Networks with Lego Filters

Layer-Wise Training Convolutional Neural Networks With Smaller Filters for Human Activity Recognition Using Wearable Sensors

Model LEGO: Creating Models Like Disassembling and Assembling Building Blocks

LegoDNN: Block-grained Scaling of Deep Neural Networks for Mobile Vision

Learning Stackable and Skippable LEGO Bricks for Efficient, Reconfigurable, and Variable-Resolution Diffusion Modeling

CNN LEGO: Disassembling and Assembling Convolutional Neural Network

Learning Versatile Filters for Efficient Convolutional Neural Networks.

Compression of Convolutional Neural Networks With Divergent Representation of Filters

Learning Versatile Convolution Filters for Efficient Visual Recognition

An Efficient Low-Complexity Convolutional Neural Network Filter

LegoFormer: Transformers for Block-by-Block Multi-view 3D Reconstruction

Towards Evolutionary Compression.

Improving Efficiency in Convolutional Neural Network with Multilinear Filters

Effnet: An Efficient Structure for Convolutional Neural Networks

LDCNet: A Lightweight Multi-Scale Convolutional Neural Network Using Local Dense Connectivity for Image Recognition

Layer-Wise Training To Create Efficient Convolutional Neural Networks

A Convolutional Neural Network-Based Low Complexity Filter

Learning Recursive Filters for Low-Level Vision Via a Hybrid Neural Network

LegoNN: Building Modular Encoder-Decoder Models

Computational optimization of convolutional neural networks using separated filters architecture

Auto-Balanced Filter Pruning for Efficient Convolutional Neural Networks