Abstract:Knowledge distillation (KD) is a promising technique for model compression in neural machine translation. However, where the knowledge hides in KD is still not clear, which may hinder the development of KD. In this work, we first unravel this mystery from an empirical perspective and show that the knowledge comes from the top-1 predictions of teachers, which also helps us build a potential connection between word- and sequence-level KD. Further, we point out two inherent issues in vanilla word-level KD based on this finding. Firstly, the current objective of KD spreads its focus to whole distributions to learn the knowledge, yet lacks special treatment on the most crucial top-1 information. Secondly, the knowledge is largely covered by the golden information due to the fact that most top-1 predictions of teachers overlap with ground-truth tokens, which further restricts the potential of KD. To address these issues, we propose a novel method named \textbf{T}op-1 \textbf{I}nformation \textbf{E}nhanced \textbf{K}nowledge \textbf{D}istillation (TIE-KD). Specifically, we design a hierarchical ranking loss to enforce the learning of the top-1 information from the teacher. Additionally, we develop an iterative KD procedure to infuse more additional knowledge by distilling on the data without ground-truth targets. Experiments on WMT'14 English-German, WMT'14 English-French and WMT'16 English-Romanian demonstrate that our method can respectively boost Transformer$_{base}$ students by +1.04, +0.60 and +1.11 BLEU scores and significantly outperform the vanilla word-level KD baseline. Besides, our method shows higher generalizability on different teacher-student capacity gaps than existing KD techniques.

Learning Light-Weight Translation Models from Deep Transformer

Learning Deep Transformer Models For Machine Translation

Joint Structured Pruning and Dense Knowledge Distillation for Efficient Transformer Model Compression

Shallow-to-Deep Training for Neural Machine Translation

Deep Transformer Modeling Via Grouping Skip Connection for Neural Machine Translation

Very Deep Transformers for Neural Machine Translation

Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layers

Deep Transformers with Latent Depth

A Cooperative Lightweight Translation Algorithm Combined with Sparse-ReLU

Compressed-Transformer

PET: Parameter-efficient Knowledge Distillation on Transformer

What Works and Doesn’t Work, A Deep Decoder for Neural Machine Translation

Deep-to-Bottom Weights Decay: A Systemic Knowledge Review Learning Technique for Transformer Layers in Knowledge Distillation

Exploring All-In-One Knowledge Distillation Framework for Neural Machine Translation

An Empirical Study of Leveraging Knowledge Distillation for Compressing Multilingual Neural Machine Translation Models

Knowledge Translation: A New Pathway for Model Compression

Weight Distillation: Transferring the Knowledge in Neural Network Parameters

Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation

Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation

Multi-Teacher Distillation With Single Model for Neural Machine Translation

Transformer on a Diet