Abstract:With the rapid progress in natural language processing many models grow larger. However deploying these models in small devices such as mobile devices requires smaller models. Thus techniques of model compression should be used. We consider the problem of compressing word embedding of these models. Existing methods usually treat all the words as the same. These words often have the same dimension and each word cost the same memory. During our work,first we show the feasibility of further compressing word embedding by considering different impact of each word. The idea comes from the famous Zipf’s law which says few words cover most of the document. We divide the words into groups based on word frequency and encode each group differently. Second we propose a joint goal for automatically choosing the split points between different word groups and determining the size of each group. The joint goal between task performance and model compression ratio solves the problem of how to balance between both goals. By combining both goals into to an explicit formula, it’s easy to find a good solution using an end to end training strategy.It turns to an architecture search problem and we solve it using AutoML methods. The experiment show that the word embedding layer could be further compressed to 22.6% of the baseline. We also get hints on how to choose number of word groups and to balance between the model performance and compression ratio on the experimental result.

Joint Decoding of Tree Transduction Models for Sentence Compression

Joint Decoding with Multiple Translation Models.

Joint Parsing and Translation

Joint Structured Pruning and Dense Knowledge Distillation for Efficient Transformer Model Compression

Tree-based Convolution for Sentence Modeling.

On Tree-Based Neural Sentence Modeling

Discriminative Neural Sentence Modeling by Tree-Based Convolution

High-order Joint Constituency and Dependency Parsing

Tree-Structured Neural Machine for Linguistics-Aware Sentence Generation

Left-to-Right Tree-to-String Decoding with Prediction

A lightweight Chinese semantic dependency parsing model based on sentence compression

Image-To-Tree: A Tree-Structured Decoder For Image Captioning

Improving Tree-to-Tree Translation with Packed Forests.

TDv2: A Novel Tree-Structured Decoder for Offline Mathematical Expression Recognition

Joint Training and Decoding Using Virtual Nodes for Cascaded Segmentation and Tagging Tasks.

Modelling Sentence Pairs with Tree-structured Attentive Encoder

Sub-Sentence Division for Tree-Based Machine Translation.

Dependency-based Convolutional Neural Networks for Sentence Embedding

OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure

A Tree-Structured Decoder for Image-to-Markup Generation.

Joint Goal for Word Embedding Compression Based on Word Frequency