Abstract:Recent studies have shown that sequence-to-sequence (seq2seq) models struggle with compositional generalization (CG), i.e., the ability to systematically generalize to unseen compositions of seen components. There is mounting evidence that one of the reasons hindering CG is the representation of the encoder uppermost layer is entangled, i.e., the syntactic and semantic representations of sequences are entangled. However, we consider that the previously identified representation entanglement problem is not comprehensive enough. Additionally, we hypothesize that the source keys and values representations passing into different decoder layers are also entangled. Starting from this intuition, we propose \textsc{CompoSition} (\textbf{Compo}se \textbf{S}yntactic and Semant\textbf{i}c Representa\textbf{tion}s), an extension to seq2seq models which learns to compose representations of different encoder layers dynamically for different tasks, since recent studies reveal that the bottom layers of the Transformer encoder contain more syntactic information and the top ones contain more semantic information. Specifically, we introduce a \textit{composed layer} between the encoder and decoder to compose different encoder layers' representations to generate specific keys and values passing into different decoder layers. \textsc{CompoSition} achieves competitive results on two comprehensive and realistic benchmarks, which empirically demonstrates the effectiveness of our proposal. Codes are available at~\url{<a class="link-external link-https" href="https://github.com/thinkaboutzero/COMPOSITION" rel="external noopener nofollow">this https URL</a>}.

On Compositional Generalization of Neural Machine Translation

On compositional generalization of transformer-based neural machine translation

Categorizing Semantic Representations for Neural Machine Translation

Evaluating Structural Generalization in Neural Machine Translation

On Evaluating Multilingual Compositional Generalization with Translated Datasets

The paradox of the compositionality of natural language: a neural machine translation case study

Consistency Regularization Training for Compositional Generalization.

Compositional Generalization by Learning Analytical Expressions.

Benchmarking and Improving Compositional Generalization of Multi-aspect Controllable Text Generation

Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?

Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalization

A General Theory for Compositional Generalization

Learning to Generalize to More: Continuous Semantic Augmentation for Neural Machine Translation

Towards Understanding the Relationship between In-context Learning and Compositional Generalization

Go From the General to the Particular: Multi-Domain Translation with Domain Transformation Networks

Neuron Interaction Based Representation Composition for Neural Machine Translation

Compositional Generalization with Grounded Language Models

Data Factors for Better Compositional Generalization

A Study of Compositional Generalization in Neural Models

Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations

Compositional Neural Machine Translation by Removing the Lexicon from Syntax