Abstract:Recent studies have shown that sequence-to-sequence (seq2seq) models struggle with compositional generalization (CG), i.e., the ability to systematically generalize to unseen compositions of seen components. There is mounting evidence that one of the reasons hindering CG is the representation of the encoder uppermost layer is entangled, i.e., the syntactic and semantic representations of sequences are entangled. However, we consider that the previously identified representation entanglement problem is not comprehensive enough. Additionally, we hypothesize that the source keys and values representations passing into different decoder layers are also entangled. Starting from this intuition, we propose \textsc{CompoSition} (\textbf{Compo}se \textbf{S}yntactic and Semant\textbf{i}c Representa\textbf{tion}s), an extension to seq2seq models which learns to compose representations of different encoder layers dynamically for different tasks, since recent studies reveal that the bottom layers of the Transformer encoder contain more syntactic information and the top ones contain more semantic information. Specifically, we introduce a \textit{composed layer} between the encoder and decoder to compose different encoder layers' representations to generate specific keys and values passing into different decoder layers. \textsc{CompoSition} achieves competitive results on two comprehensive and realistic benchmarks, which empirically demonstrates the effectiveness of our proposal. Codes are available at~\url{<a class="link-external link-https" href="https://github.com/thinkaboutzero/COMPOSITION" rel="external noopener nofollow">this https URL</a>}.

On Provable Length and Compositional Generalization

What Algorithms can Transformers Learn? A Study in Length Generalization

A Formal Framework for Understanding Length Generalization in Transformers

Universal Length Generalization with Turing Programs

Out-of-distribution generalization via composition: a lens through induction heads in Transformers

A Theory for Length Generalization in Learning to Reason

Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations

Transformers Can Achieve Length Generalization But Not Robustly

Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks

Provable Compositional Generalization for Object-Centric Learning

Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalization

Unobserved Local Structures Make Compositional Generalization Hard

Towards Understanding the Relationship between In-context Learning and Compositional Generalization

Consistency of Compositional Generalization across Multiple Levels

Length Generalization of Causal Transformers without Position Encoding

Knowing When to Stop: Evaluation and Verification of Conformity to Output-size Specifications

Sequence Length Independent Norm-Based Generalization Bounds for Transformers

Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?

Consistency Regularization Training for Compositional Generalization.

Minimum Description Length and Generalization Guarantees for Representation Learning

Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionality