Abstract:The Transformer architecture is widely used in natural language processing. Despite its success, the design principle of the Transformer remains elusive. In this paper, we provide a novel perspective towards understanding the architecture: we show that the Transformer can be mathematically interpreted as a numerical Ordinary Differential Equation (ODE) solver for a convection-diffusion equation in a multi-particle dynamic system. In particular, how words in a sentence are abstracted into contexts by passing through the layers of the Transformer can be interpreted as approximating multiple particles' movement in the space using the Lie-Trotter splitting scheme and the Euler's method. Given this ODE's perspective, the rich literature of numerical analysis can be brought to guide us in designing effective structures beyond the Transformer. As an example, we propose to replace the Lie-Trotter splitting scheme by the Strang-Marchuk splitting scheme, a scheme that is more commonly used and with much lower local truncation errors. The Strang-Marchuk splitting scheme suggests that the self-attention and position-wise feed-forward network (FFN) sub-layers should not be treated equally. Instead, in each layer, two position-wise FFN sub-layers should be used, and the self-attention sub-layer is placed in between. This leads to a brand new architecture. Such an FFN-attention-FFN layer is "Macaron-like", and thus we call the network with this new architecture the Macaron Net. Through extensive experiments, we show that the Macaron Net is superior to the Transformer on both supervised and unsupervised learning tasks. The reproducible codes and pretrained models can be found at https://github.com/zhuohan123/macaron-net

MS-Transformer: Introduce multiple structural priors into a unified transformer for encoding sentences

Multiple Structural Priors Guided Self Attention Network for Language Understanding

Modeling Graph Structure in Transformer for Better AMR-to-Text Generation.

Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddings

Hierarchical Topic-Aware Contextualized Transformers

Constituent Attention for Vision Transformers

Bidirectional Transformer with Absolute-Position Aware Relative Position Encoding for Encoding Sentences

Understanding and Improving Transformer from a Multi-Particle Dynamic System Point of View.

R-Transformer: Recurrent Neural Network Enhanced Transformer

Improving Transformers with Dynamically Composable Multi-Head Attention

Transformer-Dw: A Transformer Network With Dynamic And Weighted Head

Improved Transformer with Multi-Head Dense Collaboration

Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning

Chunk, Align, Select: A Simple Long-sequence Processing Method for Transformers

Learning Multiscale Transformer Models for Sequence Generation.

sTransformer: A Modular Approach for Extracting Inter-Sequential and Temporal Information for Time-Series Forecasting

Transformers are Universal In-context Learners

Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer

Multi-branch Attentive Transformer

Syntax-based Transformer for Neural Machine Translation

Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View