Abstract: Transformer architecture has become the fundamental element of the widespread natural language processing~(NLP) models. With the trends of large NLP models, the increasing memory and computation costs hinder their efficient deployment on resource-limited devices. Therefore, transformer quantization attracts wide research interest. Recent work recognizes that structured outliers are the critical bottleneck for quantization performance. However, their proposed methods increase the computation overhead and still leave the outliers there. To fundamentally address this problem, this paper delves into the inherent inducement and importance of the outliers. We discover that $\boldsymbol \gamma$ in LayerNorm (LN) acts as a sinful amplifier for the outliers, and the importance of outliers varies greatly where some outliers provided by a few tokens cover a large area but can be clipped sharply without negative impacts. Motivated by these findings, we propose an outlier suppression framework including two components: Gamma Migration and Token-Wise Clipping. The Gamma Migration migrates the outlier amplifier to subsequent modules in an equivalent transformation, contributing to a more quantization-friendly model without any extra burden. The Token-Wise Clipping takes advantage of the large variance of token range and designs a token-wise coarse-to-fine pipeline, obtaining a clipping range with minimal final quantization loss in an efficient way. This framework effectively suppresses the outliers and can be used in a plug-and-play mode. Extensive experiments prove that our framework surpasses the existing works and, for the first time, pushes the 6-bit post-training BERT quantization to the full-precision (FP) level. Our code is available at https://github.com/wimh966/outlier_suppression.

Understanding and Minimising Outlier Features in Neural Network Training

BERT Busters: Outlier Dimensions that Disrupt Transformers

Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models

Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimization

Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing

Outlier-Robust Neural Network Training: Efficient Optimization of Transformed Trimmed Loss with Variation Regularization

Understanding the Difficulty of Training Transformers

Are all outliers alike? On Understanding the Diversity of Outliers for Detecting OODs

Understanding the Feature Norm for Out-of-Distribution Detection

Unified Normalization for Accelerating and Stabilizing Transformers

OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition

Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Models

Training multilayer perceptron by using optimal input normalization

T2FNorm: Extremely Simple Scaled Train-time Feature Normalization for OOD Detection

The Orthogonality of Weight Vectors: the Key Characteristics of Normalization and Residual Connections

Transformer Normalisation Layers and the Independence of Semantic Subspaces

Harnessing Orthogonality to Train Low-Rank Neural Networks

Detecting Out-of-Distribution Through the Lens of Neural Collapse

LINe: Out-of-Distribution Detection by Leveraging Important Neurons

Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks

Average of Pruning: Improving Performance and Stability of Out-of-Distribution Detection