Abstract:Effectively modeling long-range spatial correlation is crucial in context-sensitive visual computing tasks, such as human pose estimation and video classification. Enlarging receptive field is popularly adopted in building such non-local deep networks. However, current solutions, including dilation convolution or self-attention based operators, mostly suffer from either low computational efficacy or insufficient receptive field. This paper proposes spectral residual learning (SRL), a novel network architectural design for achieving fully global receptive field. A neural block that implements SRL has three key components: a local-to-global transform that projects some ordinary local features into a spectral domain, compiled operations in the spectral domain, and a global-to-local transform that converts all data back to the original local format. We show its equivalence to conducting residual learning in some spectral domain and carefully re-formulate a variety of neural layers into their spectral forms, such as ReLU or convolutions. The benefits of SRL is three-fold: first, all operations have global receptive field, namely any update affects all image positions. This can extract richer context information in various vision tasks; Secondly, the local-to-global / global-to-local transforms in SRL are defined by bi-linear unitary matrices, which is both computation and parameter economic; Lastly, SRL is a generic formulation, here instantiated by Fourier transform and real orthogonal matrix. We conduct comprehensive evaluations on two challenging tasks, including human pose estimation from images and video classification. All experiments clearly show performance improvement by large margins in comparison with conventional non-local network designs.

Non-local Neural Networks

Unifying Nonlocal Blocks for Neural Networks

Inl: Implicit Non-Local Network

Non-Local Neural Networks With Grouped Bilinear Attentional Transforms

Neural Nearest Neighbors Networks

Non‐local Neural Networks Combined with Local Importance‐based Pooling for Space‐time Video Super‐resolution

Disentangled Non-Local Neural Networks

Learning Spatio-Temporal Representation with Local and Global Diffusion

Non-local Graph Convolutional Network

HYPERSPECTRAL IMAGE CLASSIFICATION BASED ON NON-LOCAL NEURAL NETWORKS

Asymmetric Non-Local Neural Networks for Semantic Segmentation.

Fast Non-Local Neural Networks with Spectral Residual Learning

Non-Local Multi-Focus Image Fusion With Recurrent Neural Networks

Non-Local ConvLSTM for Video Compression Artifact Reduction

StNet: Local and Global Spatial-Temporal Modeling for Action Recognition

Neural Architecture Search for Lightweight Non-Local Networks

Fast, nonlocal and neural: a lightweight high quality solution to image denoising

Local Unsupervised Learning for Image Analysis

Image Compressed Sensing Using Non-local Neural Network

Non-local Recurrent Neural Memory for Supervised Sequence Modeling

Global Context Networks