Abstract:Transformers have made great progress in dealing with computer vision tasks. However, existing vision transformers do not yet possess the ability of building the interactions among features of different scales, which is perceptually important to visual inputs. The reasons are two-fold: (1) Input embeddings of each layer are equal-scale, so no cross-scale feature can be extracted; (2) to lower the computational cost, some vision transformers merge adjacent embeddings inside the self-attention module, thus sacrificing small-scale (fine-grained) features of the embeddings and also disabling the cross-scale interactions. To this end, we propose Cross-scale Embedding Layer (CEL) and Long Short Distance Attention (LSDA). On the one hand, CEL blends each embedding with multiple patches of different scales, providing the self-attention module itself with cross-scale features. On the other hand, LSDA splits the self-attention module into a short-distance one and a long-distance counterpart, which not only reduces the computational burden but also keeps both small-scale and large-scale features in the embeddings. Through the above two designs, we achieve cross-scale attention. Besides, we put forward a dynamic position bias for vision transformers to make the popular relative position bias apply to variable-sized images. Hinging on the cross-scale attention module, we construct a versatile vision architecture, dubbed CrossFormer, which accommodates variable-sized inputs. Extensive experiments show that CrossFormer outperforms the other vision transformers on image classification, object detection, instance segmentation, and semantic segmentation tasks. The code has been released: <a class="link-external link-https" href="https://github.com/cheerss/CrossFormer" rel="external noopener nofollow">this https URL</a>.

Cross-Axis Transformer with 3D Rotary Positional Embeddings

Cross-Axis Transformer with 2D Rotary Embeddings

Axial Attention in Multidimensional Transformers

CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention

CAT: Cross Attention in Vision Transformer

AxWin Transformer: A Context-Aware Vision Transformer Backbone with Axial Windows

Can We Solve 3D Vision Tasks Starting from A 2D Vision Transformer?

Orthogonal Transformer: An Efficient Vision Transformer Backbone with Token Orthogonalization

Multi-view Convolutional Vision Transformer for 3D Object Recognition.

Three things everyone should know about Vision Transformers

ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias

Cross-Space-Time 3D Human Body Pose Estimation Based on Transformer

PointCAT: Cross-Attention Transformer for point cloud

Estimating Extreme 3D Image Rotation with Transformer Cross-Attention

ACC-ViT : Atrous Convolution's Comeback in Vision Transformers

CMT: Convolutional Neural Networks Meet Vision Transformers

Transformers in Unsupervised Structure-from-Motion

Feature‐enhanced representation with transformers for multi‐view stereo

CVIformer: Cross-View Interactive Transformer for Efficient Stereoscopic Image Super-Resolution

Cross-view Transformers for real-time Map-view Semantic Segmentation

3D Vision with Transformers: A Survey