Abstract:In recent years, convolutional neural networks (CNNs) have proven their effectiveness in many challenging computer vision-based tasks, including small object classification. However, according to recent literature, this task is mainly based on 2D CNNs, and the small size of object instances makes their recognition a challenging task. Since 3D CNNs are extremely tedious and time-consuming to learn, they cannot be used in a way that requires a trade-off between accuracy and efficiency. Moreover, due to the great success of Transformers in the field of natural language processing (NLP), a spatial Transformer can also be used as a robust feature transformer and has recently been successfully applied to computer vision tasks, including image classification. By incorporating attention mechanisms into the Transformers, many NLP and computer vision tasks can achieve excellent performance and help learn the contextual encoding of the input patches. However, the complexity of these tasks generally increases with the dimension of the input feature space. In this paper, we propose a novel hybrid 2D/3D CNN-Transformer based on transfer learning and attention mechanisms for better performance on a low-resolution dataset. First, the combination of a pre-trained deep CNN and a 3D CNN can significantly reduce the complexity and result in an accurate learning algorithm. Second, a pre-trained deep CNN model is used as a robust feature extractor and combined with a spatial Transformer to improve the representational power of the developed model and take advantage of the powerful global modeling capabilities of Transformers. Finally, spatial attention and channel attention are adaptively fused by focusing on all components in the input space to capture local and global spatial correlations on non-overlapping regions of the input representation. Experimental results show that the proposed framework has significant relevance in terms of efficiency and accuracy.

3D Morphable Models as Spatial Transformer Networks

Spatial Transformer for 3D Point Clouds

Volumetric Spatial Transformer Network for Object Recognition.

Hierarchical Spatial Transformer Network

Guided Spatial Transformers for Facial Expression Recognition

Disambiguating Monocular Reconstruction of 3D Clothed Human with Spatial-Temporal Transformer

Hybrid-CT: a novel hybrid 2D/3D CNN-Transformer based on transfer learning and attention mechanisms for small object classification

Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimation

Spatial Transformer Introspective Neural Network.

Deep Spatial Transformation for Pose-Guided Person Image Generation and Animation.

Deep Image Spatial Transformation for Person Image Generation

Spatial Temporal Transformer Network for Skeleton-based Action Recognition

Probabilistic Spatial Transformer Networks

PoseMamba: Monocular 3D Human Pose Estimation with Bidirectional Global-Local Spatio-Temporal State Space Model

Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation

TMVNet : Using Transformers for Multi-view Voxel-based 3D Reconstruction

3D Human Pose Estimation with Spatial and Temporal Transformers

On Learning 3D Face Morphable Model from In-the-wild Images

U-shaped Spatial–temporal Transformer Network for 3D Human Pose Estimation

Global-to-Local Modeling for Video-based 3D Human Pose and Shape Estimation

Towards End-to-End Face Recognition Method Through Spatial Transformer