Abstract:In recent years, convolutional neural networks (CNNs) have proven their effectiveness in many challenging computer vision-based tasks, including small object classification. However, according to recent literature, this task is mainly based on 2D CNNs, and the small size of object instances makes their recognition a challenging task. Since 3D CNNs are extremely tedious and time-consuming to learn, they cannot be used in a way that requires a trade-off between accuracy and efficiency. Moreover, due to the great success of Transformers in the field of natural language processing (NLP), a spatial Transformer can also be used as a robust feature transformer and has recently been successfully applied to computer vision tasks, including image classification. By incorporating attention mechanisms into the Transformers, many NLP and computer vision tasks can achieve excellent performance and help learn the contextual encoding of the input patches. However, the complexity of these tasks generally increases with the dimension of the input feature space. In this paper, we propose a novel hybrid 2D/3D CNN-Transformer based on transfer learning and attention mechanisms for better performance on a low-resolution dataset. First, the combination of a pre-trained deep CNN and a 3D CNN can significantly reduce the complexity and result in an accurate learning algorithm. Second, a pre-trained deep CNN model is used as a robust feature extractor and combined with a spatial Transformer to improve the representational power of the developed model and take advantage of the powerful global modeling capabilities of Transformers. Finally, spatial attention and channel attention are adaptively fused by focusing on all components in the input space to capture local and global spatial correlations on non-overlapping regions of the input representation. Experimental results show that the proposed framework has significant relevance in terms of efficiency and accuracy.

Volumetric Spatial Transformer Network for Object Recognition.

OVPT: Optimal Viewset Pooling Transformer for 3D Object Recognition.

Multi-view Convolutional Vision Transformer for 3D Object Recognition.

Spatial Transformer for 3D Point Clouds

Multi-view 3D Reconstruction with Transformer

PVT-SSD: Single-Stage 3D Object Detector with Point-Voxel Transformer

SparseVoxNet: 3-D Object Recognition With Sparsely Aggregation of 3-D Dense Blocks

STFormer3D: Spatio-Temporal Transformer Based 3D Object Detection for Intelligent Driving.

VTP: Volumetric Transformer for Multi-view Multi-person 3D Pose Estimation

TMVNet : Using Transformers for Multi-view Voxel-based 3D Reconstruction

Hybrid CNN-Transformer Features for Visual Place Recognition

DS-Trans: A 3D Object Detection Method Based on a Deformable Spatiotemporal Transformer for Autonomous Vehicles

Multi-Scale Spatial Transformer Network for LiDAR-Camera 3D Object Detection.

Complementary spatial transformer network for real-time 3D object recognition

Hybrid-CT: a novel hybrid 2D/3D CNN-Transformer based on transfer learning and attention mechanisms for small object classification

CodedVTR: Codebook-based Sparse Voxel Transformer with Geometric Guidance

Toward Real-Time 3D Object Recognition: A Lightweight Volumetric CNN Framework Using Multitask Learning

Towards End-to-End Face Recognition Method Through Spatial Transformer

MVTr: Multi-Feature Voxel Transformer for 3D Object Detection

3D Former: Monocular Scene Reconstruction with 3D SDF Transformers