Abstract:Transformers have captured growing attention in computer vision, thanks to its large capacity and global processing capabilities. However, transformers are data hungry, and their ability to generalize is constrained compared to Convolutional Neural Networks (ConvNets), especially when trained with limited data due to the absence of the built-in spatial inductive biases present in ConvNets. In this paper, we strive to optimally combine the strengths of both convolution and transformers for image classification tasks. Towards this end, we present a novel lightweight hybrid network that pairs Convolution with Transformers via Representation Learning Fusion and Multi-Level Feature Cross-Attention named CTRL-F. Our network comprises a convolution branch and a novel transformer module named multi-level feature cross-attention (MFCA). The MFCA module operates on multi-level feature representations obtained at different convolution stages. It processes small patch tokens and large patch tokens extracted from these multi-level feature representations via two separate transformer branches, where both branches communicate and exchange knowledge through cross-attention mechanism. We fuse the local responses acquired from the convolution path with the global responses acquired from the MFCA module using novel representation fusion techniques dubbed adaptive knowledge fusion (AKF) and collaborative knowledge fusion (CKF). Experiments demonstrate that our CTRL-F variants achieve state-of-the-art performance, whether trained from scratch on large data or even with low-data regime. For Instance, CTRL-F achieves top-1 accuracy of 82.24% and 99.91% when trained from scratch on Oxford-102 Flowers and PlantVillage datasets respectively, surpassing state-of-the-art models which showcase the robustness of our model on image classification tasks. Code at: <a class="link-external link-https" href="https://github.com/hosamsherif/CTRL-F" rel="external noopener nofollow">this https URL</a>

What problem does this paper attempt to address?

The paper aims to address the following key issues: 1. **Combining the advantages of Convolutional Neural Networks (ConvNets) and Transformers**: Researchers have found that while Transformers exhibit strong performance in computer vision tasks, especially in handling global information, they require a large amount of data and their generalization ability is somewhat lacking when trained on limited datasets compared to Convolutional Neural Networks. Therefore, one of the goals of the paper is to design a hybrid model that effectively integrates the strengths of both architectures. 2. **Improving performance in image classification tasks**: By combining the local feature extraction capabilities of Convolutional Neural Networks with the global processing capabilities of Transformers, the paper proposes a new model named CTRL-F, which aims to improve the accuracy of image classification tasks, particularly in scenarios with limited data. 3. **Optimizing computational efficiency and resource utilization**: Considering the demand for computational resources in practical applications, the paper also focuses on constructing an efficient and lightweight hybrid model that can run in real-time on resource-constrained devices. Specifically, the main contributions of the paper include: - Proposing a new hybrid model, CTRL-F, which combines Convolutional Neural Networks and Transformers to achieve better image classification performance. - Designing a module called Multi-level Feature Cross Attention (MFCA), which operates on multi-scale feature maps generated at different convolution stages to capture long-range dependencies. - Developing two effective knowledge fusion techniques—Adaptive Knowledge Fusion (AKF) and Collaborative Knowledge Fusion (CKF)—to effectively combine the local responses of the convolutional path with the global responses of the transformer path. Through extensive experimental validation, the paper demonstrates that CTRL-F outperforms some existing state-of-the-art models on various benchmark datasets, achieving good results whether trained on large or small datasets. This proves the effectiveness and robustness of CTRL-F in image classification tasks.

CTRL-F: Pairing Convolution with Transformer for Image Classification via Multi-Level Feature Cross-Attention and Representation Learning Fusion

A Novel Transformer Network with a CNN-Enhanced Cross-Attention Mechanism for Hyperspectral Image Classification

Multiscale 3-D-2-D Mixed CNN and Lightweight Attention-Free Transformer for Hyperspectral and LiDAR Classification

A Transformer-Based Object Detector with Coarse-Fine Crossing Representations

CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification

Multi-Modal Image Fusion Via Deep Laplacian Pyramid Hybrid Network

Image Fusion Transformer

Multimodal Fusion Transformer for Remote Sensing Image Classification

MAFormer: A transformer network with multi-scale attention fusion for visual recognition

Conformer: Local Features Coupling Global Representations for Visual Recognition

CITNet: Convolution Interaction Transformer Network for Hyperspectral and LiDAR Image Classification

CAT: Cross Attention in Vision Transformer

Cross-Modality Fusion Transformer for Multispectral Object Detection

MFF-Trans: Multi-level Feature Fusion Transformer for Fine-Grained Visual Classification

A hybrid convolution transformer for hyperspectral image classification

CATFace: Cross-Attribute-Guided Transformer with Self-Attention Distillation for Low-Quality Face Recognition

LGFCTR: Local and Global Feature Convolutional Transformer for Image Matching

Combining transformers with CNN for multi-focus image fusion

CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention

A free lunch from ViT:Adaptive Attention Multi-scale Fusion Transformer for Fine-grained Visual Recognition

MCANet: Hierarchical cross-fusion lightweight transformer based on multi-ConvHead attention for object detection