Abstract:Fine-grained image datasets have small inter-class differences and large intra-class differences, which is a difficulty of the fine-grained image classification. Traditional fine-grained image classification methods only focus on the visual features of images. However, this limitation can be eliminated when these methods are improved with multimodal information. This paper proposes an improved fine-grained image classification method with multimodal information that includes multimodal data preprocessing, multimodal feature extraction, multi-temporal feature fusion and decision correction. The preprocessing method proposed solves the problems of scattered distribution, difficult processing and uneven contribution to prediction of multimodal data through normalization, packing phrases and weighted concatenating methods. When extracting multimodal features, the SAMLP (Self-Attention MLP) module proposed combines self-attention with MLP to capture the internal correlation of multimodal information. The multi-temporal feature fusion proposed is divided into early feature fusion and late feature fusion. The former refers to adding multimodal information markers to the original image, and the latter refers to designing a multi-cascade dynamic MLP structure to fuse visual features and multimodal features. In view of the limitation of feature fusion, a decision strategy is proposed to revise the prediction results of fused features according to the prediction results of multimodal features. Ablation experiment on INAT18-1K and INAT21-1K datasets shows that our method is effective in improving classification with multimodal information. Experiments on the INAT2021_mini large dataset show that the comprehensive method in this paper has higher accuracy and negligible efficiency loss compared with the state-of-the-art method.

Convolutional Attention Network with Maximizing Mutual Information for Fine-Grained Image Classification

Research on Image Classification Method of Features of Combinatorial Convolution

A Hyperspectral Image Classification Method Based on the Nonlocal Attention Mechanism of a Multiscale Convolutional Neural Network.

HAM: Hybrid Attention Module in Deep Convolutional Neural Networks for Image Classification

A Fine-Grained Image Classification and Detection Method Based on Convolutional Neural Network Fused with Attention Mechanism

Dual attention guided multi-scale CNN for fine-grained image classification

Improving Fine-grained Image Classification with Multimodal Information

Fine-grained image classification method based on hybrid attention module

The Application of Two-Level Attention Models in Deep Convolutional Neural Network for Fine-Grained Image Classification

Fine-Grained Image Classification Based on Cross-Attention Network

Weakly supervised fine-grained image classification via two-level attention activation model

Learning Multi-attention Convolutional Neural Network for Fine-Grained Image Recognition

Multi-Attention Multi-Class Constraint for Fine-grained Image Recognition

Subtler mixed attention network on fine-grained image classification

Improving Deep Learning on Point Cloud by Maximizing Mutual Information Across Layers

Multi-level Dictionary Learning for Fine-Grained Images Categorization with Attention Model

Coarse to Fine: Multi-label Image Classification with Global/Local Attention

Midcn: A Multiple Instance Deep Convolutional Network For Image Classification

Attention Multihop Graph and Multiscale Convolutional Fusion Network for Hyperspectral Image Classification

Coarse-to-fine Trained Multi-Scale Convolutional Neural Networks for Image Classification.