Abstract:Fine-grained image datasets have small inter-class differences and large intra-class differences, which is a difficulty of the fine-grained image classification. Traditional fine-grained image classification methods only focus on the visual features of images. However, this limitation can be eliminated when these methods are improved with multimodal information. This paper proposes an improved fine-grained image classification method with multimodal information that includes multimodal data preprocessing, multimodal feature extraction, multi-temporal feature fusion and decision correction. The preprocessing method proposed solves the problems of scattered distribution, difficult processing and uneven contribution to prediction of multimodal data through normalization, packing phrases and weighted concatenating methods. When extracting multimodal features, the SAMLP (Self-Attention MLP) module proposed combines self-attention with MLP to capture the internal correlation of multimodal information. The multi-temporal feature fusion proposed is divided into early feature fusion and late feature fusion. The former refers to adding multimodal information markers to the original image, and the latter refers to designing a multi-cascade dynamic MLP structure to fuse visual features and multimodal features. In view of the limitation of feature fusion, a decision strategy is proposed to revise the prediction results of fused features according to the prediction results of multimodal features. Ablation experiment on INAT18-1K and INAT21-1K datasets shows that our method is effective in improving classification with multimodal information. Experiments on the INAT2021_mini large dataset show that the comprehensive method in this paper has higher accuracy and negligible efficiency loss compared with the state-of-the-art method.

Heterogeneous Image Features Integration Via Multi-modal Semi-supervised Learning Model

Heterogeneous Image Feature Integration Via Multi-Modal Spectral Clustering

Heterogeneous Features Integration via Semi-supervised Multi-modal Deep Networks.

Heterogeneous Visual Features Fusion via Sparse Multimodal Machine

Integrating hierarchical feature selection and classifier training for multi-label image annotation.

Multi-modal Learning for Social Image Classification

Classify Social Image by Integrating Multi-Modal Content

Multi-Modal Curriculum Learning for Semi-Supervised Image Classification

Semi-supervised Cross-Modal Learning for Cross Modal Retrieval and Image Annotation.

Semi-supervised learning based on intra-view heterogeneity and inter-view compatibility for image classification

Heterogeneous feature learning network for multimodal remote sensing image collaborative classification

Semisupervised image classification by mutual learning of multiple self‐supervised models

Multi-Modal Multi-Label Semantic Indexing Of Images Based On Hybrid Ensemble Learning

Heterogeneous Multi-task Semantic Feature Learning for Classification

Image classification by multimodal subspace learning

Multi-Modal Multi-Label Semantic Indexing of Images Using Unlabeled Data

Multimodal Learning for Multi-Label Image Classification.

Semi-supervised image clustering with multi-modal information

Improving Fine-grained Image Classification with Multimodal Information

Multimodal Ground-Based Remote Sensing Cloud Classification Via Learning Heterogeneous Deep Features.

Adaptive Semi-Supervised Feature Selection for Cross-Modal Retrieval