Abstract:Fine-grained image datasets have small inter-class differences and large intra-class differences, which is a difficulty of the fine-grained image classification. Traditional fine-grained image classification methods only focus on the visual features of images. However, this limitation can be eliminated when these methods are improved with multimodal information. This paper proposes an improved fine-grained image classification method with multimodal information that includes multimodal data preprocessing, multimodal feature extraction, multi-temporal feature fusion and decision correction. The preprocessing method proposed solves the problems of scattered distribution, difficult processing and uneven contribution to prediction of multimodal data through normalization, packing phrases and weighted concatenating methods. When extracting multimodal features, the SAMLP (Self-Attention MLP) module proposed combines self-attention with MLP to capture the internal correlation of multimodal information. The multi-temporal feature fusion proposed is divided into early feature fusion and late feature fusion. The former refers to adding multimodal information markers to the original image, and the latter refers to designing a multi-cascade dynamic MLP structure to fuse visual features and multimodal features. In view of the limitation of feature fusion, a decision strategy is proposed to revise the prediction results of fused features according to the prediction results of multimodal features. Ablation experiment on INAT18-1K and INAT21-1K datasets shows that our method is effective in improving classification with multimodal information. Experiments on the INAT2021_mini large dataset show that the comprehensive method in this paper has higher accuracy and negligible efficiency loss compared with the state-of-the-art method.

Cost-Sensitive Deep Metric Learning for Fine-Grained Image Classification.

FGCM: Noisy Label Learning via Fine-Grained Confidence Modeling

Confusion-Based Metric Learning for Regularizing Zero-Shot Image Retrieval and Clustering

Scalable Angular Discriminative Deep Metric Learning for Face Recognition

Deep Compositional Metric Learning

Metric Learning with Adaptive Density Discrimination

Dynamic Metric Learning with Cross-Level Concept Distillation.

Deep Factorized Metric Learning

Class-specific discriminative metric learning for scene recognition

Exponential Discriminative Metric Embedding in Deep Learning

Introspective Deep Metric Learning

Probabilistic Deep Metric Learning for Hyperspectral Image Classification

When Deep Learning Meets Metric Learning: Remote Sensing Image Scene Classification via Learning Discriminative CNNs

Deep Metric Learning Via Adaptive Learnable Assessment

Deep Multimodal Distance Metric Learning Using Click Constraints for Image Ranking.

Reconstruction Regularized Deep Metric Learning for Multi-label Image Classification

Relationship constraint deep metric learning

Improving Fine-grained Image Classification with Multimodal Information

Dual attention guided multi-scale CNN for fine-grained image classification

Deep Multiple Metric Learning for Time Series Classification

LAM3L: Locally Adaptive Maximum Margin Metric Learning for Visual Data Classification