Abstract:We propose a novel end-to-end image-text matching approach considering semantic uncertainty (SU-ITM), aiming to deal with the one-to-many semantic diversity involved in image-text matching in order to capture the associations between them more comprehensively and improve the robustness of the model. Traditional methods map images and texts as definite points in an embedding space to measure cross-modal similarity. However, the point-based embedding cannot capture the semantic uncertainty, leading to a large bias in the matching results. To address this problem, we model the one-to-many associations between image and text in a way that establishes a probability distribution, incorporating the uncertainty information into the final semantic representation of the text. In addition, we optimize the image-text matching loss so that the different text features approximate the image features in a distributed manner while maintaining the discriminative nature of the semantic representation, effectively reducing the matching uncertainty. Notably, our method achieves end-to-end training by not using pre-trained target detection branches throughout the training process. We fully demonstrate the excellent performance of our method in the image-text matching task through experimental validation on Flickr30k and MSCOCO. Excellent performance levels of 546.1 and 545.0 are achieved on the R@SUM metric for Flickr30k and MSCOCO 1k, respectively.

Novel Image Matching Confidence Fusion Evaluation Algorithm Based on Support Vector Machine

Source Cell-phone Identification Based on Multi-feature Fusion.

Modulation Transfer Function (MTF) Measurement Method Based on Support Vector Machine (SVM)

An Image Matching Algorithm Based On Singular Value Decomposition and Belief Propagation

Show Your Faith: Cross-Modal Confidence-Aware Network for Image-Text Matching.

Encoding human knowledge for visual pattern recognition

An Image Matching Algorithm Based on Improved Rank Transform

Research on an SVM Coupling Algorithm of Image Recognition

Improved SVD Matching Based on Delaunay Triangulation and Belief Propagation

Iris Recognition Based on Multialgorithmic Fusion

Gender Classification Based on Support Vector Machine with Automatic Confidence

Fusing image representations for classification using support vector machines

Decision Fusion Based Cartridge Identification Using Support Vector Machine

Study on a SVM-based data fusion method.

A Reliable Algorithm for Image Matching Based on SIFT

Multi-source Remote Sensing Image Fusion Based on Support Vector Machine

A Novel Image Matching Method Based on SIFT

An End-to-End Image-Text Matching Approach Considering Semantic Uncertainty

Multi-Sensor Image Fusion Using Optimized Support Vector Machine and Multiscale Weighted Principal Component Analysis

Novel Image Fusion Algorithm with Novel Performance Evaluation Method

Geometry Consistency Aware Confidence Evaluation for Feature Matching