Abstract:Multimodal image matching is a core basis for information fusion, change detection, and image-based navigation. However, multimodal images may simultaneously suffer from severe nonlinear radiation distortion (NRD) and complex geometric differences, which pose great challenges to existing methods. Although deep learning-based methods had shown potential in image matching, they mainly focus on same-source images or single types of multimodal images such as optical-synthetic aperture radar (SAR). One of the main obstacles is the lack of public data for different types of multimodal images. In this paper, we make two major contributions to the community of multimodal image matching: First, we collect six typical types of images, including optical-optical, optical-infrared, optical-SAR, optical-depth, optical-map, and nighttime, to construct a multimodal image dataset with a total of 1200 pairs. This dataset has good diversity in image categories, feature classes, resolutions, geometric variations, etc. Second, we propose a scale and rotation invariant feature transform (SRIF) method, which achieves good matching performance without relying on data characteristics. This is one of the advantages of our SRIF over deep learning methods. SRIF obtains the scales of FAST keypoints by projecting them into a simple pyramid scale space, which is based on the study that methods with/without scale space have similar performance under small scale change factors. This strategy largely reduces the complexity compared to traditional Gaussian scale space. SRIF also proposes a local intensity binary transform (LIBT) for SIFT-like feature description, which can largely enhance the structure information inside multimodal images. Extensive experiments on these 1200 image pairs show that our SRIF outperforms current state-of-the-arts by a large margin, including RIFT, CoFSM, LNIFT, and MS-HLMO. Both the created dataset and the code of SRIF will be publicly available in https://github.com/LJY-RS/SRIF.

Ornament Image Retrieval Using Multimodal Fusion

Sketch-Based 3D Model Retrieval via Multi-feature Fusion

DSSN: dual shallow Siamese network for fashion image retrieval

Content-based jewellery item retrieval using the local region-based histograms

Content-based image retrieval of Indian traditional textile motifs using deep feature fusion

Interpretable Multimodal Retrieval for Fashion Products.

Cross-Modal Image-Recipe Retrieval Via Intra- and Inter-Modality Hybrid Fusion.

Multi-Modal Image Retrieval for Complex Queries using Small Codes

Effective Image Retrieval Via Multilinear Multi-index Fusion.

Adversarially Trained Deep Neural Semantic Hashing Scheme for Subjective Search in Fashion Inventory

Cross-Modal Fusion Distillation for Fine-Grained Sketch-Based Image Retrieval

Multimodal image matching: A scale-invariant algorithm and an open dataset

Object Retrieval and Localization in Large Art Collections using Deep Multi-Style Feature Fusion and Iterative Voting

Modality-Agnostic Attention Fusion for visual search with text feedback

Fusion Based Feature Extraction and Optimal Feature Selection in Remote Sensing Image Retrieval

Visual and textual fusion for semantically supervised region-based retrieval

Video Retrieval Model Based on Multimodal Information Fusion

A Novel Bacterial Foraging Optimization Based Multimodal Medical Image Fusion Approach

Efficient Text-Image Semantic Search: a Multi-modal Vision-Language Approach for Fashion Retrieval

Rank-Aware Graph Fusion With Contextual Dissimilarity Measurement For Image Retrieval

Fabric image retrieval based on multi-modal feature fusion