Abstract:With the rapid advent and abundance of remote sensing data in different modalities, cross-modal retrieval tasks have gained importance in the research community. Cross-modal retrieval belongs to the research paradigm in which the query is of one modality and the retrieved output is of the other modality. In this paper, the remote sensing (RS) data modalities considered are the earth observation optical data (aerial photos) and the corresponding hand-drawn sketches. The main challenge of the cross-modal retrieval research objective for optical remote sensing images and the corresponding sketches is the distribution gap between the shared embedding space of the modalities. Prior attempts to resolve this issue have not yielded satisfactory outcomes regarding accurately retrieving cross-modal sketch-image RS data. The state-of-the-art architectures used conventional convolutional architectures, which focused on local pixel-wise information about the modalities to be retrieved. This limits the interaction between the sketch texture and the corresponding image, making these models susceptible to overfitting datasets with particular scenarios. To circumvent this limitation, we suggest establishing multi-modal correspondence using a novel architecture of the combined self and cross-attention algorithms, SPCA-Net to minimize the modality gap by employing attention mechanisms for the query and other modalities. Efficient cross-modal retrieval is achieved through the suggested attention architecture, which empirically emphasizes the global information of the relevant query modality and bridges the domain gap through a unique pairwise cross-attention network. In addition to the novel architecture, this paper introduces a unique loss function, label-specific supervised contrastive loss , tailored to the intricacies of the task and to enhance the discriminative power of the learned embeddings. Extensive evaluations are conducted on two sketch-image remote sensing datasets, Earth-on-Canvas and RSketch. Under the same experimental conditions, the performance metrics of our proposed model beat the state-of-the-art architectures by significant margins of 16.7%, 18.9%, 33.7%, and 40.9% correspondingly.

High Boost 3-D Attention Network for Cross-Spectral Periocular Recognition

Perturbed Attention-Assisted Siamese Network for Cross-Spectral Periocular Recognition

Approximate Multidimensional Discrete Cosine Transform-Based Deep Attention Network for Cross-Spectral Periocular Recognition

Holistic Feature Reconstruction-Based 3-D Attention Mechanism for Cross-Spectral Periocular Recognition

PerimetryNet: A Multiscale Fine Grained Deep Network for Three-Dimensional Eye Gaze Estimation Using Visual Field Analysis

Attention in Attention for Hyperspectral with High Spatial Resolution (H) Image Classification

Channel and band attention embedded 3D CNN for model development of hyperspectral image in object-scale analysis

One shot learning approach for cross spectrum periocular verification

Enhanced near-infrared periocular recognition through collaborative rendering of hand crafted and deep features

Spectral-Spatial-Temporal Attention Network for Hyperspectral Tracking.

Boosting Multi-modal Ocular Recognition via Spatial Feature Reconstruction and Unsupervised Image Quality Estimation

Cross-Spectral Attention for Unsupervised RGB-IR Face Verification and Person Re-identification

Near-infrared and visible-light periocular recognition with Gabor features using frequency-adaptive automatic eye detection

Cross-Sensor Periocular Biometrics in a Global Pandemic: Comparative Benchmark and Novel Multialgorithmic Approach

Boosting cross-modal retrieval in remote sensing via a novel unified attention network

A Deep Feature Fusion Network Based on Multiple Attention Mechanisms for Joint Iris-Periocular Biometric Recognition

Spectrum-driven Mixed-frequency Network for Hyperspectral Salient Object Detection

S 3 AM: A Spectral-Similarity-Based Spatial Attention Module for Hyperspectral Image Classification.

Spectral-Spatial-Sensorial Attention Network with Controllable Factors for Hyperspectral Image Classification

Hyperspectral Image Classification Based on 3D Coordination Attention Mechanism Network

Hyperspectral Image Classification Based on 3-D Octave Convolution With Spatial–Spectral Attention Network