Abstract:Local feature matching, which aims to establish the matches between image pairs, is a pivotal component of multiple visual applications. While current transformer-based works exhibit remarkable performance, they mechanically alternate self- and cross-attention in a predetermined order without considering their prioritization, culminating in inadequate enhancement of visual descriptors. Moreover, when calculating attention matrices to integrate global context, current methods only explicitly model the correlation among the feature channels without taking their importance into account, leaving insufficient message propagation. In this work, we develop a dynamic sparse attention perception (DSAP) matcher to tackle the aforementioned issues. To resolve the first issue, DSAP presents a dynamic perception strategy that enables the network to dynamically implement feature enhancement via modifying both forward and backward propagation. During forward propagation, DPS assigns a learnable perception score to each transformer layer and employs an exponential moving average algorithm to calculate the current score. After that, DPS utilizes an indicator function to binarize the score, allowing DSAP to adaptively determine the appropriate utilization of self- or cross-attention at current iteration. During backward propagation, DPS employs a gradient estimator that adjusts the gradient of perception scores, thus rendering them differentiable. To tackle the second issue, DSAP introduces a weighted sparse transformer (WSFormer) that recalibrates attention matrices by concurrently considering both channel importance and channel correlation. WSFormer predicts attention vectors to weight attention matrices while constructing multiple sparse attention matrices to integrate various global messages, thus highlighting informative channels and inhibiting redundant message propagation. Extensive experiments in public datasets and real environment demonstrate that DSAP achieves exceptional performances across various downstream tasks, including relative pose estimation and visual localization. The code is available at https://github.com/mooncake199809/DSAP.

Learning Deep Local Features with Multiple Dynamic Attentions for Large-Scale Image Retrieval.

Learning Feature Embedding with Strong Neural Activations for Fine-Grained Retrieval

Learning Discriminative Features for Image Retrieval.

All the attention you need: Global-local, spatial-channel attention for image retrieval

Learning Token-Based Representation for Image Retrieval

Deep Learning for Content-Based Image Retrieval: A Comprehensive Study

Learning Click-Based Deep Structure-Preserving Embeddings with Visual Attention.

Efficient Multi-Scale Attention Module with Cross-Spatial Learning

FFLDGA-Net: Image retrieval method based on Feature Fusion Learnable Descriptor Graph Attention Network

Bio-inspired deep neural local acuity and focus learning for visual image recognition

Learning to Discover Multi-Class Attentional Regions for Multi-Label Image Recognition

Attention Weighted Local Descriptors

A Feature Learning Approach For Image Retrieval

Multi-level Dictionary Learning for Fine-Grained Images Categorization with Attention Model

Deep Spatial Attention Hashing Network for Image Retrieval.

ELA: Efficient Local Attention for Deep Convolutional Neural Networks

DSAP: Dynamic Sparse Attention Perception Matcher for Accurate Local Feature Matching

Deep Convolutional Hamming Ranking Network for Large Scale Image Retrieval

Multi-Scale Context Attention Network for Image Retrieval.

Coarse-to-Fine: Learning Compact Discriminative Representation for Single-Stage Image Retrieval

Multi-Attention Multi-Class Constraint for Fine-grained Image Recognition