Abstract:Local feature matching, which aims to establish the matches between image pairs, is a pivotal component of multiple visual applications. While current transformer-based works exhibit remarkable performance, they mechanically alternate self- and cross-attention in a predetermined order without considering their prioritization, culminating in inadequate enhancement of visual descriptors. Moreover, when calculating attention matrices to integrate global context, current methods only explicitly model the correlation among the feature channels without taking their importance into account, leaving insufficient message propagation. In this work, we develop a dynamic sparse attention perception (DSAP) matcher to tackle the aforementioned issues. To resolve the first issue, DSAP presents a dynamic perception strategy that enables the network to dynamically implement feature enhancement via modifying both forward and backward propagation. During forward propagation, DPS assigns a learnable perception score to each transformer layer and employs an exponential moving average algorithm to calculate the current score. After that, DPS utilizes an indicator function to binarize the score, allowing DSAP to adaptively determine the appropriate utilization of self- or cross-attention at current iteration. During backward propagation, DPS employs a gradient estimator that adjusts the gradient of perception scores, thus rendering them differentiable. To tackle the second issue, DSAP introduces a weighted sparse transformer (WSFormer) that recalibrates attention matrices by concurrently considering both channel importance and channel correlation. WSFormer predicts attention vectors to weight attention matrices while constructing multiple sparse attention matrices to integrate various global messages, thus highlighting informative channels and inhibiting redundant message propagation. Extensive experiments in public datasets and real environment demonstrate that DSAP achieves exceptional performances across various downstream tasks, including relative pose estimation and visual localization. The code is available at https://github.com/mooncake199809/DSAP.

TKwinFormer: Top k Window Attention in Vision Transformers for Feature Matching

Select & Re-Rank: Effectively and Efficiently Matching Multimodal Data with Dynamically Evolving Attention

DSAP: Dynamic Sparse Attention Perception Matcher for Accurate Local Feature Matching

Feature matching based on local windows aggregation

Efficient Linear Attention for Fast and Accurate Keypoint Matching

ParaFormer: Parallel Attention Transformer for Efficient Feature Matching

Dynamic Keypoint Detection Network for Image Matching

ETO:Efficient Transformer-based Local Feature Matching by Organizing Multiple Homography Hypotheses

Global-Local Similarity for Efficient Fine-Grained Image Recognition with Vision Transformers

LoFLAT: Local Feature Matching using Focused Linear Attention Transformer

Adaptive Spot-Guided Transformer for Consistent Local Feature Matching.

On the Connection between Local Attention and Dynamic Depth-wise Convolution

AMatFormer: Efficient Feature Matching via Anchor Matching Transformer

High-Performance Transformer Tracking

Guide Local Feature Matching by Overlap Estimation

Twins: Revisiting the Design of Spatial Attention in Vision Transformers

LoFTR: Detector-Free Local Feature Matching with Transformers

Efficient LoFTR: Semi-Dense Local Feature Matching with Sparse-Like Speed

A Feature Matching Method Based on Multi-Level Refinement Strategy

LGFCTR: Local and Global Feature Convolutional Transformer for Image Matching