Abstract:RGB-Infrared multi-modal object detection utilizes diverse and complementary information, showing some advantages in intelligent transportation field. The main challenge of RGB-Infrared object detection is how to fuse the two modalities. The difficulty of fusion is reflected in two aspects: 1) large visual differences between modalities make it difficult to learn effective complementary features, 2) some misaligned RGB-Infrared images increase the difficulty of fusion. To this end, based on feature pyramid commonly used in object detection, we propose Multi-modal Feature Pyramid Transformer (MFPT) to fuse the two modalities. The proposed MFPT learns semantic and modal complementary information to enhance each modal features via intra-modal feature pyramid transformer and inter-modal feature pyramid transformer. The intra-modal feature pyramid transformer enables features to interact across space and scales, improving the semantic representations of features in each modality. The inter-modal feature pyramid transformer conducts feature interaction between modalities, enabling each modality to learn complementary features from other modalities. Meanwhile, the inter-modal feature pyramid transformer can also learn distance independent dependencies between modalities, which are not sensitive to misaligned images. Furthermore, a local attention mechanism is introduced within different windows into MFPT to achieve efficient correlation between regions of different scales or different modalities. Experimental results on two RGB-Infrared detection datasets demonstrate the proposed method is superior to state-of-the-art methods.

A Multispectral-Infrared Object Detection Method Based on Cross-Modality Image Feature Filtering Fusion

Infrared and visible image fusion method based on saliency detection and target-enhancement

Infrared and Radar Fusion Detection Method Based on Heterogeneous Data Preprocessing

Multi-Modal Target Detection Method Based on Adaptive Feature Search

Research on Single Object Detection Technology Based on Infrared Multi-spectrum Fusion

ICAFusion: Iterative cross-attention guided feature fusion for multispectral object detection

Cross-Modality Attentive Feature Fusion for Object Detection in Multispectral Remote Sensing Imagery

Multiscale Progressive Fusion Filter Network for Infrared Small Target Detection

Multi-Modal Object Detection Method Based on Dual-Branch Asymmetric Attention Backbone and Feature Fusion Pyramid Network

Infrared Target Detection And Location For Visual Surveillance Using Fusion Scheme Of Visible And Infrared Images

Multi-Frame Image Fusion Method Combining Spatial-Temporal Saliency Detection and Nsct

Adaptive multimodal feature fusion with frequency domain gate for remote sensing object detection

Multi-Level Image Fusion and Enhancement for Target Detection

An object detection algorithm based on infrared-visible dual modal feature fusion

Infrared dual-band image fusion with simplified pulse coupled neural network and visual saliency map in nonsubsampled shearlet transform domain

Background-Aware Cross-Attention Multiscale Fusion for Multispectral Object Detection

Study on Airborne Multispectral Imaging Fusion Detection Technology

Cross-Modality Fusion Transformer for Multispectral Object Detection

Removal and Selection: Improving RGB-Infrared Object Detection via Coarse-to-Fine Fusion

Multi-Stage Multi-Scale Local Feature Fusion for Infrared Small Target Detection

Multi-Modal Feature Pyramid Transformer for RGB-Infrared Object Detection