Abstract:Fusing multi-modal information from visible (VIS) and thermal infrared (TIR) images is crucial for object detection in fully adapting to varied lighting conditions. However, the existing models usually treat VIS and TIR images as independent information and extract corresponding features from separate networks due to the scarcity of training data with labeled instances from both VIS and TIR registration images. To fill this gap, a novel Multi-Modal Fusion NETwork (M2FNet) based on the Transformer architecture is proposed in this paper, which contains two effective modules: the Union-Modal Attention (UMA) and the Cross-Modal Attention (CMA). The UMA module aggregates multi-spectral features from VIS and TIR images and then extracts multi-modal features via a convolutional neural network (CNN) backbone. The CMA module is designed to learn cross-attention features from VIS and TIR pairwise features by Transformer architecture. Evaluation results by the mean average precision (mAP) metric show that the M2FNet method significantly advances the baseline methods trained using only VIS or TIR images by 10.71 % and 2.97 %, respectively. The increments in mAP are observed in the M2FNet method compared with the existing multi-modal methods on two public datasets. Sensitivity analysis of eight illumination thresholds shows that the M2FNet method presents robustness performance on varied illumination conditions and achieves the maximum increase in accuracy of 25.6 %. Moreover, this method is subsequently applied to a new testing dataset, VI2DA (Visible-Infrared paired Video and Image DAtaset), observed by diverse sensors and platforms for testing the generalization ability of object detectors, which will be publicly available at https://github.com/TIR-OD/Datasets.

GM-DETR: Generalized Muiltispectral DEtection TRansformer with Efficient Fusion Encoder for Visible-Infrared Detection

MS-DETR: Multispectral Pedestrian Detection Transformer with Loosely Coupled Fusion and Modality-Balanced Optimization

DAMSDet: Dynamic Adaptive Multispectral Detection Transformer with Competitive Query Selection and Adaptive Feature Fusion

Cross Teaching Between Single-Spectral and Multi-Spectral Detection Transformers for Remote Sensing Object Detection

DPDETR: Decoupled Position Detection Transformer for Infrared-Visible Object Detection

Transformer fusion and histogram layer multispectral pedestrian detection network

DFECF-DET: All-Weather Detector Based on Differential Feature Enhancement and Cross-Modal Fusion With Visible and Infrared Sensors

DMM: Disparity-guided Multispectral Mamba for Oriented Object Detection in Remote Sensing

Multispectral Object Detection Based on Multilevel Feature Fusion and Dual Feature Modulation

Transformer Based Remote Sensing Object Detection with Enhanced Multispectral Feature Extraction.

Cross-Modality Fusion Transformer for Multispectral Object Detection

Cross-Modality Attentive Feature Fusion for Object Detection in Multispectral Remote Sensing Imagery

Improving RGB-infrared object detection with cascade alignment-guided transformer

MV-DETR: Multi-modality indoor object detection by Multi-View DEtecton TRansformers

ETFormer: an Efficient Transformer Based on Multimodal Hybrid Fusion and Representation Learning for RGB-D-T Salient Object Detection

HATF: Multi-Modal Feature Learning for Infrared and Visible Image Fusion via Hybrid Attention Transformer

RGB-T Object Detection via Group Shuffled Multi-receptive Attention and Multi-modal Supervision

M2FNet: Multi-modal Fusion Network for Object Detection from Visible and Thermal Infrared Images

ICAFusion: Iterative cross-attention guided feature fusion for multispectral object detection

Cross-modal Deformable DETR for RGB-D Object Detection

Removal and Selection: Improving RGB-Infrared Object Detection via Coarse-to-Fine Fusion