Abstract:Multimodal fusion object detection aims to improve detection accuracy by integrating information from multiple modalities. RGB image-based object detection makes finely sorting solid waste according to material challenging. This article built a dual-camera acquisition platform using a line-scan color camera and a hyperspectral camera to collect RGB and hyperspectral images. In order to use RGB images and hyperspectral images for feature fusion more effectively, we propose an asymmetric multiscale feature fusion network (AMFFNet) based on RGB-near-infrared (NIR) multisensor fusion technology. Specifically, we designed a hyperspectral image convolution unit (HICU) to fully extract multiscale features from hyperspectral images. Second, we concatenate the hyperspectral feature maps and the feature maps output by feature pyramid networks (FPNs) of the RGB image feature extraction stage to achieve asymmetric multiscale feature fusion. In addition, a dimensionality reduction strategy (DRS) is proposed to remove the bands that are redundant and have a low signal-to-noise ratio in hyperspectral images. Ablation studies have confirmed the effectiveness of AMFFNet components, and we have conducted extensive integration experiments on the article-based solid waste dataset. The experimental results and analysis show that AMFFNet based on mask region-based convolutional neural network (Mask RCNN), faster RCNN, and RetinaNet detectors outperforms the original model by 5.05%, 3.57%, and 6.36% on AP at IoU = 0.5 and by 3.1%, 2.05%, and 3.73% on COCO’s standard AP metric, and the parameters of the network increased by 0.47–0.56 M. In conclusion, the proposed method effectively improves the performance of the object detection model for fine identification of solid waste.

Enhanced Spectral-Spatial Fusion Network for Multispectral Object Detection in Ground-Aerial Images

Multi-spectral Image Fusion for Moving Object Detection

Cross-Modality Fusion Transformer for Multispectral Object Detection

Modality-inter Fusion and Enhancement Network for Dual-Spectral Object Detection

Transformer fusion and histogram layer multispectral pedestrian detection network

Cross-Modality Attentive Feature Fusion for Object Detection in Multispectral Remote Sensing Imagery

Dual Stream Cross Domain Feature Fusion for Land-Oceanic Object Detection

Multispectral Object Detection Based on Multilevel Feature Fusion and Dual Feature Modulation

Background-Aware Cross-Attention Multiscale Fusion for Multispectral Object Detection

MFMG-Net: Multispectral Feature Mutual Guidance Network for Visible–Infrared Object Detection

Multispectral Fusion for Object Detection with Cyclic Fuse-and-Refine Blocks

ICAFusion: Iterative cross-attention guided feature fusion for multispectral object detection

Multimodal Transformer Using Cross-Channel attention for Object Detection in Remote Sensing Images

Multi-Modal Object Detection Method Based on Dual-Branch Asymmetric Attention Backbone and Feature Fusion Pyramid Network

Long-Tailed Object Detection for Multimodal Remote Sensing Images

Multispectral Fusion Transformer Network for RGB-Thermal Urban Scene Semantic Segmentation

Feature Enhancement and Fusion for RGB-T Salient Object Detection

A Task-Balanced Multiscale Adaptive Fusion Network for Object Detection in Remote Sensing Images

AMFFNet: Asymmetric Multiscale Feature Fusion Network of RGB-NIR for Solid Waste Detection.

Airborne Small Target Detection Method Based on Multimodal and Adaptive Feature Fusion

Adaptive Multilevel Fusion Refinement Network for Object Detection in Remote Sensing Images