Abstract:Multi-modality image fusion and segmentation play a vital role in autonomous driving and robotic operation. Early efforts focus on boosting the performance for only one task, \emph{e.g.,} fusion or segmentation, making it hard to reach~`Best of Both Worlds'. To overcome this issue, in this paper, we propose a \textbf{M}ulti-\textbf{i}nteractive \textbf{F}eature learning architecture for image fusion and \textbf{Seg}mentation, namely SegMiF, and exploit dual-task correlation to promote the performance of both tasks. The SegMiF is of a cascade structure, containing a fusion sub-network and a commonly used segmentation sub-network. By slickly bridging intermediate features between two components, the knowledge learned from the segmentation task can effectively assist the fusion task. Also, the benefited fusion network supports the segmentation one to perform more pretentiously. Besides, a hierarchical interactive attention block is established to ensure fine-grained mapping of all the vital information between two tasks, so that the modality/semantic features can be fully mutual-interactive. In addition, a dynamic weight factor is introduced to automatically adjust the corresponding weights of each task, which can balance the interactive feature correspondence and break through the limitation of laborious tuning. Furthermore, we construct a smart multi-wave binocular imaging system and collect a full-time multi-modality benchmark with 15 annotated pixel-level categories for image fusion and segmentation. Extensive experiments on several public datasets and our benchmark demonstrate that the proposed method outputs visually appealing fused images and perform averagely $7.66\%$ higher segmentation mIoU in the real-world scene than the state-of-the-art approaches. The source code and benchmark are available at \url{<a class="link-external link-https" href="https://github.com/JinyuanLiu-CV/SegMiF" rel="external noopener nofollow">this https URL</a>}.

MIMF: Mutual Information-Driven Multimodal Fusion

Multi-Modality Cascaded Fusion Technology for Autonomous Driving

Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis

Multi-Modal Fusion Based on Depth Adaptive Mechanism for 3D Object Detection

DMFF: dual-way multimodal feature fusion for 3D object detection

Robust-FusionNet: Deep Multimodal Sensor Fusion for 3-D Object Detection Under Severe Weather Conditions

A Novel Multimodal Fusion Network Based on a Joint Coding Model for Lane Line Segmentation

MMLF: Multi-modal Multi-class Late Fusion for Object Detection with Uncertainty Estimation

E2E-MFD: Towards End-to-End Synchronous Multimodal Fusion Detection

MultiFuser: Multimodal Fusion Transformer for Enhanced Driver Action Recognition

Bridging the Gap between Multi-focus and Multi-modal: A Focused Integration Framework for Multi-modal Image Fusion

A Generalized Multi-Modal Fusion Detection Framework

Multimodal Fusion Network for 3-D Lane Detection

MEFusion: Unsupervised Mutual Enhancement for Multimodal Image Fusion

Multimodal Fusion Method Based on Self-Attention Mechanism

Multi-Modal Attention Guided Real-Time Lane Detection

Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation

MFST: Multi-Modal Feature Self-Adaptive Transformer for Infrared and Visible Image Fusion

Learnable fusion mechanisms for multimodal object detection in autonomous vehicles

Mutual information maximization and feature space separation and bi-bimodal mo-dality fusion for multimodal sentiment analysis

Deep Multimodal Fusion Network for Semantic Segmentation Using Remote Sensing Image and LiDAR Data