Abstract:Multi-modality image fusion and segmentation play a vital role in autonomous driving and robotic operation. Early efforts focus on boosting the performance for only one task, \emph{e.g.,} fusion or segmentation, making it hard to reach~`Best of Both Worlds'. To overcome this issue, in this paper, we propose a \textbf{M}ulti-\textbf{i}nteractive \textbf{F}eature learning architecture for image fusion and \textbf{Seg}mentation, namely SegMiF, and exploit dual-task correlation to promote the performance of both tasks. The SegMiF is of a cascade structure, containing a fusion sub-network and a commonly used segmentation sub-network. By slickly bridging intermediate features between two components, the knowledge learned from the segmentation task can effectively assist the fusion task. Also, the benefited fusion network supports the segmentation one to perform more pretentiously. Besides, a hierarchical interactive attention block is established to ensure fine-grained mapping of all the vital information between two tasks, so that the modality/semantic features can be fully mutual-interactive. In addition, a dynamic weight factor is introduced to automatically adjust the corresponding weights of each task, which can balance the interactive feature correspondence and break through the limitation of laborious tuning. Furthermore, we construct a smart multi-wave binocular imaging system and collect a full-time multi-modality benchmark with 15 annotated pixel-level categories for image fusion and segmentation. Extensive experiments on several public datasets and our benchmark demonstrate that the proposed method outputs visually appealing fused images and perform averagely $7.66\%$ higher segmentation mIoU in the real-world scene than the state-of-the-art approaches. The source code and benchmark are available at \url{<a class="link-external link-https" href="https://github.com/JinyuanLiu-CV/SegMiF" rel="external noopener nofollow">this https URL</a>}.

StitchFusion: Weaving Any Visual Modalities to Enhance Multimodal Semantic Segmentation

U3M: Unbiased Multiscale Modal Fusion Model for Multimodal Semantic Segmentation

Semantic segmentation guided feature point classification and seam fusion for image stitching

Centering the Value of Every Modality: Towards Efficient and Resilient Modality-agnostic Semantic Segmentation

Interpretation on Multi-modal Visual Fusion

Deep Unfolding Multi-modal Image Fusion Network Via Attribution Analysis

FusionSAM: Latent Space driven Segment Anything Model for Multimodal Fusion and Segmentation

Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion

Semantic Guidance Fusion Network for Cross-Modal Semantic Segmentation

ExFuse: Enhancing Feature Fusion for Semantic Segmentation

Bridging the Gap between Multi-focus and Multi-modal: A Focused Integration Framework for Multi-modal Image Fusion

Local-to-Global Cross-Modal Attention-Aware Fusion for HSI-X Semantic Segmentation

A Data Augmentation Method Based on Multi-Modal Image Fusion for Detection and Segmentation

Towards Deeper and Better Multi-view Feature Fusion for 3D Semantic Segmentation

From Text to Pixels: A Context-Aware Semantic Synergy Solution for Infrared and Visible Image Fusion

Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation

Multi-Sem Fusion: Multimodal Semantic Fusion for 3-D Object Detection

A Crossmodal Multiscale Fusion Network for Semantic Segmentation of Remote Sensing Data

Progressive Fusion for Multimodal Integration

Interactive Fusion and Correlation Network for Three-Modal Images Few-Shot Semantic Segmentation

LMFNet: An Efficient Multimodal Fusion Approach for Semantic Segmentation in High-Resolution Remote Sensing