Abstract:Multi-modality image fusion and segmentation play a vital role in autonomous driving and robotic operation. Early efforts focus on boosting the performance for only one task, \emph{e.g.,} fusion or segmentation, making it hard to reach~`Best of Both Worlds'. To overcome this issue, in this paper, we propose a \textbf{M}ulti-\textbf{i}nteractive \textbf{F}eature learning architecture for image fusion and \textbf{Seg}mentation, namely SegMiF, and exploit dual-task correlation to promote the performance of both tasks. The SegMiF is of a cascade structure, containing a fusion sub-network and a commonly used segmentation sub-network. By slickly bridging intermediate features between two components, the knowledge learned from the segmentation task can effectively assist the fusion task. Also, the benefited fusion network supports the segmentation one to perform more pretentiously. Besides, a hierarchical interactive attention block is established to ensure fine-grained mapping of all the vital information between two tasks, so that the modality/semantic features can be fully mutual-interactive. In addition, a dynamic weight factor is introduced to automatically adjust the corresponding weights of each task, which can balance the interactive feature correspondence and break through the limitation of laborious tuning. Furthermore, we construct a smart multi-wave binocular imaging system and collect a full-time multi-modality benchmark with 15 annotated pixel-level categories for image fusion and segmentation. Extensive experiments on several public datasets and our benchmark demonstrate that the proposed method outputs visually appealing fused images and perform averagely $7.66\%$ higher segmentation mIoU in the real-world scene than the state-of-the-art approaches. The source code and benchmark are available at \url{<a class="link-external link-https" href="https://github.com/JinyuanLiu-CV/SegMiF" rel="external noopener nofollow">this https URL</a>}.

Multi-Graph Based Hierarchical Semantic Fusion for Cross-Modal Representation

Deep Multi-Graph Hierarchical Enhanced Semantic Representation for Cross-Modal Retrieval

Dual Semantic Fusion Hashing for Multi-Label Cross-Modal Retrieval

Complementarity is the king: Multi-modal and multi-grained hierarchical semantic enhancement network for cross-modal retrieval

Unsupervised multi-perspective fusing semantic alignment for cross-modal hashing retrieval

A cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learning

Multi-task hierarchical convolutional network for visual-semantic cross-modal retrieval

Multi-Granularity Semantic Information Integration Graph for Cross-Modal Hash Retrieval

Dense Multimodal Fusion for Hierarchically Joint Representation

Dual graph-structured semantics multi-subspace learning for cross-modal retrieval

Multi-Modal Fusion-Based Multi-Task Semantic Communication System

Iterative graph attention memory network for cross-modal retrieval

Cross-Modal Hash Method Based on Multi-Scale Fusion and Projection Matching Constraint

Semantic Guidance Fusion Network for Cross-Modal Semantic Segmentation

Anchor Graph Structure Fusion Hashing for Cross-Modal Similarity Search

Local-to-Global Cross-Modal Attention-Aware Fusion for HSI-X Semantic Segmentation

Cross-modal retrieval based on multi-dimensional feature fusion hashing

Cross‐modal fusion encoder via graph neural network for referring image segmentation

Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation

Weighted Graph-structured Semantics Constraint Network for Cross-Modal Retrieval

Multi-modal Fine-grained Retrieval with Local and Global Cross-Attention.