Abstract:Multi-modality image fusion and segmentation play a vital role in autonomous driving and robotic operation. Early efforts focus on boosting the performance for only one task, \emph{e.g.,} fusion or segmentation, making it hard to reach~`Best of Both Worlds'. To overcome this issue, in this paper, we propose a \textbf{M}ulti-\textbf{i}nteractive \textbf{F}eature learning architecture for image fusion and \textbf{Seg}mentation, namely SegMiF, and exploit dual-task correlation to promote the performance of both tasks. The SegMiF is of a cascade structure, containing a fusion sub-network and a commonly used segmentation sub-network. By slickly bridging intermediate features between two components, the knowledge learned from the segmentation task can effectively assist the fusion task. Also, the benefited fusion network supports the segmentation one to perform more pretentiously. Besides, a hierarchical interactive attention block is established to ensure fine-grained mapping of all the vital information between two tasks, so that the modality/semantic features can be fully mutual-interactive. In addition, a dynamic weight factor is introduced to automatically adjust the corresponding weights of each task, which can balance the interactive feature correspondence and break through the limitation of laborious tuning. Furthermore, we construct a smart multi-wave binocular imaging system and collect a full-time multi-modality benchmark with 15 annotated pixel-level categories for image fusion and segmentation. Extensive experiments on several public datasets and our benchmark demonstrate that the proposed method outputs visually appealing fused images and perform averagely $7.66\%$ higher segmentation mIoU in the real-world scene than the state-of-the-art approaches. The source code and benchmark are available at \url{<a class="link-external link-https" href="https://github.com/JinyuanLiu-CV/SegMiF" rel="external noopener nofollow">this https URL</a>}.

Contour-Based Multi-Source Information Fusion for Motion Segmentation

Motionobject segmentation algorithm based on spatial-temporal information

Multi-Frame Image Fusion Method Combining Spatial-Temporal Saliency Detection and Nsct

Contour Based Automatic Scene Segmentation in Image Sequences

Multi-Motion Segmentation: Combining Geometric Model-Fitting and Optical Flow for RGB Sensors

A Multi-Scale Recurrent Framework for Motion Segmentation With Event Camera

Multiobjective multiple features fusion: A case study in image segmentation

Contour-Based Motion Segmentation Using Few Priors

Motion Objects Segmentation Using a New Level Set Based Method

Detail enhanced multi-source fusion using visual weight map extraction based on multi scale edge preserving decomposition

A Hybride Active Contour Model Driven by Global and Local Image Information

Multi-object Tracking Via MHT with Multiple Information Fusion in Surveillance Video

Two-scale decomposition-based multifocus image fusion framework combined with image morphology and fuzzy set theory

Multi-focus Image Fusion Based on Similarity Characteristics

Multi-Focus Image Fusion via Boundary Finding and Multi-Scale Morphological Focus-Measure

A Multi-View Dynamic Fusion Framework: How to Improve the Multimodal Brain Tumor Segmentation from Multi-Views?

Multi-Focus Image Fusion Method Based on Multi-Scale Decomposition of Information Complementary

Dynamic Multiple Object Segmentation with Spatio-Temporal Filtering

Multisensor video fusion based on spatial-temporal salience detection

MV-MOS: Multi-View Feature Fusion for 3D Moving Object Segmentation

Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation