Abstract:Existing image fusion approaches are committed to using a single deep network to solve different image fusion problems, achieving promising performance in recent years. However, devoid of the ground-truth output, in these methods, only the appearance from source images can be exploited during the training process to generate the fused images, resulting in suboptimal solutions. To this end, we advocate a self-evolutionary training formula by introducing a novel memory unit architecture (MUFusion). In this unit, specifically, we utilize the intermediate fusion results obtained during the training process to further collaboratively supervise the fused image. In this way, our fusion results can not only learn from the original input images, but also benefit from the intermediate output of the network itself. Furthermore, an adaptive unified loss function is designed based on this memory unit, which is composed of two loss items, i.e. , content loss and memory loss. In particular, the content loss is calculated based on the activity level maps of source images, which can constrain the output image to contain specific information. On the other hand, the memory loss is obtained based on the previous output of our model, which is utilized to force the network to yield fusion results with higher quality. Considering the handcrafted activity level maps cannot consistently reflect the accurate salience judgement, we put two adaptive weight items between them to prevent this degradation phenomenon. In general, our MUFusion can effectively handle a series of image fusion tasks, including infrared and visible image fusion, multi-focus image fusion, multi-exposure image fusion, and medical image fusion. Particularly, the source images are concatenated in the channel dimension. After that, a densely connected feature extraction network with two scales is used to extract the deep features of the source images. Following this, the fusion result is obtained by two feature reconstruction blocks with skip connections from the feature extraction network. Qualitative and quantitative experiments on 4 image fusion subtasks demonstrate the superiority of our MUFusion, compared to the state-of-the-art methods.

Fusionmlp: A Mlp-Based Unified Image Fusion Framework

StackMFF: End-to-end Multi-Focus Image Stack Fusion Network

Multi-Modal Image Fusion Via Deep Laplacian Pyramid Hybrid Network

TransFuse: A Unified Transformer-based Image Fusion Framework using Self-supervised Learning

Trans2Fuse: Empowering image fusion through self-supervised learning and multi-modal transformations via transformer networks

UNIFusion: A Lightweight Unified Image Fusion Network

LeGFusion: Locally Enhanced Global Learning for Multimodal Image Fusion

LeGFusion: Locally-enhanced Global Learning for Multi-Modal Image Fusion

Fusion from Decomposition: A Self-Supervised Approach for Image Fusion and Beyond

MM-Net: A MixFormer-Based Multi-Scale Network for Anatomical and Functional Image Fusion

U2Fusion: A Unified Unsupervised Image Fusion Network

Multiscale Feature Interactive Network for Multifocus Image Fusion

A General Spatial-Frequency Learning Framework for Multimodal Image Fusion

FusionMamba: Efficient Remote Sensing Image Fusion with State Space Model

A Novel State Space Model with Local Enhancement and State Sharing for Image Fusion

DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception

SFPFusion: An Improved Vision Transformer Combining Super Feature Attention and Wavelet-Guided Pooling for Infrared and Visible Images Fusion

MUFusion: A general unsupervised image fusion network based on memory unit

DAGM-fusion: A dual-path CT-MRI image fusion model based multi-axial gated MLP

TUFusion: A Transformer-based Universal Fusion Algorithm for Multimodal Images

FuseFormer: A Transformer for Visual and Thermal Image Fusion